Gu, Y; Zheng, X; Aste, T; (2024) Unraveling the Enigma of Double Descent: an in-depth Analysis Through the Lens of Learned Feature Space. In: 12th International Conference on Learning Representations, ICLR 2024. International Conference on Learning Representations (ICLR): Vienna, Austria.

Preview

Text 287_Unraveling_the_Enigma_of_D.pdf - Published Version Download (996kB) | Preview

Abstract

Double descent presents a counter-intuitive aspect within the machine learning domain, and researchers have observed its manifestation in various models and tasks. While some theoretical explanations have been proposed for this phenomenon in specific contexts, an accepted theory for its occurring mechanism in deep learning remains yet to be established. In this study, we revisit the phenomenon of double descent and demonstrate that the presence of noisy data strongly influences its occurrence. By comprehensively analysing the feature space of learned representations, we unveil that double descent arises in imperfect models trained with noisy data. We argue that while small and intermediate models before the interpolation threshold follow the traditional bias-variance trade-off, over-parameterized models interpolate noisy samples among robust data thus acquiring the capability to separate the information from the noise. The source code is available at https://github.com/Yufei-Gu-451/double_descent_inference.git.

Download activity - last month

Export as CSVXMLJSON

Download activity - last 12 months

Export as CSVXMLJSON

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item