On the Role of Optimization in Double Descent: A Least Squares Study
Dec 6, 2021
Speakers
Ilja Kuzborskij
Speaker · 0 followers
Csaba Szepesvari
Speaker · 8 followers
Omar Rivasplata
Speaker · 0 followers
Amal Rannen-Triki
Speaker · 0 followers
Razvan Pascanu
Speaker · 0 followers
About
Empirically it has been observed that the performance of deep neural networks steadily improves as we increase model size, contradicting the classical view on overfitting and generalization. Recently, the double descent phenomena has been proposed to reconcile this observation with theory, suggesting that the test error has a second descent when the model becomes sufficiently overparametrized, as the model size itself acts as an implicit regularizer. In this paper we add to the growing body of work in this space, providing a careful study of learning dynamics as a function of model size for the least squares scenario. We show an excess risk bound for the gradient descent solution of the least squares objective. The bound depends on the smallest non-zero eigenvalue of the covariance matrix of the input features, via a functional form that has the double descent behaviour. This gives a new perspective on the double descent curves reported in the literature.Our analysis of the excess risk allows to decouple the effect of optimisation and generalisation error. In particular, we find that in case of noiseless regression, double descent is explained solely by optimisation-related quantities, which was missed in studies focusing on the Moore-Penrose pseudo-inverse solution. We believe that our derivation provides an alternative view compared to existing work, shedding some light on a possible cause of this phenomena, at least in the considered least squares setting. We empirically explore if our predictions hold for neural networks, in particular whether the covariance of intermediary hidden activations has a similar behaviour as the one predicted by our derivations.Empirically it has been observed that the performance of deep neural networks steadily improves as we increase model size, contradicting the classical view on overfitting and generalization. Recently, the double descent phenomena has been proposed to reconcile this observation with theory, suggesting that the test error has a second descent when the model becomes sufficiently overparametrized, as the model size itself acts as an implicit regularizer. In this paper we add to the growing body of w…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Ecological Theory of Reinforcement Learning: How Does Task Design Influence Agent Learning?
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Optimal Policies Tend To Seek Power
Watch later
Alex Turner, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Target Entropy Annealing for Discrete Soft Actor-Critic
Watch later
Yaosheng Xu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Adaptive Diffusion in Graph Neural Networks
Watch later
Jialin Zhao, …
Total of 2 viewers voted for saving the presentation to eternal vault which is 0.2%
Formalizing Generalization and Adversarial Robustness of Neural Networks to Weight Perturbations
Watch later
Yu-Lin Tsai, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%