Label Noise SGD Provably Prefers Flat Global Minimizers
Dec 6, 2021
Speakers
Alex Damian
Speaker · 1 follower
Jason D. Lee
Speaker · 0 followers
About
In overparametrized models, the noise in stochastic gradient descent (SGD) implicitly regularizes the optimization trajectory and determines which local minimum SGD converges to. Motivated by empirical studies that demonstrate that training with noisy labels improves generalization, we study the implicit regularization effect of SGD with label noise. We show that SGD with label noise converges to a stationary point of a regularized loss L(θ) +λ R(θ), where L(θ) is the training loss, λ is an effective regularization parameter depending on the step size, strength of the label noise, and the batch size, and R(θ) is an explicit regularizer that penalizes sharp minimizers. Our analysis uncovers an additional regularization effect of large learning rates beyond the linear scaling rule that penalizes large eigenvalues of the Hessian more than small ones. We also prove extensions to classification with general loss functions, significantly strengthening the prior work of Blanc et al. to global convergence and large learning rates and of HaoChen et al. to general models.In overparametrized models, the noise in stochastic gradient descent (SGD) implicitly regularizes the optimization trajectory and determines which local minimum SGD converges to. Motivated by empirical studies that demonstrate that training with noisy labels improves generalization, we study the implicit regularization effect of SGD with label noise. We show that SGD with label noise converges to a stationary point of a regularized loss L(θ) +λ R(θ), where L(θ) is the training loss, λ is an effe…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
PASS: An ImageNet replacement for self-supervised pretraining without humans
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Status-quo policy gradient in Multi-Agent Reinforcement Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning
Watch later
Shiming Chen, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Neural Flows: Efficient Alternative to Neural ODEs
Watch later
Marin Biloš, …
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%
CARMS: Categorical-Antithetic-REINFORCE Multi-Sample Gradient Estimator
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Towards Reliable and Robust Model Explanations
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%