Interpolation can hurt robust generalization even when there is no noise
Dec 6, 2021
Speakers
Alexandru Tifrea
Speaker · 0 followers
Konstantin Donhauser
Speaker · 0 followers
Michael Aerni
Speaker · 0 followers
Reinhard Heckel
Speaker · 0 followers
Fanny Yang
Speaker · 0 followers
About
Numerous recent works show that overparameterization implicitly reduces variance for minimum-norm interpolators and max-margin classifiers, suggesting vanishing benefits for ridge regularization in high dimensions. Putting this narrative into perspective, this paper reveals the unexpected benefits of a non-zero ridge penalty even when there is no noise: we prove that for both overparameterized linear regression and classification, avoiding interpolation can in fact significantly improve generalization. We show that the benefits are particularly prominent for distributionally and adversarially robust risks and hence provide, to the best of our knowledge, the first theoretical analysis on the empirical phenomenon of robust overfitting.Numerous recent works show that overparameterization implicitly reduces variance for minimum-norm interpolators and max-margin classifiers, suggesting vanishing benefits for ridge regularization in high dimensions. Putting this narrative into perspective, this paper reveals the unexpected benefits of a non-zero ridge penalty even when there is no noise: we prove that for both overparameterized linear regression and classification, avoiding interpolation can in fact significantly improve generali…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Scalars are universal: Gauge-equivariant machine learning, structured like classical physics
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Leveraging machine learning for identify hydrological extreme events under global climate change
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Bellman-consistent Pessimism for Offline Reinforcement Learning
Watch later
Tengyang Xie, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Machine Learning Meets Econometrics (MLECON)
Watch later
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%