Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation
Dec 6, 2021
Speakers
Vidya Muthukumar
Speaker · 0 followers
Christos Thrampoulidis
Speaker · 0 followers
About
The growing literature on "benign overfitting" in overparameterized models has been mostly restricted to regression or binary classification settings; however, most success stories of modern machine learning have been recorded in multiclass settings. Motivated by this discrepancy, we study benign overfitting in multiclass linear classification. Specifically, we consider the following popular training algorithms on separable data: (i) empirical risk minimization (ERM) with cross-entropy loss, which converges to the multiclass support vector machine (SVM) solution; (ii) ERM with least-squares loss, which converges to the min-norm interpolating (MNI) solution; and, (iii) the one-vs-all SVM classifier. Our first key finding is that under a simple sufficient condition, all three algorithms lead to classifiers that interpolate the training data and have equal accuracy. When the data is generated from Gaussian mixtures or a multinomial logistic model, this condition holds under high enough effective overparameterization. Second, we derive novel error bounds on the accuracy of the MNI classifier, thereby showing that all three training algorithms lead to benign overfitting under sufficient overparameterization. Ultimately, our analysis shows that good generalization is possible for SVM solutions beyond the realm in which typical margin-based bounds apply.The growing literature on "benign overfitting" in overparameterized models has been mostly restricted to regression or binary classification settings; however, most success stories of modern machine learning have been recorded in multiclass settings. Motivated by this discrepancy, we study benign overfitting in multiclass linear classification. Specifically, we consider the following popular training algorithms on separable data: (i) empirical risk minimization (ERM) with cross-entropy loss, whi…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Opening Remarks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Adversarial Training Blocks Generalization in Neural Policies
Watch later
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%
Distributed Deep Learning In Open Collaborations
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Towards Reliable and Robust Model Explanations
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Aligned Structured Sparsity Learning for Efficient Image Super-Resolution
Watch later
Yulun Zhang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%