Benign Overfitting in Deep Neural Networks under Lazy Training
24. července 2023
Řečníci
Zhenyu Zhu
Sprecher:in · 0 Follower:innen
Fanghui Liu
Sprecher:in · 0 Follower:innen
Grigorios G. Chrysos
Sprecher:in · 0 Follower:innen
Francesco Locatello
Sprecher:in · 1 Follower:in
Volkan Cevher
Sprecher:in · 0 Follower:innen
O prezentaci
This paper focuses on over-parameterized deep neural networks (DNNs) with ReLU activation functions and proves that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification while obtaining (nearly) zero-training error. For this purpose, we unify three interrelated concepts of overparameterization, benign overfitting, and the Lipschitz constant of DNNs. Our results indicate that interpolating with smoother functions leads to better generalization. Furthermore, we investigate the special case where interpolating smooth ground-truth functions is performed by DNNs under the neural tangent kernel (NTK) regime for generalization. Our result demonstrates that the generalization error converges to a constant order than only depends on label noise and initialization noise, which theoretically verifies benign overfitting. Our analysis provides a tight lower bound on the normalized margin under non-smooth activation functions, as well as the minimum eigenvalue of NTK under high-dimensional settings, which has its own interest in learning theory.This paper focuses on over-parameterized deep neural networks (DNNs) with ReLU activation functions and proves that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification while obtaining (nearly) zero-training error. For this purpose, we unify three interrelated concepts of overparameterization, benign overfitting, and the Lipschitz constant of DNNs. Our results indicate that interpolating with smoother functions leads to better generalization.…
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Evaluating Self-Supervised Learning via Risk Decomposition
Später ansehen
Yann Dubois, …
Half-Hop: A graph upsampling approach for slowing down message passing
Später ansehen
Mehdi Azabou, …
On the Within-Group Fairness of Screening Classifiers
Später ansehen
Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data
Später ansehen
Hien Dang, …
