Benign Overfitting in Deep Neural Networks under Lazy Training
Jul 24, 2023
Speakers
Zhenyu Zhu
Speaker · 0 followers
Fanghui Liu
Speaker · 0 followers
Grigorios G. Chrysos
Speaker · 0 followers
Francesco Locatello
Speaker · 1 follower
Volkan Cevher
Speaker · 0 followers
About
This paper focuses on over-parameterized deep neural networks (DNNs) with ReLU activation functions and proves that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification while obtaining (nearly) zero-training error. For this purpose, we unify three interrelated concepts of overparameterization, benign overfitting, and the Lipschitz constant of DNNs. Our results indicate that interpolating with smoother functions leads to better generalization. Furthermore, we investigate the special case where interpolating smooth ground-truth functions is performed by DNNs under the neural tangent kernel (NTK) regime for generalization. Our result demonstrates that the generalization error converges to a constant order than only depends on label noise and initialization noise, which theoretically verifies benign overfitting. Our analysis provides a tight lower bound on the normalized margin under non-smooth activation functions, as well as the minimum eigenvalue of NTK under high-dimensional settings, which has its own interest in learning theory.This paper focuses on over-parameterized deep neural networks (DNNs) with ReLU activation functions and proves that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification while obtaining (nearly) zero-training error. For this purpose, we unify three interrelated concepts of overparameterization, benign overfitting, and the Lipschitz constant of DNNs. Our results indicate that interpolating with smoother functions leads to better generalization.…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Subequivariant Graph Reinforcement Learning in 3D Environments
Watch later
Runfa Chen, …
Distributed Contextual Linear Bandits with Minimax Optimal Communication Cost
Watch later
Sanae Amani, …
Backpropagation Alternatives and Scalable AI
Watch later
Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models
Watch later
Dongjun Kim, …
