Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks
Dec 10, 2023
Speakers
Eshaan Nichani
Speaker · 0 followers
Alex Damian
Speaker · 1 follower
Jason D. Lee
Speaker · 0 followers
About
One of the central questions in the theory of deep learning is to understand how neural networks learn hierarchical features. The ability of deep networks to extract salient features is crucial to both their outstanding generalization ability and the modern deep learning paradigm of pretraining and finetuneing. However, this feature learning process remains poorly understood from a theoretical perspective, with existing analyses largely restricted to two-layer networks. In this work we show that three-layer neural networks have provably richer feature learning capabilities than two-layer networks. We analyze the features learned by a three-layer network trained with layer-wise gradient descent, and present a general purpose theorem which upper bounds the sample complexity and width needed to achieve low test error when the target has specific hierarchical structure. We instantiate our framework in specific statistical learning settings – single-index models and functions of quadratic features – and show that in the latter setting three-layer networks obtain a sample complexity improvement over all existing guarantees for two-layer networks. Crucially, this sample complexity improvement relies on the ability of three-layer networks to efficiently learn *nonlinear* features. We then establish a concrete optimization-based depth separation by constructing a function which is efficiently learnable via gradient descent on a three-layer network, yet cannot be learned efficiently by a two-layer network. Our work makes progress towards understanding the provable benefit of three-layer neural networks over two-layer networks in the feature learning regime.One of the central questions in the theory of deep learning is to understand how neural networks learn hierarchical features. The ability of deep networks to extract salient features is crucial to both their outstanding generalization ability and the modern deep learning paradigm of pretraining and finetuneing. However, this feature learning process remains poorly understood from a theoretical perspective, with existing analyses largely restricted to two-layer networks. In this work we show that…
Organizer
NeurIPS 2023
Account · 646 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Does Invariant Graph Learning via Environment Augmentation Learn Invariance?
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
No-Regret Learning in Dynamic Competition with Reference Effects Under Logit Demand
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Semi-Ensemble: A Simple Approach to Over-parameterized Model Interpolation
Watch later
Jiwoon Lee, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Reproducibility Study of ”Label-Free Explainability for Unsupervised Models”
Watch later
Gergely Papp, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The Pulse of Ethical Machine Learning in Health
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Aquaculture Mapping: Detecting and Classifying Aquaculture Ponds using Deep Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%