High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation
Nov 28, 2022
Speakers
Murat A. Erdogdu
Sprecher:in · 0 Follower:innen
Taiji Suzuki
Sprecher:in · 1 Follower:in
Zhichao Wang
Sprecher:in · 0 Follower:innen
About
We study the first gradient descent step on the first-layer parameters W in a two-layer neural network: f(x) = 1/√(N)a^⊤σ(W^⊤x), where W∈ℝ^d× N, a∈ℝ^N are randomly initialized, and the training objective is the empirical MSE loss: 1/n∑_i=1^n (f(x_i)-y_i)^2. In the proportional asymptotic limit where n,d,N→∞ at the same rate, and an idealized student-teacher setting where the teacher f^* is a single-index model, we compute the prediction risk of ridge regression on the conjugate kernel after one gradient step on W with learning rate η. We consider two scalings of the first step learning rate η. For small η, we establish a Gaussian equivalence property for the trained feature map, and prove that the learned kernel improves upon the initial random features model, but cannot defeat the best linear model on the input. Whereas for sufficiently large η, we prove that for certain f^*, the same ridge estimator on trained features can go beyond this “linear regime” and outperform a wide range of (fixed) kernels. Our results demonstrate that even one gradient step can lead to a considerable advantage over random features, and highlight the role of learning rate scaling in the initial phase of training.We study the first gradient descent step on the first-layer parameters W in a two-layer neural network: f(x) = 1/√(N)a^⊤σ(W^⊤x), where W∈ℝ^d× N, a∈ℝ^N are randomly initialized, and the training objective is the empirical MSE loss: 1/n∑_i=1^n (f(x_i)-y_i)^2. In the proportional asymptotic limit where n,d,N→∞ at the same rate, and an idealized student-teacher setting where the teacher f^* is a single-index model, we compute the prediction risk of ridge regression on the conjugate kernel after one…
Organizer
NeurIPS 2022
Konto · 961 Follower:innen
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Lagrangian Model Based Reinforcement Learning
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Rethinking the Reverse-engineering of Trojan Triggers
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
From Advice-Taking to Active Learnig
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
A Mixture Of Surprises for Unsupervised Reinforcement Learning
Später ansehen
Andrew Zhao, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
A Robust Phased Elimination Algorithm for Corruption-Tolerant Gaussian Process Bandits
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
What is Where by Looking (WWbL) - Weakly-Supervised Open-World Phrase-Grounding without Text Inputs
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%