The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization
28. listopadu 2022
Řečníci
Mihai Nica
Speaker · 0 followers
Daniel M. Roy
Speaker · 0 followers
O prezentaci
The logit outputs of a feedforward neural network at initialization are conditionally Gaussian, given a random covariance matrix defined by the penultimate layer. In this work, we study the distribution of this random matrix. Recent work has shown that shaping the activation function as network depth grows large is necessary for this covariance matrix to be non-degenerate. However, the current infinite-width-style understanding of this shaping method is unsatisfactory for large depth: infinite-width analyses ignore the microscopic fluctuations from layer to layer, but these fluctuations accumulate over many layers. To overcome this shortcoming, we study the random covariance matrix in the shaped infinite-depth-and-width limit. We identify the precise scaling of the activation function necessary to arrive at a non-trivial limit, and show that the random covariance matrix is governed by a stochastic differential equation (SDE) that we call the Neural Covariance SDE. Using simulations, we show that the SDE closely matches the distribution of the random covariance matrix of finite networks. Additionally, we recover an if-and-only-if condition for exploding and vanishing norms of large shaped networks based on the activation function.The logit outputs of a feedforward neural network at initialization are conditionally Gaussian, given a random covariance matrix defined by the penultimate layer. In this work, we study the distribution of this random matrix. Recent work has shown that shaping the activation function as network depth grows large is necessary for this covariance matrix to be non-degenerate. However, the current infinite-width-style understanding of this shaping method is unsatisfactory for large depth: infinite-w…
Organizátor
NeurIPS 2022
Account · 963 followers
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Provable Generalization of Overparameterized Meta-learning Trained with SGD
Watch later
Yu Huang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
HAMUX: A Universal Abstraction for Hierarchical Hopfield Networks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On Enforcing Better Conditioned Meta-Learning for Rapid Few-Shot Adaptation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
When are Local Queries Useful for Robust Learning?
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
![[Re] Value Alignment Verification](https://ma.slideslive.com/library/presentations/38995277/thumbnail/re-value-alignment-verification_bijyJ3_small.jpg)
[Re] Value Alignment Verification
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Scalable Bayesian optimization for high-dimensional urban transportation problems
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%