The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization
Nov 28, 2022
Speakers
Mihai Nica
Speaker · 0 followers
Daniel M. Roy
Speaker · 0 followers
About
The logit outputs of a feedforward neural network at initialization are conditionally Gaussian, given a random covariance matrix defined by the penultimate layer. In this work, we study the distribution of this random matrix. Recent work has shown that shaping the activation function as network depth grows large is necessary for this covariance matrix to be non-degenerate. However, the current infinite-width-style understanding of this shaping method is unsatisfactory for large depth: infinite-width analyses ignore the microscopic fluctuations from layer to layer, but these fluctuations accumulate over many layers. To overcome this shortcoming, we study the random covariance matrix in the shaped infinite-depth-and-width limit. We identify the precise scaling of the activation function necessary to arrive at a non-trivial limit, and show that the random covariance matrix is governed by a stochastic differential equation (SDE) that we call the Neural Covariance SDE. Using simulations, we show that the SDE closely matches the distribution of the random covariance matrix of finite networks. Additionally, we recover an if-and-only-if condition for exploding and vanishing norms of large shaped networks based on the activation function.The logit outputs of a feedforward neural network at initialization are conditionally Gaussian, given a random covariance matrix defined by the penultimate layer. In this work, we study the distribution of this random matrix. Recent work has shown that shaping the activation function as network depth grows large is necessary for this covariance matrix to be non-degenerate. However, the current infinite-width-style understanding of this shaping method is unsatisfactory for large depth: infinite-w…
Organizer
NeurIPS 2022
Account · 963 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Supervised Q-Learning for Continuous Control
Watch later
Hao Sun, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Neural Attentive Circuits: A General -Purpose Model with Modular Inductive Biases
Watch later
Martin Weiss, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Action-modulated midbrain dopamine activity arises from distributed control policies
Watch later
Jack Lindsey, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Domain Invariant Q-Learning for Model-Free Robust Continuous Control under Visual Distractions
Watch later
Tom Dupuis, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning
Watch later
Byungchan Ko, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Bayesian Persuasion for Algorithmic Recourse
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%