Particle Dual Averaging: Optimization of Mean Field Neural Network with Global Convergence Rate Analysis
Dec 6, 2021
Speakers
Atsushi Nitanda
Speaker · 0 followers
Taiji Suzuki
Speaker · 1 follower
About
We propose the particle dual averaging (PDA) method, which generalizes the dual averaging method in convex optimization to the optimization over probability distributions with quantitative runtime guarantee. The algorithm consists of an inner loop and outer loop: the inner loop utilizes the Langevin algorithm to approximately solve for a stationary distribution, which is then optimized in the outer loop. The method can thus be interpreted as an extension of the Langevin algorithm to naturally handle nonlinear functional on the probability space. An important application of the proposed method is the optimization of neural network in the mean field regime, which is theoretically attractive due to the presence of nonlinear feature learning, but quantitative convergence rate can be challenging to obtain. By adapting finite-dimensional convex optimization theory into the space of distributions, we not only establish global convergence of PDA for two-layer mean field neural networks under more general settings and simpler analysis, but also provide quantitative polynomial runtime guarantee. Our theoretical results are supported by numerical simulations on neural networks with reasonable size.We propose the particle dual averaging (PDA) method, which generalizes the dual averaging method in convex optimization to the optimization over probability distributions with quantitative runtime guarantee. The algorithm consists of an inner loop and outer loop: the inner loop utilizes the Langevin algorithm to approximately solve for a stationary distribution, which is then optimized in the outer loop. The method can thus be interpreted as an extension of the Langevin algorithm to naturally ha…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Information Directed Reward Learning for Reinforcement Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Coding Session 2
Watch later
Shirley Ho, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
CCVS: Context-aware Controllable Video Synthesis
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
DiBS: Differentiable Bayesian Structure Learning
Watch later
Lars Lorch, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Emergent Discrete Communication in Semantic Spaces
Watch later
Mycal Tucker, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Shift Invariance Can Reduce Adversarial Robustness
Watch later
Vasu Singla, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%