Particle Dual Averaging: Optimization of Mean Field Neural Network with Global Convergence Rate Analysis
Dec 6, 2021
Speakers
Atsushi Nitanda
Speaker · 0 followers
Taiji Suzuki
Speaker · 1 follower
About
We propose the particle dual averaging (PDA) method, which generalizes the dual averaging method in convex optimization to the optimization over probability distributions with quantitative runtime guarantee. The algorithm consists of an inner loop and outer loop: the inner loop utilizes the Langevin algorithm to approximately solve for a stationary distribution, which is then optimized in the outer loop. The method can thus be interpreted as an extension of the Langevin algorithm to naturally handle nonlinear functional on the probability space. An important application of the proposed method is the optimization of neural network in the mean field regime, which is theoretically attractive due to the presence of nonlinear feature learning, but quantitative convergence rate can be challenging to obtain. By adapting finite-dimensional convex optimization theory into the space of distributions, we not only establish global convergence of PDA for two-layer mean field neural networks under more general settings and simpler analysis, but also provide quantitative polynomial runtime guarantee. Our theoretical results are supported by numerical simulations on neural networks with reasonable size.We propose the particle dual averaging (PDA) method, which generalizes the dual averaging method in convex optimization to the optimization over probability distributions with quantitative runtime guarantee. The algorithm consists of an inner loop and outer loop: the inner loop utilizes the Langevin algorithm to approximately solve for a stationary distribution, which is then optimized in the outer loop. The method can thus be interpreted as an extension of the Langevin algorithm to naturally ha…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Fairness and privacy aspects of ImageNet
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Overparameterization Improves Robustness to Covariate Shift in High-Dimensions
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Play to Grade: Testing Coding Games as Classifying Markov Decision Process
Watch later
Allen Nie, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Continual Auxiliary Task Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning Parameterized Task Structure for Generalization to Unseen Entities
Watch later
Anthony Liu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Fairness via Representation Neutralization
Watch later
Mengnan Du, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%