An Even More Optimal Stochastic Optimization Algorithm: Minibatching and Interpolation Learning
Dec 6, 2021
Speakers
Blake Woodworth
Speaker · 0 followers
Nathan Srebro
Speaker · 0 followers
About
We present and analyze an algorithm for optimizing smooth and convex or strongly convex objectives using minibatch stochastic gradient estimates. The algorithm is optimal with respect to its dependence on both the minibatch size and minimum expected loss simultaneously. This improves over the optimal method of Lan, which is insensitive to the minimum expected loss; over the optimistic acceleration of Cotter et al., which has suboptimal dependence on the minibatch size; and over the algorithm of Liu and Belkin, which is limited to least squares problems and is also similarly suboptimal. Applied to interpolation learning, the improvement over Cotter et al. and Liu and Belkin translates to a linear, rather than square-root, parallelization speedup.We present and analyze an algorithm for optimizing smooth and convex or strongly convex objectives using minibatch stochastic gradient estimates. The algorithm is optimal with respect to its dependence on both the minibatch size and minimum expected loss simultaneously. This improves over the optimal method of Lan, which is insensitive to the minimum expected loss; over the optimistic acceleration of Cotter et al., which has suboptimal dependence on the minibatch size; and over the algorithm of…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
The impact of weather information on machine-learning probabilistic electricity demand predictions
Watch later
Yifu Ding, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement
Watch later
Sam Daulton, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Task-Driven Discovery of Perceptual Schemas for Generalization in Reinforcement Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Task-Adaptive Neural Network Search with Meta-Contrastive Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Automatic Symmetry Discovery with Lie Algebra Convolutional Network
Watch later
Nima Dehmamy, …
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%
KitchenShift: Evaluating Zero-Shot Generalization of Imitation-Based Policy Learning Under Domain Shifts
Watch later
Eliot Xing, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%