STORM+: Fully Adaptive SGD with Momentum for Nonconvex Optimization
Dez 6, 2021
Sprecher:innen
Kfir Y. Levy
Sprecher:in · 0 Follower:innen
Volkan Cevher
Sprecher:in · 0 Follower:innen
Über
In this work we investigate stochastic non-convex optimization problems where the objective is an expectation over smooth loss functions, and the goal is to find an approximate stationary point. The most popular approach to handling such problems is variance reduction techniques, which are also known to obtain tight convergence rates, matching the lower bounds in this case. Nevertheless, these techniques require a careful maintenance of anchor points in conjunction with appropriately selected “`"mega-batchsizes". This leads to a challenging hyperparameter tuning problem, that weakens their practicality. Recently, [Cutkosky & Orabona, 2019] have shown that one can employ recursive momentum in order to avoid the use of anchor points and large batchsizes, and still obtain the optimal rate for this setting. Yet, their method called STORM crucially relies on the knowledge of the smoothness, as well a bound on the gradient norms. We propose STORM^+, a new method that is completely parameter-free, does not require large batch-sizes, and obtains the optimal O(1/T^1/3) rate for finding an approximate stationary point. Our work builds on the STORM algorithm, in conjunction with a novel approach to adaptively set the learning rate and momentum parameters.In this work we investigate stochastic non-convex optimization problems where the objective is an expectation over smooth loss functions, and the goal is to find an approximate stationary point. The most popular approach to handling such problems is variance reduction techniques, which are also known to obtain tight convergence rates, matching the lower bounds in this case. Nevertheless, these techniques require a careful maintenance of anchor points in conjunction with appropriately selected “…
Organisator
NeurIPS 2021
Konto · 1,9k Follower:innen
Über NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
ToAlign: Task-Oriented Alignment for Unsupervised Domain Adaptation
Später ansehen
Guoqiang Wei, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
AI workloads inside databases
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
2nd Workshop on Self-Supervised Learning: Theory and Practice
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Fast Axiomatic Attribution for Neural Networks
Später ansehen
Robin Hesse, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Confident Anchor-Induced Multi-Source Free Domain Adaptation
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Machine Learning for Healthcare
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%