Convergence of adaptive algorithms for constrained weakly convex optimization
Dec 6, 2021
Speakers
Ahmet Alacaoglu
Speaker · 0 followers
Yura Malitsky
Speaker · 0 followers
Volkan Cevher
Speaker · 0 followers
About
We analyze the adaptive first order algorithm AMSGrad, for solving a constrained stochastic optimization problem with a weakly convex objective. We prove the Õ(t^-1/4) rate of convergence for the norm of the gradient of Moreau envelope, which is the standard stationarity measure for this class of problems. It matches the known rates that adaptive algorithms enjoy for the specific case of unconstrained smooth nonconvex stochastic optimization. Our analysis works with mini-batch size of 1, constant first and second order moment parameters, and possibly unbounded optimization domains. Finally, we illustrate the applications and extensions of our results to specific problems and algorithms.We analyze the adaptive first order algorithm AMSGrad, for solving a constrained stochastic optimization problem with a weakly convex objective. We prove the Õ(t^-1/4) rate of convergence for the norm of the gradient of Moreau envelope, which is the standard stationarity measure for this class of problems. It matches the known rates that adaptive algorithms enjoy for the specific case of unconstrained smooth nonconvex stochastic optimization. Our analysis works with mini-batch size of 1, constan…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
ErrorCompensatedX: error compensation for variance reduced algorithms
Watch later
Hanlin Tang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Black Box Probabilistic Numerics
Watch later
Onur Teymur, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Increasing Data Diversity with Iterative Sampling to Improve Performance
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Closing Remarks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
CHIP: CHannel Independence-based Pruning for Compact Neural Networks
Watch later
Yang Sui, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Leveraging Spatial and Temporal Correlations in Sparsified Mean Estimation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%