Toward Understanding Why Adam Converges Faster Than SGD for Transformers
Dec 2, 2022
Speakers
Yuanzhi Li
Speaker · 2 followers
About
While stochastic gradient descent (SGD) is still the most popular optimization algorithm in deep learning, adaptive algorithms such as Adam have established empirical advantages over SGD in some deep learning applications such as training transformers. However, it remains a question why Adam converges significantly faster than SGD in these scenarios. In this paper, we explore one explanation of why Adam converges faster than SGD using a new concept directional sharpness. We argue that the performance of optimization algorithms is closely related to the directional sharpness of the update steps, and show SGD has much worse directional sharpness compared to adaptive algorithms. We further observe that only a small fraction of the coordinates causes the bad sharpness and slow convergence of SGD, and propose to use coordinate-wise clipping as a solution to SGD and other optimization algorithms. We demonstrate the effect of coordinate-wise clipping in sharpness reduction and speeding up the convergence of optimization algorithms under various settings, and conclude that the sharpness reduction effect of adaptive coordinate-wise scaling is the reason for Adam’s success in practice.While stochastic gradient descent (SGD) is still the most popular optimization algorithm in deep learning, adaptive algorithms such as Adam have established empirical advantages over SGD in some deep learning applications such as training transformers. However, it remains a question why Adam converges significantly faster than SGD in these scenarios. In this paper, we explore one explanation of why Adam converges faster than SGD using a new concept directional sharpness. We argue that the perfor…
Organizer
NeurIPS 2022
Account · 953 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Multivariate Time-Series Forecasting with Temporal Polynomial Graph Neural Networks
Watch later
Yijing Liu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Inverse Design for Fluid-Structure Interactions using Graph Network Simulators
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Benchmarking Node Outlier Detection on Graphs
Watch later
Kay Liu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Panel: Greenhouse gas emissions and climate vulnerability impact assessment
Watch later
Peetak Mitra, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Temporary Goals for Exploration
Watch later
Haoyang Xu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Scalable Deterministic Global Optimization Algorithm for Training Optimal Decision Tree
Watch later
Kaixun Hua, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%