Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees
Jul 24, 2023
Speakers
Anastasiia Koloskova
Speaker · 0 followers
Hadrien Hendrikx
Speaker · 0 followers
Sebastian Stich
Speaker · 0 followers
About
Gradient clipping is a popular modification to standard (stochastic) gradient descent, at every iteration limiting the gradient norm to a certain value c >0. It is widely used for example for stabilizing the training of deep learning models (Goodfellow et al., 2016), or for enforcing differential privacy (Abadi et al., 2016). Despite popularity and simplicity of the clipping mechanism, its convergence guarantees often require specific values of c and strong noise assumptions. In this paper, we give convergence guarantees that show precise dependence on arbitrary clipping thresholds c and show that our guarantees are tight with both deterministic and stochastic gradients. In particular, we show that (i) for deterministic gradient descent, the clipping threshold only affects the higher-order terms of convergence, (ii) in the stochastic setting convergence to the true optimum cannot be guaranteed under the standard noise assumption, even under arbitrary small step-sizes. We give matching upper and lower bounds for convergence of the gradient norm when running clipped SGD, and illustrate these results with experiments.Gradient clipping is a popular modification to standard (stochastic) gradient descent, at every iteration limiting the gradient norm to a certain value c >0. It is widely used for example for stabilizing the training of deep learning models (Goodfellow et al., 2016), or for enforcing differential privacy (Abadi et al., 2016). Despite popularity and simplicity of the clipping mechanism, its convergence guarantees often require specific values of c and strong noise assumptions. In this paper, …
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Quantifying the Variability Collapse of Neural Networks
Watch later
Jing Xu, …
The Melting Pot of Neural Algorithmic Reasoning
Watch later
Offline Meta Reinforcement Learning with In-Distribution Online Adaptation
Watch later
Jianhao Wang, …
ContraBAR: Contrastive Bayes-Adaptive Deep RL
Watch later
Era Choshen, …
An Information-Theoretic Analysis of Nonstationary Bandit Learning
Watch later
Seungki Min, …
