Curvature-Corrected Learning Dynamics in Deep Neural Networks

Jul 12, 2020



Deep neural networks exhibit highly non-convex loss landscape, which results in complex learning dynamics under steepest gradient descent. Second order optimization methods, such as natural gradient descent, can facilitate learning by compensating for ill-conditioned curvature. However, the exact nature of such curvature-corrected learning process remains largely unknown. Here, we derive exact solutions to curvature-corrected learning rules for the restricted case of deep linear neural networks. Our analysis reveals that natural gradient descent follows the same path as gradient descent, only adjusting the temporal dynamics along the path. This preserves the implicit bias of gradient-based learning, such as weight balance across layers. However, block-diagonal approximations of natural gradient, which are widely used in most second order methods (e.g. K-FAC), significantly distort the dynamics to follow highly divergent paths, destroying weight balance across layers. We introduce partially curvature-corrected learning rule, which provides most of the benefit of full curvature correction in terms of convergence speed with superior numerical stability while preserving the core property of gradient descent under block-diagonal approximations.



About ICML 2020

The International Conference on Machine Learning (ICML) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence known as machine learning. ICML is globally renowned for presenting and publishing cutting-edge research on all aspects of machine learning used in closely related areas like artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, and robotics. ICML is one of the fastest growing artificial intelligence conferences in the world. Participants at ICML span a wide range of backgrounds, from academic and industrial researchers, to entrepreneurs and engineers, to graduate students and postdocs.

Store presentation

Should this presentation be stored for 1000 years?

How do we store presentations

Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%


Recommended Videos

Presentations on similar topic, category or speaker