Jul 12, 2020
Deep neural networks exhibit highly non-convex loss landscape, which results in complex learning dynamics under steepest gradient descent. Second order optimization methods, such as natural gradient descent, can facilitate learning by compensating for ill-conditioned curvature. However, the exact nature of such curvature-corrected learning process remains largely unknown. Here, we derive exact solutions to curvature-corrected learning rules for the restricted case of deep linear neural networks. Our analysis reveals that natural gradient descent follows the same path as gradient descent, only adjusting the temporal dynamics along the path. This preserves the implicit bias of gradient-based learning, such as weight balance across layers. However, block-diagonal approximations of natural gradient, which are widely used in most second order methods (e.g. K-FAC), significantly distort the dynamics to follow highly divergent paths, destroying weight balance across layers. We introduce partially curvature-corrected learning rule, which provides most of the benefit of full curvature correction in terms of convergence speed with superior numerical stability while preserving the core property of gradient descent under block-diagonal approximations.
The International Conference on Machine Learning (ICML) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence known as machine learning. ICML is globally renowned for presenting and publishing cutting-edge research on all aspects of machine learning used in closely related areas like artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, and robotics. ICML is one of the fastest growing artificial intelligence conferences in the world. Participants at ICML span a wide range of backgrounds, from academic and industrial researchers, to entrepreneurs and engineers, to graduate students and postdocs.
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Presentations on similar topic, category or speaker