On the Maximum Hessian Eigenvalue and Generalization
Dec 2, 2022
Speakers
Simran Kaur
Speaker · 0 followers
Jeremy Cohen
Speaker · 0 followers
Zachary Lipton
Speaker · 0 followers
About
The mechanisms by which certain training interventions, such as increasing learning rates and applying batch normalization, improve the generalization of deep networks remains a mystery. Prior works have speculated that "flatter" solutions generalize better than "sharper" solutions to unseen data, motivating several metrics for measuring flatness (particularly λmax, the largest eigenvalue of the Hessian of the loss); and algorithms, such as Sharpness-Aware Minimization (SAM), that directly optimize for flatness. Other works question the link between λmax and generalization. In this paper, we present findings that call λmax's influence on generalization further into question. We show that: (1) while larger learning rates reduce λmax for all batch sizes, generalization benefits sometimes vanish at larger batch sizes; (2) by scaling batch size and learning rate simultaneously, we can change λmax without affecting generalization; (3) while SAM produces smaller λmax for all batch sizes, generalization benefits (also) vanish with larger batch sizes; (4) for dropout, excessively high dropout probabilities can degrade generalization, even as they promote smaller λmax; and (5) while batch-normalization does not consistently produce smaller λmax, it nevertheless confers generalization benefits. While our experiments affirm the generalization benefits of large learning rates and SAM for minibatch SGD, the GD-SGD discrepancy demonstrates limits to λmax's ability to explain generalization in neural networks.The mechanisms by which certain training interventions, such as increasing learning rates and applying batch normalization, improve the generalization of deep networks remains a mystery. Prior works have speculated that "flatter" solutions generalize better than "sharper" solutions to unseen data, motivating several metrics for measuring flatness (particularly λmax, the largest eigenvalue of the Hessian of the loss); and algorithms, such as Sharpness-Aware Minimization (SAM), that directly optim…
Organizer
NeurIPS 2022
Account · 962 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Power Grid Congestion Management via Topology Optimization with AlphaZero
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Selection of XAI Methods Matters: Evaluation of Feature Attribution Methods for Oculomotoric Biometric Identification
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Physics-Informed Implicit Representations of Equilibrium Network Flows
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Aligning Robot Representations with Humans
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Deconstructing Distributions: A Pointwise Framework of Learning
Watch later
Gal Kaplun, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Convolutional neural networks-based 3D reconstruction from 2D medical images guided by generative models
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%