Why Spectral Normalization Stabilizes GANs: Analysis and Improvements
Dec 6, 2021
Speakers
Vyas Sekar
Speaker · 0 followers
Giulia Fanti
Speaker · 0 followers
About
Spectral normalization (SN) is a widely-used technique for improving the stability and sample quality of Generative Adversarial Networks (GANs). However, current understanding of SN's efficacy is limited. In this work, we show that SN controls two important failure modes of GAN training: exploding and vanishing gradients. Our proofs illustrate a (perhaps unintentional) connection with the successful LeCun initialization. This connection helps to explain why the most popular implementation of SN for GANs requires no hyper-parameter tuning, whereas stricter implementations of SN have poor empirical performance out-of-the-box. Unlike LeCun initialization which only controls gradient vanishing at the beginning of training, SN preserves this property throughout training. Building on this theoretical understanding, we propose a new spectral normalization technique: Bidirectional Scaled Spectral Normalization (BSSN), which incorporates insights from later improvements to LeCun initialization: Xavier initialization and Kaiming initialization. Theoretically, we show that BSSN gives better gradient control than SN. Empirically, we demonstrate that it outperforms SN in sample quality and training stability on several benchmark datasets.Spectral normalization (SN) is a widely-used technique for improving the stability and sample quality of Generative Adversarial Networks (GANs). However, current understanding of SN's efficacy is limited. In this work, we show that SN controls two important failure modes of GAN training: exploding and vanishing gradients. Our proofs illustrate a (perhaps unintentional) connection with the successful LeCun initialization. This connection helps to explain why the most popular implementation of SN…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Privately Learning Subspaces
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A generative nonparametric Bayesian model for whole genomes
Watch later
Alan N. Amin, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection
Watch later
Yuxin Fang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Real-Time Optimization for Fast and Complex Control Systems
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Oracle Complexity in Nonsmooth Nonconvex Optimization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Real-time Generalized Sensor Fusion with Transformers
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%