R-Drop: Regularized Dropout for Neural Networks
Dec 6, 2021
Speakers
Xiaobo Liang
Speaker · 0 followers
Tie-Yan Liu
Speaker · 1 follower
About
Dropout is a powerful and widely used technique to regularize the training of deep neural networks. In this paper, we introduce a simple regularization strategy upon dropout in model training, namely R-Drop, which forces the output distributions of different sub models generated by dropout to be consistent with each other. Specifically, for each training sample, R-Drop minimizes the bidirectional KL-divergence between the output distributions of two sub models sampled by dropout. Theoretical analysis reveals that R-Drop reduces the freedom of the model parameters and complements dropout. Experiments on 5 widely used deep learning tasks (18 datasets in total), including neural machine translation, abstractive summarization, language understanding, language modeling, and image classification, show that R-Drop is universally effective. In particular, it yields substantial improvements when applied to fine-tune large-scale pre-trained models, e.g., ViT, RoBERTa-large, and BART, and achieves state-of-the-art (SOTA) performances with the vanilla Transformer model on WMT14 English→German translation (30.91 BLEU) and WMT14 English→French translation (43.95 BLEU), even surpassing models trained with extra large-scale data and expert-designed advanced variants of Transformer models. Our code is available at GitHub[<https://github.com/dropreg/R-Drop>].Dropout is a powerful and widely used technique to regularize the training of deep neural networks. In this paper, we introduce a simple regularization strategy upon dropout in model training, namely R-Drop, which forces the output distributions of different sub models generated by dropout to be consistent with each other. Specifically, for each training sample, R-Drop minimizes the bidirectional KL-divergence between the output distributions of two sub models sampled by dropout. Theoretical an…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
Watch later
Ming Yin, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Influence Patterns for Explaining Information Flow in BERT
Watch later
Caleb Lu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning latent causal graphs via mixture oracles
Watch later
Bohdan Kivva, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
How Should a Machine Learning Researcher Think About AI Ethics?
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On UMAP's True Loss Function
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Complete Axiomatization of Forward Differentiation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%