DICE: Diversity In Deep Ensembles Via Conditional Redundancy Adversarial Estimation
May 3, 2021
Speakers
Alexandre Ramé
Sprecher:in · 0 Follower:innen
Matthieu Cord
Sprecher:in · 0 Follower:innen
About
Deep ensembles perform better than a single network thanks to the diversity among their members. Recent approaches regularize predictions to increase diversity; however, they also drastically decrease individual members’ performances. In this paper, we argue that learning strategies for deep ensembles need to tackle the trade-off between ensemble diversity and individual accuracies. Motivated by arguments from information theory and leveraging recent advances in neural estimation of conditional mutual information, we introduce a novel training criterion called DICE: it increases diversity by reducing spurious correlations among features. The main idea is that features extracted from pairs of members should only share information useful for target class prediction without being conditionally redundant. Therefore, besides the classification loss with information bottleneck, we adversarially prevent features from being conditionally predictable from each other. We manage to reduce simultaneous errors while protecting class information. We obtain state-of-the-art accuracy results on CIFAR-10/100: for example, an ensemble of 5 networks trained with DICE matches an ensemble of 7 networks trained independently. We further analyze the consequences on calibration, uncertainty estimation, out-of-distribution detection and online co-distillation.Deep ensembles perform better than a single network thanks to the diversity among their members. Recent approaches regularize predictions to increase diversity; however, they also drastically decrease individual members’ performances. In this paper, we argue that learning strategies for deep ensembles need to tackle the trade-off between ensemble diversity and individual accuracies. Motivated by arguments from information theory and leveraging recent advances in neural estimation of conditional…
About ICLR 2021
The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning. ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
On Position Embeddings in BERT
Später ansehen
Benyou Wang, …
A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
Später ansehen
Combining Label Propagation and Simple Models Out-Performs Graph Neural Networks
Später ansehen
Qian Huang, …
Efficient Click-Through Rate Prediction for Developing Countries via Tabular Learning
Später ansehen
Joonyoung Yi, …
Boosting Classification Accuracy of Fertile Sperm Cell Images leveraging cDCGAN
Später ansehen
Dipam Paul, …
