Predicting Deep Neural Network Generalization with Perturbation Response Curves
Dec 6, 2021
Speakers
Yair Schiff
Speaker · 0 followers
Brian Quanz
Speaker · 0 followers
Pin-Yu Chen
Speaker · 3 followers
About
The field of Deep Learning is rich with empirical evidence of human-like performance on a variety of prediction tasks. However, despite these successes, the recent Predicting Generalization in Deep Learning (PGDL) NeurIPS 2020 competition suggests that there is a need for more robust and efficient measures of network generalization. In this work, we propose a new framework for evaluating the generalization capabilities of trained networks. We use perturbation response (PR) curves that capture the accuracy change of a given network as a function of varying levels of training sample perturbation. From these PR curves, we derive novel statistics that capture generalization capability. Specifically, we introduce two new measures for accurately predicting generalization gaps: the Gi-score and Pal-score, that are inspired by the Gini coefficient and Palma ratio (measures of income inequality), that accurately predict generalization gaps. Using our framework applied to intra and inter class sample mixup, we attain better predictive scores than the current state-of-the-art measures on a majority of tasks in the PGDL competition. In addition, we show that our framework and the proposed statistics can be used to capture to what extent a trained network is invariant to a given parametric input transformation, such as rotation or translation. Therefore, these generalization gap prediction statistics also provide a useful means for selecting the optimal network architectures and hyperparameters that are invariant to a certain perturbation.The field of Deep Learning is rich with empirical evidence of human-like performance on a variety of prediction tasks. However, despite these successes, the recent Predicting Generalization in Deep Learning (PGDL) NeurIPS 2020 competition suggests that there is a need for more robust and efficient measures of network generalization. In this work, we propose a new framework for evaluating the generalization capabilities of trained networks. We use perturbation response (PR) curves that capture th…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Mixture Proportion Estimation and PU Learning: A Modern Approach
Watch later
Saurabh Garg, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Planning from Pixels in Environments with Combinatorially Hard Search Spaces
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
OpenML Benchmarking Suites
Watch later
Bernd Bischl, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Privacy-Aware Rejection Sampling
Watch later
Jordan Awan, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Modeling Worlds in Text
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Spotlight Introduction 2
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%