Understanding and Improving Early Stopping for Learning with Noisy Labels
Dec 6, 2021
Speakers
Yingbin Bai
Speaker · 0 followers
Erkun Yang
Speaker · 0 followers
Yanhua Yang
Speaker · 0 followers
Jiatong Li
Speaker · 0 followers
Yinian Mao
Speaker · 0 followers
Tongliang Liu
Speaker · 0 followers
About
The memorization effect of deep neural network (DNN) plays a pivotal role in many state-of-the-art label-noise learning methods. To exploit this property, the early stopping trick, which stops the optimization at the early stage of training, is usually adopted. Current methods generally decide the early stopping point by considering a DNN as a whole. However, a DNN can be considered as a composition of a series of layers, and we find that the latter layers in a DNN are much more sensitive to label noise, while their former counterparts are quite robust. Therefore, selecting a stopping point for the whole network may make different DNN layers antagonistically affected each other, thus degrading the final performance. In this paper, we propose to separate a DNN into different parts and progressively train them to address this problem. Instead of the early stopping, which trains a whole DNN all at once, we initially train former DNN layers by optimizing the DNN with a relatively large number of epochs. During training, we progressively train the latter DNN layers by using a smaller number of epochs with the preceding layers fixed to counteract the impact of noisy labels. We term the proposed method as progressive early stopping (PES). Despite its simplicity, compared with the early stopping, PES can help to obtain more promising and stable results. Furthermore, by combining PES with existing approaches on noisy label training, we achieve state-of-the-art performance on image classification benchmarks.The memorization effect of deep neural network (DNN) plays a pivotal role in many state-of-the-art label-noise learning methods. To exploit this property, the early stopping trick, which stops the optimization at the early stage of training, is usually adopted. Current methods generally decide the early stopping point by considering a DNN as a whole. However, a DNN can be considered as a composition of a series of layers, and we find that the latter layers in a DNN are much more sensitive to lab…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Deep generative models of protein structures create new and diverse proteins
Watch later
Zeming Lin, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Exploiting Local Convergence of Quasi-Newton Methods Globally: Adaptive Sample Size Approach
Watch later
Qiujiang Jin, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning after Deployment: The Missed Tale of Supervision
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Emergent Communication Under Varying Sizes and Connectivities
Watch later
Jooyeon Kim, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
MADE: Exploration via Maximizing Deviation from Explored Regions
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
BARTScore: Evaluating Generated Text as Text Generation
Watch later
Weizhe Yuan, …
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%