Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning
Dec 6, 2021
Speakers
About
Continual learning learns a sequence of tasks incrementally with the goal of achieving two main objectives: overcoming catastrophic forgetting (CF) and encouraging knowledge transfer (KT) across tasks. However, most existing techniques focus only on overcoming CF and have no mechanism to encourage KT, and thus do not do well in KT. Although several papers claimed that they could deal with both CF and KT, our experiments show that they suffer from serious CF when the tasks do not have much shared knowledge. In this paper, we study the continual learning of a sequence of natural language processing (NLP) tasks. In NLP, fine-tuning a BERT-like language model using in-domain data is regarded as one of the most effective approaches. However, this approach suffers from serious CF for continual learning. In this paper, we present a novel model called AFK to solve these problems. Experimental results demonstrate the effectiveness of AFK.Continual learning learns a sequence of tasks incrementally with the goal of achieving two main objectives: overcoming catastrophic forgetting (CF) and encouraging knowledge transfer (KT) across tasks. However, most existing techniques focus only on overcoming CF and have no mechanism to encourage KT, and thus do not do well in KT. Although several papers claimed that they could deal with both CF and KT, our experiments show that they suffer from serious CF when the tasks do not have much shared…
Organizer
NeurIPS 2021
Account · 1.9k followers
Categories
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker