A Win-win Deal: Towards Sparse and Robust Pre-trained Language Models
Dec 6, 2022
Speakers
Yuanxin Liu
Speaker · 0 followers
Fandong Meng
Speaker · 0 followers
Jiangnan Li
Speaker · 0 followers
Weiping Wang
Speaker · 0 followers
About
Despite the remarkable success of pre-trained language models (PLMs), they still face two challenges: First, large-scale PLMs are inefficient in terms of memory footprint and computation. Second, on the downstream tasks, PLMs tend to rely on the dataset bias and struggle to generalize to out-of-distribution (OOD) data. In response to the efficiency problem, recent studies show that dense PLMs can be replaced with sparse subnetworks without hurting the performance. Such subnetworks can be found in three scenarios: 1) the fine-tuned PLMs, 2) the raw PLMs and then fine-tuned in isolation, and even inside 3) PLMs without any parameter fine-tuning. However, these results are only obtained in the in-distribution (ID) setting. In this paper, we extend the study on PLMs subnetworks to the OOD setting, investigating whether sparsity and robustness to dataset bias can be achieved simultaneously. To this end, we conduct extensive experiments with the pre-trained BERT model on three natural language understanding (NLU) tasks. Our results demonstrate that sparse and robust subnetworks (SRNets) can consistently be found in BERT, across the aforementioned three scenarios, using different training and compression methods. Furthermore, we explore the upper bound of SRNets using the OOD information and show that there exist sparse and almost unbiased BERT subnetworks. Finally, we refine the SRNets searching process in terms of efficiency and performance, which involves: 1) the appropriate timing to start searching SRNets during full BERT fine-tuning, and 2) how to identify SRNets at high sparsity. Our codes will be released on publication.Despite the remarkable success of pre-trained language models (PLMs), they still face two challenges: First, large-scale PLMs are inefficient in terms of memory footprint and computation. Second, on the downstream tasks, PLMs tend to rely on the dataset bias and struggle to generalize to out-of-distribution (OOD) data. In response to the efficiency problem, recent studies show that dense PLMs can be replaced with sparse subnetworks without hurting the performance. Such subnetworks can be found i…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
A Rotated Hyperboilc Wrapped Normal Distribution for Hierarchical Representation Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Skill Acquisition by Instruction Augmentation on Offline Datasets
Watch later
Ted Xiao, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Trials of developing OPT-175B
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions
Watch later
Binghui Peng, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Control Graph as Unified IO for Morphology-Task Generalization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time
Watch later
Tailin Wu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%