A Win-win Deal: Towards Sparse and Robust Pre-trained Language Models
Dec 6, 2022
Speakers
Yuanxin Liu
Speaker · 0 followers
Fandong Meng
Speaker · 0 followers
Jiangnan Li
Speaker · 0 followers
Weiping Wang
Speaker · 0 followers
About
Despite the remarkable success of pre-trained language models (PLMs), they still face two challenges: First, large-scale PLMs are inefficient in terms of memory footprint and computation. Second, on the downstream tasks, PLMs tend to rely on the dataset bias and struggle to generalize to out-of-distribution (OOD) data. In response to the efficiency problem, recent studies show that dense PLMs can be replaced with sparse subnetworks without hurting the performance. Such subnetworks can be found in three scenarios: 1) the fine-tuned PLMs, 2) the raw PLMs and then fine-tuned in isolation, and even inside 3) PLMs without any parameter fine-tuning. However, these results are only obtained in the in-distribution (ID) setting. In this paper, we extend the study on PLMs subnetworks to the OOD setting, investigating whether sparsity and robustness to dataset bias can be achieved simultaneously. To this end, we conduct extensive experiments with the pre-trained BERT model on three natural language understanding (NLU) tasks. Our results demonstrate that sparse and robust subnetworks (SRNets) can consistently be found in BERT, across the aforementioned three scenarios, using different training and compression methods. Furthermore, we explore the upper bound of SRNets using the OOD information and show that there exist sparse and almost unbiased BERT subnetworks. Finally, we refine the SRNets searching process in terms of efficiency and performance, which involves: 1) the appropriate timing to start searching SRNets during full BERT fine-tuning, and 2) how to identify SRNets at high sparsity. Our codes will be released on publication.Despite the remarkable success of pre-trained language models (PLMs), they still face two challenges: First, large-scale PLMs are inefficient in terms of memory footprint and computation. Second, on the downstream tasks, PLMs tend to rely on the dataset bias and struggle to generalize to out-of-distribution (OOD) data. In response to the efficiency problem, recent studies show that dense PLMs can be replaced with sparse subnetworks without hurting the performance. Such subnetworks can be found i…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Data Science & Inequalities
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
![[Re] Background-Aware Pooling and Noise-Aware Loss for Weakly-Supervised Semantic Segmentation](https://ma.slideslive.com/library/presentations/38992711/thumbnail/re-backgroundaware-pooling-and-noiseaware-loss-for-weaklysupervised-semantic-segmentation_EdaI55_small.jpg)
[Re] Background-Aware Pooling and Noise-Aware Loss for Weakly-Supervised Semantic Segmentation
Watch later
Aryan Mehta, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video
Watch later
Tianhao Wu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Grow and Merge: A Unified Framework for Continuous Categories Discovery
Watch later
Xinwei Zhang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Locally Hierarchical Auto-Regressive Modeling for Image Generation
Watch later
Tackgeun You, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Optimal Positive Generation via Latent Transformation for Contrastive Learning
Watch later
Yinqi Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%