Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing
Nov 28, 2022
Speakers
Yonggan Fu
Speaker · 0 followers
Yang Zhang
Speaker · 1 follower
Kaizhi Qian
Speaker · 0 followers
Zhongzhi Yu
Speaker · 0 followers
Cheng-I Lai
Speaker · 0 followers
Yingyan Lin
Speaker · 0 followers
About
Self-supervised learning (SSL) for rich speech representations has achieved empirical success in low-resource Automatic Speech Recognition (ASR) and other speech processing tasks, which can mitigate the necessity of a large amount of transcribed speech and thus has driven a growing demand for on-device ASR and other speech processing. However, advanced speech SSL models have become increasingly large, which contradicts the limited on-device resources. This gap could be more severe in multilingual/multitask scenarios requiring simultaneously recognizing multiple languages or executing multiple speech processing tasks. Additionally, strongly overparameterized speech SSL models tend to suffer from overfitting when being finetuned on low-resource speech corpus. This work aims to enhance the practical usage of speech SSL models towards a win-win in both enhanced efficiency and alleviated overfitting via our proposed S^3-Router framework, which for the first time discovers that simply discarding no more than 10% of model weights via only finetuning model connections of speech SSL models can achieve better accuracy over standard weight finetuning on downstream speech processing tasks. More importantly, S^3-Router can serve as an all-in-one technique to enable (1) a new finetuning scheme, (2) an efficient multilingual/multitask solution, (3) a state-of-the-art pruning technique, and (4) a new tool to quantitatively analyze the learned speech representation. All codes will be released upon acceptance.Self-supervised learning (SSL) for rich speech representations has achieved empirical success in low-resource Automatic Speech Recognition (ASR) and other speech processing tasks, which can mitigate the necessity of a large amount of transcribed speech and thus has driven a growing demand for on-device ASR and other speech processing. However, advanced speech SSL models have become increasingly large, which contradicts the limited on-device resources. This gap could be more severe in multilingua…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Luckiness in Multiscale Online Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Multi-agent Dynamic Algorithm Configuration
Watch later
Ke Xue, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Graph Neural Networks are Dynamic Programmers
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Neural-Symbolic Entangled Framework for Complex Query Answering
Watch later
Zezhong Xu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On the Robustness of Graph Neural Diffusion to Topology Perturbations
Watch later
Yang Song, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning Generalized Policy Automata for Relational Stochastic Shortest Path Problems
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%