Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing
Nov 28, 2022
Speakers
Yonggan Fu
Speaker · 0 followers
Yang Zhang
Speaker · 1 follower
Kaizhi Qian
Speaker · 0 followers
Zhongzhi Yu
Speaker · 0 followers
Cheng-I Lai
Speaker · 0 followers
Yingyan Lin
Speaker · 0 followers
About
Self-supervised learning (SSL) for rich speech representations has achieved empirical success in low-resource Automatic Speech Recognition (ASR) and other speech processing tasks, which can mitigate the necessity of a large amount of transcribed speech and thus has driven a growing demand for on-device ASR and other speech processing. However, advanced speech SSL models have become increasingly large, which contradicts the limited on-device resources. This gap could be more severe in multilingual/multitask scenarios requiring simultaneously recognizing multiple languages or executing multiple speech processing tasks. Additionally, strongly overparameterized speech SSL models tend to suffer from overfitting when being finetuned on low-resource speech corpus. This work aims to enhance the practical usage of speech SSL models towards a win-win in both enhanced efficiency and alleviated overfitting via our proposed S^3-Router framework, which for the first time discovers that simply discarding no more than 10% of model weights via only finetuning model connections of speech SSL models can achieve better accuracy over standard weight finetuning on downstream speech processing tasks. More importantly, S^3-Router can serve as an all-in-one technique to enable (1) a new finetuning scheme, (2) an efficient multilingual/multitask solution, (3) a state-of-the-art pruning technique, and (4) a new tool to quantitatively analyze the learned speech representation. All codes will be released upon acceptance.Self-supervised learning (SSL) for rich speech representations has achieved empirical success in low-resource Automatic Speech Recognition (ASR) and other speech processing tasks, which can mitigate the necessity of a large amount of transcribed speech and thus has driven a growing demand for on-device ASR and other speech processing. However, advanced speech SSL models have become increasingly large, which contradicts the limited on-device resources. This gap could be more severe in multilingua…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Provable General Function Class Representation Learning in Multitask Bandits and MDP
Watch later
Rui Lu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Imputation and forecasting for Multi-Output Gaussian Processes in Smart Grid
Watch later
Jiangjiao Xu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Multi-skill Mobile Manipulation for Object Rearrangement
Watch later
Jiayuan Gu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Hyperbolic Embedding Inference for Structured Multi-Label Prediction
Watch later
Bo Xiong, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Indigenous in AI: Lightning Round
Watch later
Caleb Moses, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%