Pre-training for Speech Translation: CTC Meets Optimal Transport
Jul 24, 2023
Sprecher:innen
Phuong-Hang Le
Sprecher:in · 0 Follower:innen
Hongyu Gong
Sprecher:in · 0 Follower:innen
Changhan Wang
Sprecher:in · 0 Follower:innen
Benjamin Lecouteux
Sprecher:in · 0 Follower:innen
Didier Schwab
Sprecher:in · 0 Follower:innen
Über
The gap between speech and text modalities is a major challenge in speech-to-text translation (ST). Different methods have been proposed for reducing this gap, but most of them require architectural changes in ST training. In this work, we propose to mitigate this issue at the pre-training stage, requiring no change in the ST model. First, we show that the connectionist temporal classification (CTC) loss can reduce the modality gap by design. We provide a quantitative comparison with the more common cross-entropy loss, showing that pre-training with CTC consistently achieves better final ST accuracy. Nevertheless, CTC is only a partial solution and thus, in our second contribution, we propose a novel pre-training method combining CTC and optimal transport to further reduce this gap. Our method pre-trains a Siamese-like model composed of two encoders, one for acoustic inputs and the other for textual inputs, such that they produce representations that are close to each other in the Wasserstein space. Extensive experiments on the standard CoVoST-2 and MuST-C datasets show that our pre-training method applied to vanilla encoder-decoder Transformer achieves state-of-the-art performance under the no-external data setting, and performs on par with recent strong multi-task learning systems trained with external data. Finally, our method can also be applied on top of these multi-task systems, leading to further improvements for these models.The gap between speech and text modalities is a major challenge in speech-to-text translation (ST). Different methods have been proposed for reducing this gap, but most of them require architectural changes in ST training. In this work, we propose to mitigate this issue at the pre-training stage, requiring no change in the ST model. First, we show that the connectionist temporal classification (CTC) loss can reduce the modality gap by design. We provide a quantitative comparison with the more co…
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
Rockmate: an Efficient, Fast, Automatic and Generic Tool for Re-materialization in PyTorch
Später ansehen
Xunyi Zhao, …
OCD: Learning to Overfit with Conditional Diffusion Models
Später ansehen
Titanium 3D Microstructure for Physics-based Generative Models: A Dataset and Primer
Später ansehen
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
Später ansehen
Youwei Liang, …
Finding Generalization Measures by Contrasting Signal and Noise
Später ansehen
Jiaye Teng, …
