Understanding and Improving Lexical Choice in Non-Autoregressive Translation
3. května 2021
Řečníci
Liang Ding
Řečník · 1 sledující
Longyue Wang
Řečník · 1 sledující
Derek F. Wong
Řečník · 0 sledujících
Dacheng Tao
Řečník · 1 sledující
Zhaopeng Tu
Řečník · 2 sledující
O prezentaci
Knowledge distillation (KD) is essential for training non-autoregressive translation (NAT) models by reducing the complexity of the raw data with an autoregressive teacher model. In this study, we empirically show that as a side effect of this training, the lexical choice errors on low-frequency words are propagated to the NAT model from the teacher model. To alleviate this problem, we propose to expose the raw data to NAT models to restore the useful information of low-frequency words, which are missed in the distilled data. To this end, we introduce an extra Kullback-Leibler divergence term derived by comparing the lexical choice of NAT model and that embedded in the raw data. Experimental results across language pairs and model architectures demonstrate the effectiveness and universality of the proposed approach. Extensive analyses confirm our claim that our approach improves performance by reducing the lexical choice errors on low-frequency words. Encouragingly, our approach pushes the SOTA NAT performance on the WMT14 English-German and WMT16 Romanian-English datasets up to 27.8 and 33.8 BLEU points, respectively. The codes and models will be released.Knowledge distillation (KD) is essential for training non-autoregressive translation (NAT) models by reducing the complexity of the raw data with an autoregressive teacher model. In this study, we empirically show that as a side effect of this training, the lexical choice errors on low-frequency words are propagated to the NAT model from the teacher model. To alleviate this problem, we propose to expose the raw data to NAT models to restore the useful information of low-frequency words, which ar…
O organizátorovi (ICLR 2021)
The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning. ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
Zhlédnout později
Junwen Bai, …
Learning Generalizable Visual Representations via Interactive Gameplay
Zhlédnout později
Luca Weihs, …
Learning to Represent Action Values as a Hypergraph on the Action Vertices
Zhlédnout později
Invited Talk - Alyosha Efros
Zhlédnout později
MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering
Zhlédnout později
