Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning
3. května 2021
Řečníci
Longyue Wang
Řečník · 1 sledující
Derek Wong
Řečník · 0 sledujících
Liang Ding
Řečník · 1 sledující
Lidia Chao
Řečník · 0 sledujících
Zhaopeng Tu
Řečník · 2 sledující
O prezentaci
Encoder layer fusion (EncoderFusion) is a technique to fuse all the encoder layers (instead of the uppermost layer) for sequence-to-sequence (Seq2Seq) models, which has proven effective on various NLP tasks. However, it is still not entirely clear why and when EncoderFusion should work. In this paper, our main contribution is to take a step further in understanding EncoderFusion. Many of previous studies believe that the success of EncoderFusion comes from exploiting surface and syntactic information embedded in lower encoder layers. Unlike them, we find that the encoder embedding layer is more important than other intermediate encoder layers. In addition, the uppermost decoder layer consistently pays more attention to the encoder embedding layer across NLP tasks. Based on this observation, we propose a simple fusion method, SurfaceFusion, by fusing only the encoder embedding layer for the softmax layer. Experimental results show that SurfaceFusion outperforms EncoderFusion on several NLP benchmarks, including machine translation, text summarization, and grammatical error correction. It obtains the state-of-the-art performance on WMT16 Romanian-English and WMT14 English-French translation tasks. Extensive analyses reveal that SurfaceFusion learns more expressive bilingual word embeddings by building a closer relationship between relevant source and target embeddings. The source code will be released.Encoder layer fusion (EncoderFusion) is a technique to fuse all the encoder layers (instead of the uppermost layer) for sequence-to-sequence (Seq2Seq) models, which has proven effective on various NLP tasks. However, it is still not entirely clear why and when EncoderFusion should work. In this paper, our main contribution is to take a step further in understanding EncoderFusion. Many of previous studies believe that the success of EncoderFusion comes from exploiting surface and syntactic inform…
O organizátorovi (ICLR 2021)
The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning. ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Bridging the Gap Between Adversarial Robustness and Optimization Bias
Zhlédnout později
Computational Separation Between CNNs and FCNs
Zhlédnout později
Eran Malach, …
Geometric and Topological Representation Learning
Zhlédnout později
Fast Geometric Projections for Local Robustness Certification
Zhlédnout později
Ensembles of GANs for synthetic training data generation
Zhlédnout později
Using Latent Space Regression to Analyze and Leverage Compositionality in GANs
Zhlédnout později
Lucy Chai, …