Long-Short Transformer: Efficient Transformers for Language and Vision
Dec 6, 2021
Speakers
Chaowei Xiao
Speaker · 0 followers
Mohammad Shoeybi
Speaker · 0 followers
Tom Goldstein
Speaker · 0 followers
Anima Anandkumar
Speaker · 1 follower
Bryan Catanzaro
Speaker · 1 follower
About
Transformers have achieved success in both language and vision domains. However, it is prohibitively expensive to scale them to long sequences such as long documents or high-resolution images, because self-attention mechanism has quadratic compute time and memory complexities with respect to the input sequence length. In this paper, we propose Long-Short Transformer (Transformer-LS), an efficient transformer architecture for modeling long sequences with linear complexity for both language and vision. It aggregates a novel long-range attention with dynamic projection to model distant correlations and a short-term attention to capture fine-grained local correlations. We propose a dual normalization strategy to account for the scale mismatch between the two attention mechanisms. Long-Short Transformer can be applied to both autoregressive and bidirectional models without additional complexity. Our method outperforms the state-of-the-art models on multiple tasks in language and vision domains, including the Long Range Arena benchmark, autoregressive language modeling, and ImageNet classification. For instance, Transformer-LS achieves 0.97 test BPC on enwik8 using half the number of parameters than previous state-of-the-art method, while being faster and is able to handle 3x as long sequences compared to its full-attention version on the same hardware. On ImageNet, it improves the accuracies of a state-of-the-art model with full attention, while being more scalable to high-resolution images.Transformers have achieved success in both language and vision domains. However, it is prohibitively expensive to scale them to long sequences such as long documents or high-resolution images, because self-attention mechanism has quadratic compute time and memory complexities with respect to the input sequence length. In this paper, we propose Long-Short Transformer (Transformer-LS), an efficient transformer architecture for modeling long sequences with linear complexity for both language and vi…
Organizer
NeurIPS 2021
Account · 1.9k followers
Categories
AI & Data Science
Category · 10.8k presentations
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies
Watch later
Tim Seyde, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
From Optimality to Robustness: Adaptive Re-Sampling Strategies in Stochastic Bandits
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Unifying Width-Reduced Methods for Quasi-Self-Concordant Optimization
Watch later
Deeksha Adil, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Pointwise Bounds for Distribution Estimation under Communication Constraints
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Density-aware Chamfer Distance as a Comprehensive Metric for Point Cloud Completion
Watch later
Tong Wu, …
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%