Blending Anti-Aliasing into Vision Transformer
Dec 6, 2021
Speakers
About
The transformer architectures, based on self-attention mechanism and convolution-free design, recently found superior performance and booming applications in computer vision. However, the discontinuous patch-wise tokenization process implicitly introduces jagged artifacts into attention maps, arising the traditional problem of aliasing for vision transformers. Aliasing effect occurs when discrete patterns are used to produce high frequency or continuous information, resulting in the indistinguishable distortions. Recent researches have found that modern convolution networks still suffer from this phenomenon. In this work, we analyze the uncharted problem of aliasing in vision transformer and explore to incorporate anti-aliasing properties. Specifically, we propose a plug-and-play Aliasing-Reduction Module (ARM) to alleviate the aforementioned issue. We investigate the effectiveness and generalization of the proposed method across multiple tasks and various vision transformer families. This lightweight design consistently attains a clear boost over several famous structures. Furthermore, our module also improves data efficiency and robustness of vision transformers.The transformer architectures, based on self-attention mechanism and convolution-free design, recently found superior performance and booming applications in computer vision. However, the discontinuous patch-wise tokenization process implicitly introduces jagged artifacts into attention maps, arising the traditional problem of aliasing for vision transformers. Aliasing effect occurs when discrete patterns are used to produce high frequency or continuous information, resulting in the indistinguis…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Training Neural Networks is ER-complete
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Counterfactual Invariance to Spurious Correlations
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Logarithmic Regret via Sublinear Hints
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
RedCaps: Web-curated image-text data created by the people, for the people
Watch later
Karan Desai, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Weight, Block or Unit? Exploring Sparsity Tradeoffs for Speech Enhancement on Tiny Neural Accelerators
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Extracting Deformation-Aware Local Features by Learning to Deform
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%