Transformer in Transformer
Dec 6, 2021
Speakers
Jianyuan Guo
Speaker · 0 followers
Chunjing Xu
Speaker · 0 followers
Yunhe Wang
Speaker · 0 followers
About
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (, 16×16) as “visual sentences” and present to further divide them into smaller patches (, 4×4) as “visual words”. The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, , we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost.Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. I…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
The Semi-Random Satisfaction of Voting Axioms
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The global optimization of functions with low effective dimension - better than worst-case?
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Methods: Self-prediction and Contrastive learning
Watch later
Total of 2 viewers voted for saving the presentation to eternal vault which is 0.2%
Learning Model Predictive Control for Iterative Tasks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Estimating the Unique Information of Continuous Variables
Watch later
Ari Pakman, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Intelligent Sight and Sound (ISS): A Chronic Cancer Facial Pain Dataset
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%