ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
Dec 6, 2021
Speakers
Qiming Zhang
Speaker · 0 followers
Jing Zhang
Speaker · 0 followers
Dacheng Tao
Speaker · 1 follower
About
Transformers have shown great potential in various computer vision tasks owing to their strong capability in modeling long-range dependency using the self-attention mechanism. Nevertheless, vision transformers treat an image as 1D sequences of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance. Alternatively, they require large-scale training data and longer training schedules to learn the IB implicitly. In this paper, we propose a novel Vision Transformer Advanced by Exploring intrinsic IB from convolutions, , ViTAE. Technically, ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context by using multiple convolutions with different dilation rates. In this way, it acquires an intrinsic scale invariance IB and is able to learn robust feature representation for objects at various scales. Moreover, in each transformer layer, ViTAE has a convolution block in parallel to the multi-head self-attention module, whose features are fused and fed into the feed-forward network. Consequently, it has the intrinsic locality IB and is able to learn local features and global dependencies collaboratively. Experiments on ImageNet as well as downstream tasks prove the superiority of ViTAE over the baseline transformer and concurrent works. Source code and pretrained models will be made public.Transformers have shown great potential in various computer vision tasks owing to their strong capability in modeling long-range dependency using the self-attention mechanism. Nevertheless, vision transformers treat an image as 1D sequences of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance. Alternatively, they require large-scale training data and longer training schedules to learn the IB implicitly. In this paper, we p…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Decreasing Annotation Burden of Pairwise Comparisons with Human-in-the-Loop Sorting: Application in Medical Image Artifact Rating
Watch later
Ikbeom Jang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Frank Wolfe in Probability Space
Watch later
Carson Kent, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Closing Remarks: eXplainable Al approchaes for debugging and diagnosis
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Large Batch Active Learning
Watch later
Gui Citovsky, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
WHY-21: Causal Inference & Machine Learning: Why now?
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Independent mechanism analysis, a new concept?
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%