ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
Dez 6, 2021
Sprecher:innen
Qiming Zhang
Řečník · 0 sledujících
Jing Zhang
Řečník · 0 sledujících
Dacheng Tao
Řečník · 1 sledující
Über
Transformers have shown great potential in various computer vision tasks owing to their strong capability in modeling long-range dependency using the self-attention mechanism. Nevertheless, vision transformers treat an image as 1D sequences of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance. Alternatively, they require large-scale training data and longer training schedules to learn the IB implicitly. In this paper, we propose a novel Vision Transformer Advanced by Exploring intrinsic IB from convolutions, , ViTAE. Technically, ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context by using multiple convolutions with different dilation rates. In this way, it acquires an intrinsic scale invariance IB and is able to learn robust feature representation for objects at various scales. Moreover, in each transformer layer, ViTAE has a convolution block in parallel to the multi-head self-attention module, whose features are fused and fed into the feed-forward network. Consequently, it has the intrinsic locality IB and is able to learn local features and global dependencies collaboratively. Experiments on ImageNet as well as downstream tasks prove the superiority of ViTAE over the baseline transformer and concurrent works. Source code and pretrained models will be made public.Transformers have shown great potential in various computer vision tasks owing to their strong capability in modeling long-range dependency using the self-attention mechanism. Nevertheless, vision transformers treat an image as 1D sequences of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance. Alternatively, they require large-scale training data and longer training schedules to learn the IB implicitly. In this paper, we p…
Organisator
NeurIPS 2021
Účet · 1,9k sledujících
Über NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
On the Power of Differentiable Learning versus PAC and SQ Learning
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
CROCS: Clustering and Retrieval of Cardiac Signals Based on Patient Disease Class, Sex, and Age
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Delayed Gradient Averaging: Tolerate the Communication Latency for Federated Learning
Zhlédnout později
Ligeng Zhu, …
Pro uložení prezentace do věčného trezoru hlasovalo 1 diváků, což je 0.1 %
Opening Remark
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Unsupervised Change Detection of Extreme Events Using ML On-Board
Zhlédnout později
Vit Ruzicka, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Online Robust Reinforcement Learning with Model Uncertainty
Zhlédnout později
Yue Wang, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %