Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation
28. listopadu 2022
Řečníci
Chiyuan Zhang
Řečník · 0 sledujících
Balaji Lakshminarayanan
Řečník · 5 sledujících
Alex Beutel
Řečník · 0 sledujících
Xuezhi Wang
Řečník · 0 sledujících
O prezentaci
We investigate the robustness of vision transformers (ViTs) through the lens of their special patch-based architectural structure, i.e., they process an image as a sequence of image patches. We find that ViTs are surprisingly insensitive to patch-based transformations, even when the transformation largely destroys the original semantics and makes the image unrecognizable by humans. This indicates that ViTs heavily use features that survived such transformations but are generally not indicative of the semantic class to humans. Further investigations show that these features are useful but non-robust, as ViTs trained on them can achieve high in-distribution accuracy, but break down under distribution shifts. From this understanding, we ask: can training the model to rely less on these features improve ViT robustness and out-of-distribution performance? We use the images transformed with our patch-based operations as negatively augmented views and offer losses to regularize the training away from using non-robust features. This is a complementary view to existing research that mostly focuses on augmenting inputs with semantic-preserving transformations to enforce models' invariance. We show that patch-based negative augmentation consistently improves robustness of ViTs on ImageNet based robustness benchmarks across 20+ different experimental settings. Furthermore, we find our patch-based negative augmentation are complementary to traditional (positive) data augmentation techniques and batch-based negative examples in contrastive learning. All the code will be open-sourced.We investigate the robustness of vision transformers (ViTs) through the lens of their special patch-based architectural structure, i.e., they process an image as a sequence of image patches. We find that ViTs are surprisingly insensitive to patch-based transformations, even when the transformation largely destroys the original semantics and makes the image unrecognizable by humans. This indicates that ViTs heavily use features that survived such transformations but are generally not indicative o…
Organizátor
NeurIPS 2022
Účet · 961 sledujících
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Resource-Adaptive Federated Learning with All-In-One Neural Composition
Zhlédnout později
Yiqun Mei, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Re-QGAN: an optimized adversarial quantum circuit learning framework
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Causal Analysis of the TOPCAT Trial: Spironolactone for Preserved Cardiac Function Heart Failure
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 1 diváků, což je 0.1 %
Scale-invariant Learning by Physics Inversion
Zhlédnout později
Philipp Holl, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Myriad: a real-world testbed to bridge trajectory optimization and deep learning
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Homomorphic Matrix Completion
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %