Optimizing Relevance Maps of Vision Transformers Improves Robustness
Nov 28, 2022
Speakers
Hila Chefer
Speaker · 0 followers
Idan Schwartz
Speaker · 0 followers
About
It has been observed that visual classification models often rely mostly on the image background, neglecting the foreground, which hurts their robustness to distribution changes. To alleviate this shortcoming, we propose to monitor the model's relevancy signal and manipulate it such that the model is focused on the foreground object. This is done as a finetuning step, involving relatively few samples consisting of pairs of images and their associated foreground masks. Specifically, we encourage the model's relevancy map (i) to assign lower relevance to background regions, (ii) to consider as much information as possible from the foreground, and (iii) we encourage the decisions to have high confidence. When applied to Vision Transformer (ViT) models, a marked improvement in robustness to domain-shifts is observed. Moreover, the foreground masks can be obtained automatically, from a self-supervised variant of the ViT model itself; therefore no additional supervision is required.It has been observed that visual classification models often rely mostly on the image background, neglecting the foreground, which hurts their robustness to distribution changes. To alleviate this shortcoming, we propose to monitor the model's relevancy signal and manipulate it such that the model is focused on the foreground object. This is done as a finetuning step, involving relatively few samples consisting of pairs of images and their associated foreground masks. Specifically, we encourage…
Organizer
NeurIPS 2022
Account · 962 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Set-based Meta-Interpolation for Few-Task Meta-Learning
Watch later
Seanie Lee, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization
Watch later
Siqi Shen, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Score-based generative models detect manifolds
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Watch later
Shunyu Yao, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Tutorial on Graduate Student Descent
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The Importance of Being Correlated: Implications of Dependence in Joint Spectral Inference across Multiple Networks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%