Do Input Gradients Highlight Discriminative Features?
Dec 6, 2021
Speakers
Harshay Shah
Speaker · 0 followers
Prateek Jain
Speaker · 0 followers
Praneeth Netrapalli
Speaker · 0 followers
About
Post-hoc gradient-based interpretability methods [Simonyan et al., 2013, Smilkov et al., 2017] that provide instance-specific explanations of model predictions are often based on assumption (A): magnitude of input gradients—gradients of logits with respect to input—noisily highlight discriminative task-relevant features. In this work, we test the validity of assumption (A) using a three-pronged approach:1. We develop an evaluation framework, DiffROAR, to test assumption (A) on four image classification benchmarks. Our results suggest that (i) input gradients of standard models (i.e., trained on original data) may grossly violate (A), whereas (ii) input gradients of adversarially robust models satisfy (A).2. We then introduce BlockMNIST, an MNIST-based semi-real dataset, that by design encodes a priori knowledge of discriminative features. Our analysis on BlockMNIST leverages this information to validate as well as characterize differences between input gradient attributions of standard and robust models.3. Finally, we theoretically prove that our empirical findings hold on a simplified version of the BlockMNIST dataset. Specifically, we prove that input gradients of standard one-hidden-layer MLPs trained on this dataset do not highlight instance-specific signal coordinates, thus grossly violating assumption (A).Our findings motivate the need to formalize and test common assumptions in interpretability in a falsifiable manner [Leavitt and Morcos, 2020]. Additionally, we believe that the DiffROAR evaluation framework and BlockMNIST-based datasets can serve as sanity checks to audit instance-specific interpretability methods.Post-hoc gradient-based interpretability methods [Simonyan et al., 2013, Smilkov et al., 2017] that provide instance-specific explanations of model predictions are often based on assumption (A): magnitude of input gradients—gradients of logits with respect to input—noisily highlight discriminative task-relevant features. In this work, we test the validity of assumption (A) using a three-pronged approach:1. We develop an evaluation framework, DiffROAR, to test assumption (A) on four image classif…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulation
Watch later
Alex J. Chan, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Motif-based Graph Self-Supervised Learning for Molecular Property Prediction
Watch later
Zaixi Zhang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Long Short-Term Transformer for Online Action Detection
Watch later
Mingze Xu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The Caltech Mouse Social Interactions 2021 Dataset
Watch later
Jennifer Sun, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Robust Implicit Networks via Contraction Thoery
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The 2021 BEETL Competition: Benchmarks for EEG Transfer Learning
Watch later
Xiaoxi Wei, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%