DeepAI AI Chat
Log In Sign Up

Revisiting Sanity Checks for Saliency Maps

by   Gal Yona, et al.

Saliency methods are a popular approach for model debugging and explainability. However, in the absence of ground-truth data for what the correct maps should be, evaluating and comparing different approaches remains a long-standing challenge. The sanity checks methodology of Adebayo et al [Neurips 2018] has sought to address this challenge. They argue that some popular saliency methods should not be used for explainability purposes since the maps they produce are not sensitive to the underlying model that is to be explained. Through a causal re-framing of their objective, we argue that their empirical evaluation does not fully establish these conclusions, due to a form of confounding introduced by the tasks they evaluate on. Through various experiments on simple custom tasks we demonstrate that some of their conclusions may indeed be artifacts of the tasks more than a criticism of the saliency methods themselves. More broadly, our work challenges the utility of the sanity check methodology, and further highlights that saliency map evaluation beyond ad-hoc visual examination remains a fundamental challenge.


page 5

page 6

page 8

page 9

page 10


Saliency for free: Saliency prediction as a side-effect of object recognition

Saliency is the perceptual capacity of our visual system to focus our at...

Evaluating Post-hoc Interpretability with Intrinsic Interpretability

Despite Convolutional Neural Networks having reached human-level perform...

Saliency Integration: An Arbitrator Model

Saliency integration approaches have aroused general concern on unifying...

Shared Interest: Large-Scale Visual Analysis of Model Behavior by Measuring Human-AI Alignment

Saliency methods – techniques to identify the importance of input featur...

Evaluating saliency methods on artificial data with different background types

Over the last years, many 'explainable artificial intelligence' (xAI) ap...

The Generalizability of Explanations

Due to the absence of ground truth, objective evaluation of explainabili...

Crowdsourcing Evaluation of Saliency-based XAI Methods

Understanding the reasons behind the predictions made by deep neural net...