DeepAI AI Chat
Log In Sign Up

Challenging common interpretability assumptions in feature attribution explanations

by   Jonathan Dinu, et al.

As machine learning and algorithmic decision making systems are increasingly being leveraged in high-stakes human-in-the-loop settings, there is a pressing need to understand the rationale of their predictions. Researchers have responded to this need with explainable AI (XAI), but often proclaim interpretability axiomatically without evaluation. When these systems are evaluated, they are often tested through offline simulations with proxy metrics of interpretability (such as model complexity). We empirically evaluate the veracity of three common interpretability assumptions through a large scale human-subjects experiment with a simple "placebo explanation" control. We find that feature attribution explanations provide marginal utility in our task for a human decision maker and in certain cases result in worse decisions due to cognitive and contextual confounders. This result challenges the assumed universal benefit of applying these methods and we hope this work will underscore the importance of human evaluation in XAI research. Supplemental materials – including anonymized data from the experiment, code to replicate the study, an interactive demo of the experiment, and the models used in the analysis – can be found at:


page 1

page 2

page 3

page 4


Don't Explain without Verifying Veracity: An Evaluation of Explainable AI with Video Activity Recognition

Explainable machine learning and artificial intelligence models have bee...

Faithful and Plausible Explanations of Medical Code Predictions

Machine learning models that offer excellent predictive performance ofte...

Learning When to Advise Human Decision Makers

Artificial intelligence (AI) systems are increasingly used for providing...

How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations

There have been several research works proposing new Explainable AI (XAI...

Impossibility Theorems for Feature Attribution

Despite a sea of interpretability methods that can produce plausible exp...

A general-purpose method for applying Explainable AI for Anomaly Detection

The need for explainable AI (XAI) is well established but relatively lit...