DeepAI AI Chat
Log In Sign Up

The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective

02/03/2022
by   Satyapriya Krishna, et al.
38

As various post hoc explanation methods are increasingly being leveraged to explain complex models in high-stakes settings, it becomes critical to develop a deeper understanding of if and when the explanations output by these methods disagree with each other, and how such disagreements are resolved in practice. However, there is little to no research that provides answers to these critical questions. In this work, we introduce and study the disagreement problem in explainable machine learning. More specifically, we formalize the notion of disagreement between explanations, analyze how often such disagreements occur in practice, and how do practitioners resolve these disagreements. To this end, we first conduct interviews with data scientists to understand what constitutes disagreement between explanations generated by different methods for the same model prediction, and introduce a novel quantitative framework to formalize this understanding. We then leverage this framework to carry out a rigorous empirical analysis with four real-world datasets, six state-of-the-art post hoc explanation methods, and eight different predictive models, to measure the extent of disagreement between the explanations generated by various popular explanation methods. In addition, we carry out an online user study with data scientists to understand how they resolve the aforementioned disagreements. Our results indicate that state-of-the-art explanation methods often disagree in terms of the explanations they output. Our findings also underscore the importance of developing principled evaluation metrics that enable practitioners to effectively compare explanations.

READ FULL TEXT

page 24

page 26

page 28

page 30

page 32

page 34

page 36

page 38

05/15/2022

Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations

As post hoc explanation methods are increasingly being leveraged to expl...
06/02/2022

Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post hoc Explanations

Despite the plethora of post hoc model explanation methods, the basic pr...
04/26/2022

Process Knowledge-infused Learning for Suicidality Assessment on Social Media

Improving the performance and natural language explanations of deep lear...
06/18/2021

Rational Shapley Values

Explaining the predictions of opaque machine learning algorithms is an i...
09/08/2022

From Shapley Values to Generalized Additive Models and back

In explainable machine learning, local post-hoc explanation algorithms a...
05/01/2020

The Grammar of Interactive Explanatory Model Analysis

When analysing a complex system, very often an answer for one question r...
06/19/2018

Instance-Level Explanations for Fraud Detection: A Case Study

Fraud detection is a difficult problem that can benefit from predictive ...