Log In Sign Up

Explaining by Removing: A Unified Framework for Model Explanation

by   Ian Covert, et al.

Researchers have proposed a wide variety of model explanation approaches, but it remains unclear how most methods are related or when one method is preferable to another. We establish a new class of methods, removal-based explanations, that are based on the principle of simulating feature removal to quantify each feature's influence. These methods vary in several respects, so we develop a framework that characterizes each method along three dimensions: 1) how the method removes features, 2) what model behavior the method explains, and 3) how the method summarizes each feature's influence. Our framework unifies 25 existing methods, including several of the most widely used approaches (SHAP, LIME, Meaningful Perturbations, permutation tests). This new class of explanation methods has rich connections that we examine using tools that have been largely overlooked by the explainability literature. To anchor removal-based explanations in cognitive psychology, we show that feature removal is a simple application of subtractive counterfactual reasoning. Ideas from cooperative game theory shed light on the relationships and trade-offs among different methods, and we derive conditions under which all removal-based explanations have information-theoretic interpretations. Through this analysis, we develop a unified framework that helps practitioners better understand model explanation tools, and that offers a strong theoretical foundation upon which future explainability research can build.


Feature Removal Is a Unifying Principle for Model Explanation Methods

Researchers have proposed a wide variety of model explanation approaches...

Global Counterfactual Explanations: Investigations, Implementations and Improvements

Counterfactual explanations have been widely studied in explainability, ...

Topological Representations of Local Explanations

Local explainability methods – those which seek to generate an explanati...

A Unified Approach to Interpreting Model Predictions

Understanding why a model makes a certain prediction can be as crucial a...

GRETEL: A unified framework for Graph Counterfactual Explanation Evaluation

Machine Learning (ML) systems are a building part of the modern tools wh...

Contextual Importance and Utility: aTheoretical Foundation

This paper provides new theory to support to the eXplainable AI (XAI) me...

Algorithms to estimate Shapley value feature attributions

Feature attributions based on the Shapley value are popular for explaini...