Order in the Court: Explainable AI Methods Prone to Disagreement

05/07/2021
by   Michael Neely, et al.
14

In Natural Language Processing, feature-additive explanation methods quantify the independent contribution of each input token towards a model's decision. By computing the rank correlation between attention weights and the scores produced by a small sample of these methods, previous analyses have sought to either invalidate or support the role of attention-based explanations as a faithful and plausible measure of salience. To investigate what measures of rank correlation can reliably conclude, we comprehensively compare feature-additive methods, including attention-based explanations, across several neural architectures and tasks. In most cases, we find that none of our chosen methods agree. Therefore, we argue that rank correlation is largely uninformative and does not measure the quality of feature-additive methods. Additionally, the range of conclusions a practitioner may draw from a single explainability algorithm are limited.

READ FULL TEXT
research
01/28/2022

Rethinking Attention-Model Explainability through Faithfulness Violation Test

Attention mechanisms are dominating the explainability of deep models. T...
research
05/06/2021

Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification

Neural network architectures in natural language processing often use at...
research
08/08/2023

Semantic Interpretation and Validation of Graph Attention-based Explanations for GNN Models

In this work, we propose a methodology for investigating the application...
research
03/27/2019

iBreakDown: Uncertainty of Model Explanations for Non-additive Predictive Models

Explainable Artificial Intelligence (XAI) brings a lot of attention rece...
research
10/13/2022

How (Not) To Evaluate Explanation Quality

The importance of explainability is increasingly acknowledged in natural...
research
09/14/2020

SCOUTER: Slot Attention-based Classifier for Explainable Image Recognition

Explainable artificial intelligence is gaining attention. However, most ...
research
12/12/2022

Drivers of the decrease of patent similarities from 1976 to 2021

The citation network of patents citing prior art arises from the legal o...

Please sign up or login with your details

Forgot password? Click here to reset