Is Attention Interpretation? A Quantitative Assessment On Sets

07/26/2022
by   Jonathan Haab, et al.
0

The debate around the interpretability of attention mechanisms is centered on whether attention scores can be used as a proxy for the relative amounts of signal carried by sub-components of data. We propose to study the interpretability of attention in the context of set machine learning, where each data point is composed of an unordered collection of instances with a global label. For classical multiple-instance-learning problems and simple extensions, there is a well-defined "importance" ground truth that can be leveraged to cast interpretation as a binary classification problem, which we can quantitatively evaluate. By building synthetic datasets over several data modalities, we perform a systematic assessment of attention-based interpretations. We find that attention distributions are indeed often reflective of the relative importance of individual instances, but that silent failures happen where a model will have high classification performance but attention patterns that do not align with expectations. Based on these observations, we propose to use ensembling to minimize the risk of misleading attention-based explanations.

READ FULL TEXT

page 5

page 10

research
06/10/2020

Why is Attention Not So Attentive?

Attention-based methods have played an important role in model interpret...
research
05/06/2021

Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification

Neural network architectures in natural language processing often use at...
research
09/17/2019

Learning to Deceive with Attention-Based Explanations

Attention mechanisms are ubiquitous components in neural architectures a...
research
09/22/2022

Improving Attention-Based Interpretability of Text Classification Transformers

Transformers are widely used in NLP, where they consistently achieve sta...
research
04/04/2021

Learning Image Aesthetic Assessment from Object-level Visual Components

As it is said by Van Gogh, great things are done by a series of small th...
research
10/23/2019

Semi-Supervised Histology Classification using Deep Multiple Instance Learning and Contrastive Predictive Coding

Convolutional neural networks can be trained to perform histology slide ...
research
08/02/2019

Retrosynthesis with Attention-Based NMT Model and Chemical Analysis of the "Wrong" Predictions

We cast retrosynthesis as a machine translation problem by introducing a...

Please sign up or login with your details

Forgot password? Click here to reset