DeepAI AI Chat
Log In Sign Up

Making Heads or Tails: Towards Semantically Consistent Visual Counterfactuals

by   Simon Vandenhende, et al.

A visual counterfactual explanation replaces image regions in a query image with regions from a distractor image such that the system's decision on the transformed image changes to the distractor class. In this work, we present a novel framework for computing visual counterfactual explanations based on two key ideas. First, we enforce that the replaced and replacer regions contain the same semantic part, resulting in more semantically consistent explanations. Second, we use multiple distractor images in a computationally efficient way and obtain more discriminative explanations with fewer region replacements. Our approach is 27% more semantically consistent and an order of magnitude faster than a competing method on three fine-grained image recognition datasets. We highlight the utility of our counterfactuals over existing works through machine teaching experiments where we teach humans to classify different bird species. We also complement our explanations with the vocabulary of parts and attributes that contributed the most to the system's decision. In this task as well, we obtain state-of-the-art results when using our counterfactual explanations relative to existing works, reinforcing the importance of semantically consistent explanations.


page 2

page 21

page 23

page 24

page 25

page 26

page 27

page 28


Counterfactual Visual Explanations

A counterfactual query is typically of the form 'For situation X, why wa...

STEEX: Steering Counterfactual Explanations with Semantics

As deep learning models are increasingly used in safety-critical applica...

Generating Visual Explanations

Clearly explaining a rationale for a classification decision to an end-u...

SCOUT: Self-aware Discriminant Counterfactual Explanations

The problem of counterfactual visual explanations is considered. A new f...

GRETEL: A unified framework for Graph Counterfactual Explanation Evaluation

Machine Learning (ML) systems are a building part of the modern tools wh...

Diffusion Visual Counterfactual Explanations

Visual Counterfactual Explanations (VCEs) are an important tool to under...

Visual Concept Recognition and Localization via Iterative Introspection

Convolutional neural networks have been shown to develop internal repres...