Causal Abstractions of Neural Networks

06/06/2021
by   Atticus Geiger, et al.
0

Structural analysis methods (e.g., probing and feature attribution) are increasingly important tools for neural network analysis. We propose a new structural analysis method grounded in a formal theory of causal abstraction that provides rich characterizations of model-internal representations and their roles in input/output behavior. In this method, neural representations are aligned with variables in interpretable causal models, and then interchange interventions are used to experimentally verify that the neural representations have the causal properties of their aligned variables. We apply this method in a case study to analyze neural models trained on Multiply Quantified Natural Language Inference (MQNLI) corpus, a highly complex NLI dataset that was constructed with a tree-structured natural logic causal model. We discover that a BERT-based model with state-of-the-art performance successfully realizes the approximate causal structure of the natural logic causal model, whereas a simpler baseline model fails to show any such structure, demonstrating that neural representations encode the compositional structure of MQNLI examples.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 18

page 19

page 20

12/01/2021

Inducing Causal Structure for Interpretable Neural Networks

In many areas, we have well-founded insights about causal structure that...
10/21/2019

Discovering the Compositional Structure of Vector Representations with Role Learning Networks

Neural networks (NNs) are able to perform tasks that rely on composition...
02/06/2019

Neural Network Attributions: A Causal Perspective

We propose a new attribution method for neural networks developed using ...
03/29/2021

Compositional Abstraction Error and a Category of Causal Models

Interventional causal models describe joint distributions over some vari...
07/25/2017

Analogs of Linguistic Structure in Deep Representations

We investigate the compositional structure of message vectors computed b...
08/21/2020

Amortized learning of neural causal representations

Causal models can compactly and efficiently encode the data-generating p...
07/02/2021

The Causal Neural Connection: Expressiveness, Learnability, and Inference

One of the central elements of any causal inference is an object called ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.