Neural Response Interpretation through the Lens of Critical Pathways

03/31/2021
by   Ashkan Khakzar, et al.
12

Is critical input information encoded in specific sparse pathways within the neural network? In this work, we discuss the problem of identifying these critical pathways and subsequently leverage them for interpreting the network's response to an input. The pruning objective – selecting the smallest group of neurons for which the response remains equivalent to the original network – has been previously proposed for identifying critical pathways. We demonstrate that sparse pathways derived from pruning do not necessarily encode critical input information. To ensure sparse pathways include critical fragments of the encoded input information, we propose pathway selection via neurons' contribution to the response. We proceed to explain how critical pathways can reveal critical input features. We prove that pathways selected via neuron contribution are locally linear (in an L2-ball), a property that we use for proposing a feature attribution method: "pathway gradient". We validate our interpretation method using mainstream evaluation experiments. The validation of pathway gradient interpretation method further confirms that selected pathways using neuron contributions correspond to critical input features. The code is publicly available.

READ FULL TEXT

page 3

page 5

page 7

page 14

page 16

page 17

page 18

research
11/25/2019

Explaining Neural Networks via Perturbing Important Learned Features

Attributing the output of a neural network to the contribution of given ...
research
10/04/2021

Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information

One principal approach for illuminating a black-box neural network is fe...
research
07/08/2022

SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance

The leap in performance in state-of-the-art computer vision methods is a...
research
05/17/2021

Fine-grained Interpretation and Causation Analysis in Deep NLP Models

This paper is a write-up for the tutorial on "Fine-grained Interpretatio...
research
12/17/2018

A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks

Neuron pruning is an efficient method to compress the network into a sli...
research
05/03/2015

Some Theoretical Properties of a Network of Discretely Firing Neurons

The problem of optimising a network of discretely firing neurons is addr...
research
06/12/2016

Critical Echo State Networks that Anticipate Input using Morphable Transfer Functions

The paper investigates a new type of truly critical echo state networks ...

Please sign up or login with your details

Forgot password? Click here to reset