On the Pitfalls of Analyzing Individual Neurons in Language Models

10/14/2021
by   Omer Antverg, et al.
0

While many studies have shown that linguistic information is encoded in hidden word representations, few have studied individual neurons, to show how and in which neurons it is encoded. Among these, the common approach is to use an external probe to rank neurons according to their relevance to some linguistic attribute, and to evaluate the obtained ranking using the same probe that produced it. We show two pitfalls in this methodology: 1. It confounds distinct factors: probe quality and ranking quality. We separate them and draw conclusions on each. 2. It focuses on encoded information, rather than information that is used by the model. We show that these are not the same. We compare two recent ranking methods and a simple one we introduce, and evaluate them with regard to both of these aspects.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/04/2020

A Tale of a Probe and a Parser

Measuring what linguistic information is encoded in neural models of lan...
research
10/06/2020

Intrinsic Probing through Dimension Selection

Most modern NLP systems make use of pre-trained contextual representatio...
research
10/06/2020

Analyzing Individual Neurons in Pre-trained Language Models

While a lot of analysis has been carried to demonstrate linguistic knowl...
research
11/03/2018

Identifying and Controlling Important Neurons in Neural Machine Translation

Neural machine translation (NMT) models learn representations containing...
research
04/08/2021

Low-Complexity Probing via Finding Subnetworks

The dominant approach in probing neural networks for linguistic properti...
research
12/13/2021

Sparse Interventions in Language Models with Differentiable Masking

There has been a lot of interest in understanding what information is ca...
research
10/05/2020

Pareto Probing: Trading Off Accuracy for Complexity

The question of how to probe contextual word representations in a way th...

Please sign up or login with your details

Forgot password? Click here to reset