A Non-Linear Structural Probe

05/21/2021
by   Jennifer C. White, et al.
0

Probes are models devised to investigate the encoding of knowledge – e.g. syntactic structure – in contextual representations. Probes are often designed for simplicity, which has led to restrictions on probe design that may not allow for the full exploitation of the structure of encoded information; one such restriction is linearity. We examine the case of a structural probe (Hewitt and Manning, 2019), which aims to investigate the encoding of syntactic structure in contextual representations through learning only linear transformations. By observing that the structural probe learns a metric, we are able to kernelize it and develop a novel non-linear variant with an identical number of parameters. We test on 6 languages and find that the radial-basis function (RBF) kernel, in conjunction with regularization, achieves a statistically significant improvement over the baseline in all languages – implying that at least part of the syntactic knowledge is encoded non-linearly. We conclude by discussing how the RBF kernel resembles BERT's self-attention layers and speculate that this resemblance leads to the RBF-based probe's stronger performance.

READ FULL TEXT

page 1

page 2

page 3

page 4

05/04/2020

A Tale of a Probe and a Parser

Measuring what linguistic information is encoded in neural models of lan...
10/05/2020

Pareto Probing: Trading Off Accuracy for Complexity

The question of how to probe contextual word representations in a way th...
04/13/2022

Probing for Constituency Structure in Neural Language Models

In this paper, we investigate to which extent contextual neural language...
10/06/2020

Intrinsic Probing through Dimension Selection

Most modern NLP systems make use of pre-trained contextual representatio...
05/15/2019

What do you learn from context? Probing for sentence structure in contextualized word representations

Contextualized representation models such as ELMo (Peters et al., 2018a)...
04/07/2020

Information-Theoretic Probing for Linguistic Structure

The success of neural networks on a diverse set of NLP tasks has led res...
06/03/2021

Representing Syntax and Composition with Geometric Transformations

The exploitation of syntactic graphs (SyGs) as a word's context has been...