Pareto Probing: Trading Off Accuracy for Complexity

by   Tiago Pimentel, et al.

The question of how to probe contextual word representations in a way that is principled and useful has seen significant recent attention. In our contribution to this discussion, we argue, first, for a probe metric that reflects the trade-off between probe complexity and performance: the Pareto hypervolume. To measure complexity, we present a number of parametric and non-parametric metrics. Our experiments with such metrics show that probe's performance curves often fail to align with widely accepted rankings between language representations (with, e.g., non-contextual representations outperforming contextual ones). These results lead us to argue, second, that common simplistic probe tasks such as POS labeling and dependency arc labeling, are inadequate to evaluate the properties encoded in contextual word representations. We propose full dependency parsing as an example probe task, and demonstrate it with the Pareto hypervolume. In support of our arguments, the results of this illustrative experiment conform closer to accepted rankings among contextual word representations.


page 6

page 7

page 16


A Tale of a Probe and a Parser

Measuring what linguistic information is encoded in neural models of lan...

Low-Complexity Probing via Finding Subnetworks

The dominant approach in probing neural networks for linguistic properti...

A Non-Linear Structural Probe

Probes are models devised to investigate the encoding of knowledge – e.g...

Polyglot Contextual Representations Improve Crosslingual Transfer

We introduce a method to produce multilingual contextual word representa...

Designing and Interpreting Probes with Control Tasks

Probes, supervised models trained to predict properties (like parts-of-s...

On the Pitfalls of Analyzing Individual Neurons in Language Models

While many studies have shown that linguistic information is encoded in ...

Unsupervised Word Polysemy Quantification with Multiresolution Grids of Contextual Embeddings

The number of senses of a given word, or polysemy, is a very subjective ...