A Latent-Variable Model for Intrinsic Probing

01/20/2022
by   Karolina Stańczak, et al.
0

The success of pre-trained contextualized representations has prompted researchers to analyze them for the presence of linguistic information. Indeed, it is natural to assume that these pre-trained representations do encode some level of linguistic knowledge as they have brought about large empirical improvements on a wide variety of NLP tasks, which suggests they are learning true linguistic generalization. In this work, we focus on intrinsic probing, an analysis technique where the goal is not only to identify whether a representation encodes a linguistic attribute, but also to pinpoint where this attribute is encoded. We propose a novel latent-variable formulation for constructing intrinsic probes and derive a tractable variational approximation to the log-likelihood. Our results show that our model is versatile and yields tighter mutual information estimates than two intrinsic probes previously proposed in the literature. Finally, we find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.

READ FULL TEXT

page 1

page 15

research
10/06/2020

Intrinsic Probing through Dimension Selection

Most modern NLP systems make use of pre-trained contextual representatio...
research
04/08/2021

A Simple Geometric Method for Cross-Lingual Linguistic Transformations with Pre-trained Autoencoders

Powerful sentence encoders trained for multiple languages are on the ris...
research
10/15/2021

Probing as Quantifying the Inductive Bias of Pre-trained Representations

Pre-trained contextual representations have led to dramatic performance ...
research
04/13/2021

DirectProbe: Studying Representations without Classifiers

Understanding how linguistic structures are encoded in contextualized em...
research
04/07/2020

Information-Theoretic Probing for Linguistic Structure

The success of neural networks on a diverse set of NLP tasks has led res...
research
02/13/2018

GILBO: One Metric to Measure Them All

We propose a simple, tractable lower bound on the mutual information con...
research
11/30/2017

An interpretable latent variable model for attribute applicability in the Amazon catalogue

Learning attribute applicability of products in the Amazon catalog (e.g....

Please sign up or login with your details

Forgot password? Click here to reset