DeepAI
Log In Sign Up

Probing as Quantifying the Inductive Bias of Pre-trained Representations

10/15/2021
by   Alexander Immer, et al.
0

Pre-trained contextual representations have led to dramatic performance improvements on a range of downstream tasks. This has motivated researchers to quantify and understand the linguistic information encoded in them. In general, this is done by probing, which consists of training a supervised model to predict a linguistic property from said representations. Unfortunately, this definition of probing has been subject to extensive criticism, and can lead to paradoxical or counter-intuitive results. In this work, we present a novel framework for probing where the goal is to evaluate the inductive bias of representations for a particular task, and provide a practical avenue to do this using Bayesian inference. We apply our framework to a series of token-, arc-, and sentence-level tasks. Our results suggest that our framework solves problems of previous approaches and that fastText can offer a better inductive bias than BERT in certain situations.

READ FULL TEXT

page 3

page 6

04/29/2020

Learning Better Universal Representations from Pre-trained Contextualized Language Models

Pre-trained contextualized language models such as BERT have shown great...
10/06/2020

Intrinsic Probing through Dimension Selection

Most modern NLP systems make use of pre-trained contextual representatio...
01/20/2022

A Latent-Variable Model for Intrinsic Probing

The success of pre-trained contextualized representations has prompted r...
08/22/2019

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

We introduce a new pre-trainable generic representation for visual-lingu...
05/24/2022

Learning for Expressive Task-Related Sentence Representations

NLP models learn sentence representations for downstream tasks by tuning...
04/30/2020

A Matter of Framing: The Impact of Linguistic Formalism on Probing Results

Deep pre-trained contextualized encoders like BERT (Delvin et al., 2019)...
08/26/2022

What Do NLP Researchers Believe? Results of the NLP Community Metasurvey

We present the results of the NLP Community Metasurvey. Run from May to ...