Probing for Understanding of English Verb Classes and Alternations in Large Pre-trained Language Models

09/11/2022
by   David K. Yi, et al.
0

We investigate the extent to which verb alternation classes, as described by Levin (1993), are encoded in the embeddings of Large Pre-trained Language Models (PLMs) such as BERT, RoBERTa, ELECTRA, and DeBERTa using selectively constructed diagnostic classifiers for word and sentence-level prediction tasks. We follow and expand upon the experiments of Kann et al. (2019), which aim to probe whether static embeddings encode frame-selectional properties of verbs. At both the word and sentence level, we find that contextual embeddings from PLMs not only outperform non-contextual embeddings, but achieve astonishingly high accuracies on tasks across most alternation classes. Additionally, we find evidence that the middle-to-upper layers of PLMs achieve better performance on average than the lower layers across all probing tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/11/2023

Towards preserving word order importance through Forced Invalidation

Large pre-trained language models such as BERT have been widely used as ...
research
04/20/2023

Word Sense Induction with Knowledge Distillation from BERT

Pre-trained contextual language models are ubiquitously employed for lan...
research
12/27/2021

Understanding RoBERTa's Mood: The Role of Contextual-Embeddings as User-Representations for Depression Prediction

Many works in natural language processing have shown connections between...
research
04/27/2023

Idioms, Probing and Dangerous Things: Towards Structural Probing for Idiomaticity in Vector Space

The goal of this paper is to learn more about how idiomatic information ...
research
10/13/2022

Sentence Ambiguity, Grammaticality and Complexity Probes

It is unclear whether, how and where large pre-trained language models c...
research
03/14/2023

Do Transformers Parse while Predicting the Masked Word?

Pre-trained language models have been shown to encode linguistic structu...
research
10/21/2022

Probing with Noise: Unpicking the Warp and Weft of Embeddings

Improving our understanding of how information is encoded in vector spac...

Please sign up or login with your details

Forgot password? Click here to reset