Using Distributional Principles for the Semantic Study of Contextual Language Models

11/23/2021
by   Olivier Ferret, et al.
0

Many studies were recently done for investigating the properties of contextual language models but surprisingly, only a few of them consider the properties of these models in terms of semantic similarity. In this article, we first focus on these properties for English by exploiting the distributional principle of substitution as a probing mechanism in the controlled context of SemCor and WordNet paradigmatic relations. Then, we propose to adapt the same method to a more open setting for characterizing the differences between static and contextual language models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/12/2021

Time Masking for Temporal Language Models

Our world is constantly evolving, and so is the content on the web. Cons...
research
11/20/2022

Pragmatic Constraint on Distributional Semantics

This paper studies the limits of language models' statistical learning i...
research
05/24/2023

Deriving Language Models from Masked Language Models

Masked language models (MLM) do not explicitly define a distribution ove...
research
05/20/2021

A comprehensive comparative evaluation and analysis of Distributional Semantic Models

Distributional semantics has deeply changed in the last decades. First, ...
research
09/13/2021

On Language Models for Creoles

Creole languages such as Nigerian Pidgin English and Haitian Creole are ...
research
09/09/2021

All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality

Similarity measures are a vital tool for understanding how language mode...
research
01/20/2023

Can Peanuts Fall in Love with Distributional Semantics?

The context in which a sentence appears can drastically alter our expect...

Please sign up or login with your details

Forgot password? Click here to reset