Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses

04/29/2021
by   Aina Garí Soler, et al.
0

Pre-trained language models (LMs) encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. We propose a novel experimental setup for analysing this knowledge in LMs specifically trained for different languages (English, French, Spanish and Greek) and in multilingual BERT. We perform our analysis on datasets carefully designed to reflect different sense distributions, and control for parameters that are highly correlated with polysemy such as frequency and grammatical category. We demonstrate that BERT-derived representations reflect words' polysemy level and their partitionability into senses. Polysemy-related information is more clearly present in English BERT embeddings, but models in other languages also manage to establish relevant distinctions between words at different polysemy levels. Our results contribute to a better understanding of the knowledge encoded in contextualised representations and open up new avenues for multilingual lexical semantics research.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 9

10/12/2020

Probing Pretrained Language Models for Lexical Semantics

The success of large pretrained language models (LMs) such as BERT and R...
10/09/2021

An Isotropy Analysis in the Multilingual BERT Embedding Space

Several studies have explored various advantages of multilingual pre-tra...
12/10/2020

As good as new. How to successfully recycle English GPT-2 to make models for other languages

Large generative language models have been very successful for English, ...
10/23/2020

GiBERT: Introducing Linguistic Knowledge into BERT through a Lightweight Gated Injection Method

Large pre-trained language models such as BERT have been the driving for...
06/15/2021

Knowledge-Rich BERT Embeddings for Readability Assessment

Automatic readability assessment (ARA) is the task of evaluating the lev...
09/17/2020

Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQA

Many NLP tasks have benefited from transferring knowledge from contextua...
06/12/2016

External Lexical Information for Multilingual Part-of-Speech Tagging

Morphosyntactic lexicons and word vector representations have both prove...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.