Probing for Multilingual Numerical Understanding in Transformer-Based Language Models

10/13/2020
by   Devin Johnson, et al.
0

Natural language numbers are an example of compositional structures, where larger numbers are composed of operations on smaller numbers. Given that compositional reasoning is a key to natural language understanding, we propose novel multilingual probing tasks tested on DistilBERT, XLM, and BERT to investigate for evidence of compositional reasoning over numerical data in various natural language number systems. By using both grammaticality judgment and value comparison classification tasks in English, Japanese, Danish, and French, we find evidence that the information encoded in these pretrained models' embeddings is sufficient for grammaticality judgments but generally not for value comparisons. We analyze possible reasons for this and discuss how our tasks could be extended in further studies.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

page 8

05/26/2020

ParsBERT: Transformer-based Model for Persian Language Understanding

The surge of pre-trained language models has begun a new era in the fiel...
07/21/2021

The Effectiveness of Intermediate-Task Training for Code-Switched Natural Language Understanding

While recent benchmarks have spurred a lot of new work on improving the ...
10/23/2020

BARThez: a Skilled Pretrained French Sequence-to-Sequence Model

Inductive transfer learning, enabled by self-supervised learning, have t...
09/08/2021

Discrete and Soft Prompting for Multilingual Models

It has been shown for English that discrete and soft prompting perform s...
03/24/2021

Are Multilingual Models Effective in Code-Switching?

Multilingual language models have shown decent performance in multilingu...
10/09/2021

Learning to Follow Language Instructions with Compositional Policies

We propose a framework that learns to execute natural language instructi...
11/01/2019

When Choosing Plausible Alternatives, Clever Hans can be Clever

Pretrained language models, such as BERT and RoBERTa, have shown large i...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.