Transformers in the loop: Polarity in neural models of language

09/08/2021
by   Lisa Bylinina, et al.
0

Representation of linguistic phenomena in computational language models is typically assessed against the predictions of existing linguistic theories of these phenomena. Using the notion of polarity as a case study, we show that this is not always the most adequate set-up. We probe polarity via so-called 'negative polarity items' (in particular, English 'any') in two pre-trained Transformer-based models (BERT and GPT-2). We show that – at least for polarity – metrics derived from language models are more consistent with data from psycholinguistic experiments than linguistic theory predictions. Establishing this allows us to more adequately evaluate the performance of language models and also to use language models to discover new insights into natural language grammar beyond existing linguistic theories. Overall, our results encourage a closer tie between experiments with human subjects and with language models. We propose methods to enable this closer tie, with language models as part of experimental pipeline, and show this pipeline at work.

READ FULL TEXT
research
07/01/2022

Is neural language acquisition similar to natural? A chronological probing study

The probing methodology allows one to obtain a partial representation of...
research
05/28/2021

Language Models Use Monotonicity to Assess NPI Licensing

We investigate the semantic knowledge of language models (LMs), focusing...
research
09/13/2021

Old BERT, New Tricks: Artificial Language Learning for Pre-Trained Language Models

We extend the artificial language learning experimental paradigm from ps...
research
09/13/2021

The Grammar-Learning Trajectories of Neural Language Models

The learning trajectories of linguistic phenomena provide insight into t...
research
05/22/2023

DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules

Existing large language models (LLMs) that mainly focus on Standard Amer...
research
05/22/2023

Prompt-based methods may underestimate large language models' linguistic generalizations

Prompting is now a dominant method for evaluating the linguistic knowled...
research
06/12/2021

Can Transformer Language Models Predict Psychometric Properties?

Transformer-based language models (LMs) continue to advance state-of-the...

Please sign up or login with your details

Forgot password? Click here to reset