Pragmatic competence of pre-trained language models through the lens of discourse connectives

09/27/2021
by   Lalchand Pandia, et al.
0

As pre-trained language models (LMs) continue to dominate NLP, it is increasingly important that we understand the depth of language capabilities in these models. In this paper, we target pre-trained LMs' competence in pragmatics, with a focus on pragmatics relating to discourse connectives. We formulate cloze-style tests using a combination of naturally-occurring data and controlled inputs drawn from psycholinguistics. We focus on testing models' ability to use pragmatic cues to predict discourse connectives, models' ability to understand implicatures relating to connectives, and the extent to which models show humanlike preferences regarding temporal dynamics of connectives. We find that although models predict connectives reasonably well in the context of naturally-occurring data, when we control contexts to isolate high-level pragmatic cues, model sensitivity is much lower. Models also do not show substantial humanlike temporal preferences. Overall, the findings suggest that at present, dominant pre-training paradigms do not result in substantial pragmatic competence in our models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/08/2022

Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models

With a growing number of BERTology work analyzing different components o...
research
06/21/2020

Labeling Explicit Discourse Relations using Pre-trained Language Models

Labeling explicit discourse relations is one of the most challenging sub...
research
10/05/2022

"No, they did not": Dialogue response dynamics in pre-trained language models

A critical component of competence in language is being able to identify...
research
09/10/2021

Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations

Current language models are usually trained using a self-supervised sche...
research
09/25/2021

Sorting through the noise: Testing robustness of information processing in pre-trained language models

Pre-trained LMs have shown impressive performance on downstream NLP task...
research
05/12/2021

How Reliable are Model Diagnostics?

In the pursuit of a deeper understanding of a model's behaviour, there i...
research
05/16/2022

What GPT Knows About Who is Who

Coreference resolution – which is a crucial task for understanding disco...

Please sign up or login with your details

Forgot password? Click here to reset