Evaluating Commonsense in Pre-trained Language Models

11/27/2019
by   Xuhui Zhou, et al.
0

Contextualized representations trained over large raw text data have given remarkable improvements for NLP tasks including question answering and reading comprehension. There have been works showing that syntactic, semantic and word sense knowledge are contained in such representations, which explains why they benefit such tasks. However, relatively little work has been done investigating commonsense knowledge contained in contextualized representations, which is crucial for human question answering and reading comprehension. We study the commonsense ability of GPT, BERT, XLNet, and RoBERTa by testing them on seven challenging benchmarks, finding that language modeling and its variants are effective objectives for promoting models' commonsense ability while bi-directional context and larger training set are bonuses. We additionally find that current models do poorly on tasks require more necessary inference steps. Finally, we test the robustness of models by making dual test cases, which are correlated so that the correct prediction of one sample should lead to correct prediction of the other. Interestingly, the models show confusion on these test cases, which suggests that they learn commonsense at the surface rather than the deep level. We release a test set, named CATs publicly, for future research.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/01/2022

Why Do Neural Language Models Still Need Commonsense Knowledge to Handle Semantic Variations in Question Answering?

Many contextualized word representations are now learned by intricate ne...
research
09/08/2019

Commonsense Knowledge + BERT for Level 2 Reading Comprehension Ability Test

Commonsense knowledge plays an important role when we read. The performa...
research
10/25/2020

Commonsense knowledge adversarial dataset that challenges ELECTRA

Commonsense knowledge is critical in human reading comprehension. While ...
research
10/12/2022

Probing Commonsense Knowledge in Pre-trained Language Models with Sense-level Precision and Expanded Vocabulary

Progress on commonsense reasoning is usually measured from performance i...
research
05/11/2020

Commonsense Evidence Generation and Injection in Reading Comprehension

Human tackle reading comprehension not only based on the given context i...
research
05/06/2022

KECP: Knowledge Enhanced Contrastive Prompting for Few-shot Extractive Question Answering

Extractive Question Answering (EQA) is one of the most important tasks i...
research
04/10/2022

Data Augmentation for Biomedical Factoid Question Answering

We study the effect of seven data augmentation (da) methods in factoid q...

Please sign up or login with your details

Forgot password? Click here to reset