Commonsense knowledge adversarial dataset that challenges ELECTRA

10/25/2020
by   Gongqi Lin, et al.
0

Commonsense knowledge is critical in human reading comprehension. While machine comprehension has made significant progress in recent years, the ability in handling commonsense knowledge remains limited. Synonyms are one of the most widely used commonsense knowledge. Constructing adversarial dataset is an important approach to find weak points of machine comprehension models and support the design of solutions. To investigate machine comprehension models' ability in handling the commonsense knowledge, we created a Question and Answer Dataset with common knowledge of Synonyms (QADS). QADS are questions generated based on SQuAD 2.0 by applying commonsense knowledge of synonyms. The synonyms are extracted from WordNet. Words often have multiple meanings and synonyms. We used an enhanced Lesk algorithm to perform word sense disambiguation to identify synonyms for the context. ELECTRA achieves the state-of-art result on the SQuAD 2.0 dataset in 2019. With scale, ELECTRA can achieve similar performance as BERT does. However, QADS shows that ELECTRA has little ability to handle commonsense knowledge of synonyms. In our experiment, ELECTRA-small can achieve 70 not perform much better. Its accuracy on SQuAD 2.0 is 88 significantly to 26 failed badly on QADS, was not as bad as ELECTRA. The result shows that even top-performing NLP models have little ability to handle commonsense knowledge which is essential in reading comprehension.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/08/2019

Commonsense Knowledge + BERT for Level 2 Reading Comprehension Ability Test

Commonsense knowledge plays an important role when we read. The performa...
research
05/17/2020

Adversarial Training for Commonsense Inference

We propose an AdversariaL training algorithm for commonsense InferenCE (...
research
09/12/2020

Improving Machine Reading Comprehension with Contextualized Commonsense Knowledge

In this paper, we aim to extract commonsense knowledge to improve machin...
research
03/26/2021

Incorporating Connections Beyond Knowledge Embeddings: A Plug-and-Play Module to Enhance Commonsense Reasoning in Machine Reading Comprehension

Conventional Machine Reading Comprehension (MRC) has been well-addressed...
research
05/19/2019

HellaSwag: Can a Machine Really Finish Your Sentence?

Recent work by Zellers et al. (2018) introduced a new task of commonsens...
research
05/11/2020

Commonsense Evidence Generation and Injection in Reading Comprehension

Human tackle reading comprehension not only based on the given context i...
research
11/27/2019

Evaluating Commonsense in Pre-trained Language Models

Contextualized representations trained over large raw text data have giv...

Please sign up or login with your details

Forgot password? Click here to reset