Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering

06/08/2021
by   Aditya Gupta, et al.
0

Disfluencies is an under-studied topic in NLP, even though it is ubiquitous in human conversation. This is largely due to the lack of datasets containing disfluencies. In this paper, we present a new challenge question answering dataset, Disfl-QA, a derivative of SQuAD, where humans introduce contextual disfluencies in previously fluent questions. Disfl-QA contains a variety of challenging disfluencies that require a more comprehensive understanding of the text than what was necessary in prior datasets. Experiments show that the performance of existing state-of-the-art question answering models degrades significantly when tested on Disfl-QA in a zero-shot setting.We show data augmentation methods partially recover the loss in performance and also demonstrate the efficacy of using gold data for fine-tuning. We argue that we need large-scale disfluency datasets in order for NLP models to be robust to them. The dataset is publicly available at: https://github.com/google-research-datasets/disfl-qa.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/03/2022

JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension

Question Answering (QA) is a task in which a machine understands a given...
research
10/25/2022

RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering

We introduce RoMQA, the first benchmark for robust, multi-evidence, mult...
research
04/14/2019

Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering

Recently, a simple combination of passage retrieval using off-the-shelf ...
research
09/23/2021

Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap

Question answering (QA) models for reading comprehension have been demon...
research
01/11/2023

ClimaBench: A Benchmark Dataset For Climate Change Text Understanding in English

The topic of Climate Change (CC) has received limited attention in NLP d...
research
06/12/2023

When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset

Annotators are not fungible. Their demographics, life experiences, and b...
research
01/22/2020

ManyModalQA: Modality Disambiguation and QA over Diverse Inputs

We present a new multimodal question answering challenge, ManyModalQA, i...

Please sign up or login with your details

Forgot password? Click here to reset