A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC

09/27/2018
by   Mark Yatskar, et al.
0

In this work, we compare three datasets which build on the paradigm defined in SQuAD for question answering: SQuAD 2.0, QuAC, and CoQA. We compare these three datasets along several of their new features: (1) unanswerable questions, (2) multi-turn interactions, and (3) abstractive answers. We show that the datasets provide complementary coverage of the first two aspects, but weak coverage of the third. Because of the datasets' structural similarity, a single extractive model can be easily adapted to any of the datasets. We show that this model can improve baseline results on both SQuAD 2.0 and CoQA. Despite the core similarity between the datasets, models trained on one dataset are ineffective on another dataset, but we do find moderate performance improvement through pretraining. To encourage evaluation of methods on all of these datasets, we release code for conversion between them.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/13/2021

Semantic Answer Similarity for Evaluating Question Answering Models

The evaluation of question answering models compares ground-truth annota...
research
03/13/2023

Generating multiple-choice questions for medical question answering with distractors and cue-masking

Medical multiple-choice question answering (MCQA) is particularly diffic...
research
03/13/2021

ParaQA: A Question Answering Dataset with Paraphrase Responses for Single-Turn Conversation

This paper presents ParaQA, a question answering (QA) dataset with multi...
research
05/12/2023

Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table

Despite recent interest in open domain question answering (ODQA) over ta...
research
05/29/2023

A Critical Evaluation of Evaluations for Long-form Question Answering

Long-form question answering (LFQA) enables answering a wide range of qu...

Please sign up or login with your details

Forgot password? Click here to reset