QuALITY: Question Answering with Long Input Texts, Yes!

12/16/2021
by   Richard Yuanzhe Pang, et al.
2

To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our questions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addition, only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Current models perform poorly on this task (55.4 and significantly lag behind human performance (93.5

READ FULL TEXT

page 15

page 16

research
05/07/2021

A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers

Readers of academic research papers often read with the goal of answerin...
research
09/23/2021

BiRdQA: A Bilingual Dataset for Question Answering on Tricky Riddles

A riddle is a question or statement with double or veiled meanings, foll...
research
09/15/2023

Investigating Answerability of LLMs for Long-Form Question Answering

As we embark on a new era of LLMs, it becomes increasingly crucial to un...
research
04/20/2018

Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehension

The current trend of extractive question answering (QA) heavily relies o...
research
09/22/2021

Recursively Summarizing Books with Human Feedback

A major challenge for scaling machine learning is training models to per...
research
07/19/2022

Can You Fool AI by Doing a 180? x2013 A Case Study on Authorship Analysis of Texts by Arata Osada

This paper is our attempt at answering a twofold question covering the a...

Please sign up or login with your details

Forgot password? Click here to reset