Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study

by   Xiangyang Mou, et al.

Recent advancements in open-domain question answering (ODQA), i.e., finding answers from large open-domain corpus like Wikipedia, have led to human-level performance on many datasets. However, progress in QA over book stories (Book QA) lags behind despite its similar task formulation to ODQA. This work provides a comprehensive and quantitative analysis about the difficulty of Book QA: (1) We benchmark the research on the NarrativeQA dataset with extensive experiments with cutting-edge ODQA techniques. This quantifies the challenges Book QA poses, as well as advances the published state-of-the-art with a ∼7% absolute improvement on Rouge-L. (2) We further analyze the detailed challenges in Book QA through human studies.[<>.] Our findings indicate that the event-centric questions dominate this task, which exemplifies the inability of existing QA models to handle event-oriented scenarios.


page 1

page 9


Frustratingly Hard Evidence Retrieval for QA Over Books

A lot of progress has been made to improve question answering (QA) in re...

Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets

Ideally Open-Domain Question Answering models should exhibit a number of...

Careful Selection of Knowledge to solve Open Book Question Answering

Open book question answering is a type of natural language based QA (NLQ...

Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models

Reasoning about time is of fundamental importance. Many facts are time-d...

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

We present a new kind of question answering dataset, OpenBookQA, modeled...

Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering

In open-domain question answering (QA), retrieve-and-read mechanism has ...

Code Repositories

Please sign up or login with your details

Forgot password? Click here to reset