Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

01/06/2021
by   Mor Geva, et al.
0

A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce StrategyQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, StrategyQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in StrategyQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well (87 on this task, while our best baseline reaches an accuracy of ∼66

READ FULL TEXT
04/28/2022

Inferring Implicit Relations with Language Models

A prominent challenge for modern language understanding systems is the a...
11/02/2020

Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

A multi-hop question answering (QA) dataset aims to test reasoning and i...
04/16/2022

Calibrating Trust of Multi-Hop Question Answering Systems with Decompositional Probes

Multi-hop Question Answering (QA) is a challenging task since it require...
06/27/2016

SelQA: A New Benchmark for Selection-based Question Answering

This paper presents a new selection-based question answering dataset, Se...
04/14/2021

TWEAC: Transformer with Extendable QA Agent Classifiers

Question answering systems should help users to access knowledge on a br...
06/17/2019

Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA

Multi-hop question answering requires a model to connect multiple pieces...
05/25/2022

Teaching Broad Reasoning Skills via Decomposition-Guided Contexts

Question-answering datasets require a broad set of reasoning skills. We ...