QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships

by   Oyvind Tafjord, et al.

Many natural language questions require recognizing and reasoning with qualitative relationships (e.g., in science, economics, and medicine), but are challenging to answer with corpus-based methods. Qualitative modeling provides tools that support such reasoning, but the semantic parsing task of mapping questions into those models has formidable challenges. We present QuaRel, a dataset of diverse story questions involving qualitative relationships that characterize these challenges, and techniques that begin to address them. The dataset has 2771 questions relating 19 different types of quantities. For example, "Jenny observes that the robot vacuum cleaner moves slower on the living room carpet than on the bedroom carpet. Which carpet has more friction?" We contribute (1) a simple and flexible conceptual framework for representing these kinds of questions; (2) the QuaRel dataset, including logical forms, exemplifying the parsing challenges; and (3) two novel models for this task, built as extensions of type-constrained semantic parsing. The first of these models (called QuaSP+) significantly outperforms off-the-shelf tools on QuaRel. The second (QuaSP+Zero) demonstrates zero-shot capability, i.e., the ability to handle new qualitative relationships without requiring additional training data, something not possible with previous models. This work thus makes inroads into answering complex, qualitative questions that require reasoning, and scaling to new relationships at low cost. The dataset and models are available at http://data.allenai.org/quarel.


page 1

page 2

page 3

page 4


ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models

We explore the use of large language models (LLMs) for zero-shot semanti...

QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions

We introduce the first open-domain dataset, called QuaRTz, for reasoning...

A Generate-Validate Approach to Answering Questions about Qualitative Relationships

Qualitative relationships describe how increasing or decreasing one prop...

Answering Complex Questions over Text by Hybrid Question Parsing and Execution

The dominant paradigm of textual question answering systems is based on ...

Can GPT-3 Perform Statutory Reasoning?

Statutory reasoning is the task of reasoning with facts and statutes, wh...

Tisane: Authoring Statistical Models via Formal Reasoning from Conceptual and Data Relationships

Proper statistical modeling incorporates domain theory about how concept...

Probing the Quantitative-Qualitative Divide in Probabilistic Reasoning

This paper explores the space of (propositional) probabilistic logical l...

Please sign up or login with your details

Forgot password? Click here to reset