Calibrating Trust of Multi-Hop Question Answering Systems with Decompositional Probes

by   Kaige Xie, et al.

Multi-hop Question Answering (QA) is a challenging task since it requires an accurate aggregation of information from multiple context paragraphs and a thorough understanding of the underlying reasoning chains. Recent work in multi-hop QA has shown that performance can be boosted by first decomposing the questions into simpler, single-hop questions. In this paper, we explore one additional utility of the multi-hop decomposition from the perspective of explainable NLP: to create explanation by probing a neural QA model with them. We hypothesize that in doing so, users will be better able to construct a mental model of when the underlying QA system will give the correct answer. Through human participant studies, we verify that exposing the decomposition probes and answers to the probes to users can increase their ability to predict system performance on a question instance basis. We show that decomposition is an effective form of probing QA systems as well as a promising approach to explanation generation. In-depth analyses show the need for improvements in decomposition systems.


page 1

page 2

page 3

page 4


Interpretable AMR-Based Question Decomposition for Multi-hop Question Answering

Effective multi-hop question answering (QA) requires reasoning over mult...

A Survey on Multi-hop Question Answering and Generation

The problem of Question Answering (QA) has attracted significant researc...

HOP, UNION, GENERATE: Explainable Multi-hop Reasoning without Rationale Supervision

Explainable multi-hop question answering (QA) not only predicts answers ...

A Road-map Towards Explainable Question Answering A Solution for Information Pollution

The increasing rate of information pollution on the Web requires novel s...

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding

Recently, there has been an increasing interest in building question ans...

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

A key limitation in current datasets for multi-hop reasoning is that the...

Best of Both Worlds: A Hybrid Approach for Multi-Hop Explanation with Declarative Facts

Language-enabled AI systems can answer complex, multi-hop questions to h...

Please sign up or login with your details

Forgot password? Click here to reset