What's in a Name? Answer Equivalence For Open-Domain Question Answering

09/11/2021
by   Chenglei Si, et al.
0

A flaw in QA evaluation is that annotations often only provide one gold answer. Thus, model predictions semantically equivalent to the answer but superficially different are considered incorrect. This work explores mining alias entities from knowledge bases and using them as additional gold answers (i.e., equivalent answers). We incorporate answers for two settings: evaluation with additional answers and model training with equivalent answers. We analyse three QA benchmarks: Natural Questions, TriviaQA, and SQuAD. Answer expansion increases the exact match score on all datasets for evaluation, while incorporating it helps model training over real-world datasets. We ensure the additional answers are valid through a human post hoc evaluation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/11/2023

Evaluating Open-Domain Question Answering in the Era of Large Language Models

Lexical matching remains the de facto evaluation method for open-domain ...
research
02/03/2023

LIQUID: A Framework for List Question Answering Dataset Generation

Question answering (QA) models often rely on large-scale training datase...
research
12/10/2021

Improving the Question Answering Quality using Answer Candidate Filtering based on Natural-Language Features

Software with natural-language user interfaces has an ever-increasing im...
research
03/14/2017

A statistical model for aggregating judgments by incorporating peer predictions

We propose a probabilistic model to aggregate the answers of respondents...
research
09/21/2020

UCD-CS at W-NUT 2020 Shared Task-3: A Text to Text Approach for COVID-19 Event Extraction on Social Media

In this paper, we describe our approach in the shared task: COVID-19 eve...
research
01/13/2022

Grow-and-Clip: Informative-yet-Concise Evidence Distillation for Answer Explanation

Interpreting the predictions of existing Question Answering (QA) models ...
research
01/12/2019

Semi-interactive Attention Network for Answer Understanding in Reverse-QA

Question answering (QA) is an important natural language processing (NLP...

Please sign up or login with your details

Forgot password? Click here to reset