Unsupervised Evaluation for Question Answering with Transformers

by   Lukas Muttenthaler, et al.

It is challenging to automatically evaluate the answer of a QA model at inference time. Although many models provide confidence scores, and simple heuristics can go a long way towards indicating answer correctness, such measures are heavily dataset-dependent and are unlikely to generalize. In this work, we begin by investigating the hidden representations of questions, answers, and contexts in transformer-based QA architectures. We observe a consistent pattern in the answer representations, which we show can be used to automatically evaluate whether or not a predicted answer span is correct. Our method does not require any labeled data and outperforms strong heuristic baselines, across 2 datasets and 7 domains. We are able to predict whether or not a model's answer is correct with 91.37 accuracy on SubjQA. We expect that this method will have broad applications, e.g., in the semi-automatic development of QA datasets


page 1

page 2

page 3

page 4


Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering

In this paper we propose a novel approach towards improving the efficien...

Subjective Question Answering: Deciphering the inner workings of Transformers in the realm of subjectivity

Understanding subjectivity demands reasoning skills beyond the realm of ...

ASQA: Factoid Questions Meet Long-Form Answers

An abundance of datasets and availability of reliable evaluation metrics...

Towards Automatic Generation of Questions from Long Answers

Automatic question generation (AQG) has broad applicability in domains s...

Unsupervised Question Answering via Answer Diversifying

Unsupervised question answering is an attractive task due to its indepen...

Question Classification with Deep Contextualized Transformer

The latest work for Question and Answer problems is to use the Stanford ...

Riiid! Answer Correctness Prediction Kaggle Challenge: 4th Place Solution Summary

This paper presents my solution to the challenge "Riiid! Answer Correctn...