Log In Sign Up

Look at the First Sentence: Position Bias in Question Answering

by   Miyoung Ko, et al.

Many extractive question answering models are trained to predict start and end positions of answers. The choice of predicting answers as positions is mainly due to its simplicity and effectiveness. In this study, we hypothesize that when the distribution of the answer positions is highly skewed in the training set (e.g., answers lie only in the k-th sentence of each passage), QA models predicting answers as positions learn spurious positional cues and fail to give answers in different positions. We first illustrate this position bias in popular extractive QA models such as BiDAF and BERT and thoroughly examine how position bias propagates through each layer of BERT. To safely deliver position information without position bias, we train models with various de-biasing methods including entropy regularization and bias ensembling. Among them, we found that using the prior distribution of answer positions as a bias model is very effective at reducing position bias recovering the performance of BERT from 35.24


Look to the Right: Mitigating Relative Position Bias in Extractive Question Answering

Extractive question answering (QA) models tend to exploit spurious corre...

SPBERTQA: A Two-Stage Question Answering System Based on Sentence Transformers for Medical Texts

Question answering (QA) systems have gained explosive attention in recen...

Mitigating the Position Bias of Transformer Models in Passage Re-Ranking

Supervised machine learning models and their evaluation strongly depends...

Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering

BERT model has been successfully applied to open-domain QA tasks. Howeve...

Quantifying the Impact of Cognitive Biases in Question-Answering Systems

Crowdsourcing can identify high-quality solutions to problems; however, ...

Rethinking the objectives of extractive question answering

This paper describes two generally applicable approaches towards the sig...

Uncovering the epistemological and ontological assumptions of software designers

The ontological and epistemological positions adopted by information sys...