Mitigating the Position Bias of Transformer Models in Passage Re-Ranking

01/18/2021
by   Sebastian Hofstätter, et al.
0

Supervised machine learning models and their evaluation strongly depends on the quality of the underlying dataset. When we search for a relevant piece of information it may appear anywhere in a given passage. However, we observe a bias in the position of the correct answer in the text in two popular Question Answering datasets used for passage re-ranking. The excessive favoring of earlier positions inside passages is an unwanted artefact. This leads to three common Transformer-based re-ranking models to ignore relevant parts in unseen passages. More concerningly, as the evaluation set is taken from the same biased distribution, the models overfitting to that bias overestimate their true effectiveness. In this work we analyze position bias on datasets, the contextualized representations, and their effect on retrieval results. We propose a debiasing method for retrieval datasets. Our results show that a model trained on a position-biased dataset exhibits a significant decrease in re-ranking effectiveness when evaluated on a debiased dataset. We demonstrate that by mitigating the position bias, Transformer-based re-ranking models are equally effective on a biased and debiased dataset, as well as more effective in a transfer-learning setting between two differently biased datasets.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

02/28/2020

Handling Position Bias for Unbiased Learning to Rank in Hotels Search

Nowadays, search ranking and recommendation systems rely on a lot of dat...
04/30/2020

Look at the First Sentence: Position Bias in Question Answering

Many extractive question answering models are trained to predict start a...
05/01/2020

Do Neural Ranking Models Intensify Gender Bias?

Concerns regarding the footprint of societal biases in information retri...
08/18/2021

Show or Tell? Visual and Verbal Representations Bias Position Recall

When we view visualizations, we not only have a visual representation of...
06/10/2021

Deep Position-wise Interaction Network for CTR Prediction

Click-through rate (CTR) prediction plays an important role in online ad...
09/01/2021

Don't Discard All the Biased Instances: Investigating a Core Assumption in Dataset Bias Mitigation Techniques

Existing techniques for mitigating dataset bias often leverage a biased ...
11/18/2021

Quality and Cost Trade-offs in Passage Re-ranking Task

Deep learning models named transformers achieved state-of-the-art result...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.