Log In Sign Up

Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models

by   Daniel Cohen, et al.

In any ranking system, the retrieval model outputs a single score for a document based on its belief on how relevant it is to a given search query. While retrieval models have continued to improve with the introduction of increasingly complex architectures, few works have investigated a retrieval model's belief in the score beyond the scope of a single value. We argue that capturing the model's uncertainty with respect to its own scoring of a document is a critical aspect of retrieval that allows for greater use of current models across new document distributions, collections, or even improving effectiveness for down-stream tasks. In this paper, we address this problem via an efficient Bayesian framework for retrieval models which captures the model's belief in the relevance score through a stochastic process while adding only negligible computational overhead. We evaluate this belief via a ranking based calibration metric showing that our approximate Bayesian framework significantly improves a retrieval model's ranking effectiveness through a risk aware reranking as well as its confidence calibration. Lastly, we demonstrate that this additional uncertainty information is actionable and reliable on down-stream tasks represented via cutoff prediction.


page 1

page 2

page 3

page 4


CODER: An efficient framework for improving retrieval through COntextualized Document Embedding Reranking

We present a framework for improving the performance of a wide class of ...

A Modern Perspective on Query Likelihood with Deep Generative Retrieval Models

Existing neural ranking models follow the text matching paradigm, where ...

On the Calibration and Uncertainty of Neural Learning to Rank Models

According to the Probability Ranking Principle (PRP), ranking documents ...

A Formal Account of Effectiveness Evaluation and Ranking Fusion

This paper proposes a theoretical framework which models the information...

UNIMIB at TREC 2021 Clinical Trials Track

This contribution summarizes the participation of the UNIMIB team to the...

Reliability-Aware Prediction via Uncertainty Learning for Person Image Retrieval

Current person image retrieval methods have achieved great improvements ...

Dual Skipping Guidance for Document Retrieval with Learned Sparse Representations

This paper proposes a dual skipping guidance scheme with hybrid scoring ...