DeepAI
Log In Sign Up

Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models

05/10/2021
by   Daniel Cohen, et al.
0

In any ranking system, the retrieval model outputs a single score for a document based on its belief on how relevant it is to a given search query. While retrieval models have continued to improve with the introduction of increasingly complex architectures, few works have investigated a retrieval model's belief in the score beyond the scope of a single value. We argue that capturing the model's uncertainty with respect to its own scoring of a document is a critical aspect of retrieval that allows for greater use of current models across new document distributions, collections, or even improving effectiveness for down-stream tasks. In this paper, we address this problem via an efficient Bayesian framework for retrieval models which captures the model's belief in the relevance score through a stochastic process while adding only negligible computational overhead. We evaluate this belief via a ranking based calibration metric showing that our approximate Bayesian framework significantly improves a retrieval model's ranking effectiveness through a risk aware reranking as well as its confidence calibration. Lastly, we demonstrate that this additional uncertainty information is actionable and reliable on down-stream tasks represented via cutoff prediction.

READ FULL TEXT

page 1

page 2

page 3

page 4

12/16/2021

CODER: An efficient framework for improving retrieval through COntextualized Document Embedding Reranking

We present a framework for improving the performance of a wide class of ...
06/25/2021

A Modern Perspective on Query Likelihood with Deep Generative Retrieval Models

Existing neural ranking models follow the text matching paradigm, where ...
01/12/2021

On the Calibration and Uncertainty of Neural Learning to Rank Models

According to the Probability Ranking Principle (PRP), ranking documents ...
07/11/2018

A Formal Account of Effectiveness Evaluation and Ranking Fusion

This paper proposes a theoretical framework which models the information...
07/27/2022

UNIMIB at TREC 2021 Clinical Trials Track

This contribution summarizes the participation of the UNIMIB team to the...
10/24/2022

Reliability-Aware Prediction via Uncertainty Learning for Person Image Retrieval

Current person image retrieval methods have achieved great improvements ...
04/23/2022

Dual Skipping Guidance for Document Retrieval with Learned Sparse Representations

This paper proposes a dual skipping guidance scheme with hybrid scoring ...