On the Calibration and Uncertainty of Neural Learning to Rank Models

01/12/2021
by   Gustavo Penha, et al.
0

According to the Probability Ranking Principle (PRP), ranking documents in decreasing order of their probability of relevance leads to an optimal document ranking for ad-hoc retrieval. The PRP holds when two conditions are met: [C1] the models are well calibrated, and, [C2] the probabilities of relevance are reported with certainty. We know however that deep neural networks (DNNs) are often not well calibrated and have several sources of uncertainty, and thus [C1] and [C2] might not be satisfied by neural rankers. Given the success of neural Learning to Rank (L2R) approaches-and here, especially BERT-based approaches-we first analyze under which circumstances deterministic, i.e. outputs point estimates, neural rankers are calibrated. Then, motivated by our findings we use two techniques to model the uncertainty of neural rankers leading to the proposed stochastic rankers, which output a predictive distribution of relevance as opposed to point estimates. Our experimental results on the ad-hoc retrieval task of conversation response ranking reveal that (i) BERT-based rankers are not robustly calibrated and that stochastic BERT-based rankers yield better calibration; and (ii) uncertainty estimation is beneficial for both risk-aware neural ranking, i.e.taking into account the uncertainty when ranking documents, and for predicting unanswerable conversational contexts.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/17/2021

Co-BERT: A Context-Aware BERT Retrieval Model Incorporating Local and Query-specific Context

BERT-based text ranking models have dramatically advanced the state-of-t...
research
04/16/2019

Understanding the Behaviors of BERT in Ranking

This paper studies the performances and behaviors of BERT in ranking tas...
research
07/19/2021

Transport away your problems: Calibrating stochastic simulations with optimal transport

Stochastic simulators are an indispensable tool in many branches of scie...
research
01/28/2021

A Graph-based Relevance Matching Model for Ad-hoc Retrieval

To retrieve more relevant, appropriate and useful documents given a quer...
research
03/15/2023

Efficient Uncertainty Estimation with Gaussian Process for Reliable Dialog Response Retrieval

Deep neural networks have achieved remarkable performance in retrieval-b...
research
05/27/2021

Calibrating Over-Parametrized Simulation Models: A Framework via Eligibility Set

Stochastic simulation aims to compute output performance for complex mod...
research
05/10/2021

Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models

In any ranking system, the retrieval model outputs a single score for a ...

Please sign up or login with your details

Forgot password? Click here to reset