DeepAI AI Chat
Log In Sign Up

SmoothI: Smooth Rank Indicators for Differentiable IR Metrics

by   Thibaut Thonet, et al.

Information retrieval (IR) systems traditionally aim to maximize metrics built on rankings, such as precision or NDCG. However, the non-differentiability of the ranking operation prevents direct optimization of such metrics in state-of-the-art neural IR models, which rely entirely on the ability to compute meaningful gradients. To address this shortcoming, we propose SmoothI, a smooth approximation of rank indicators that serves as a basic building block to devise differentiable approximations of IR metrics. We further provide theoretical guarantees on SmoothI and derived approximations, showing in particular that the approximation errors decrease exponentially with an inverse temperature-like hyperparameter that controls the quality of the approximations. Extensive experiments conducted on four standard learning-to-rank datasets validate the efficacy of the listwise losses based on SmoothI, in comparison to previously proposed ones. Additional experiments with a vanilla BERT ranking model on a text-based IR task also confirm the benefits of our listwise approach.


Optimize What You Evaluate With: A Simple Yet Effective Framework For Direct Optimization Of IR Metrics

Learning-to-rank has been intensively studied and has shown significantl...

OpenMatch: An Open-Source Package for Information Retrieval

Information Retrieval (IR) is an important task and can be used in many ...

Robust and Decomposable Average Precision for Image Retrieval

In image retrieval, standard evaluation metrics rely on score ranking, e...

New Insights into Metric Optimization for Ranking-based Recommendation

Direct optimization of IR metrics has often been adopted as an approach ...

Metric-agnostic Ranking Optimization

Ranking is at the core of Information Retrieval. Classic ranking optim...

Optimizing Rank-based Metrics with Blackbox Differentiation

Rank-based metrics are some of the most widely used criteria for perform...

Batch Evaluation Metrics in Information Retrieval: Measures, Scales, and Meaning

A sequence of recent papers has considered the role of measurement scale...