Ranking with Confidence for Large Scale Comparison Data

by   Filipa Valdeira, et al.

In this work, we leverage a generative data model considering comparison noise to develop a fast, precise, and informative ranking algorithm from pairwise comparisons that produces a measure of confidence on each comparison. The problem of ranking a large number of items from noisy and sparse pairwise comparison data arises in diverse applications, like ranking players in online games, document retrieval or ranking human perceptions. Although different algorithms are available, we need fast, large-scale algorithms whose accuracy degrades gracefully when the number of comparisons is too small. Fitting our proposed model entails solving a non-convex optimization problem, which we tightly approximate by a sum of quasi-convex functions and a regularization term. Resorting to an iterative reweighted minimization and the Primal-Dual Hybrid Gradient method, we obtain PD-Rank, achieving a Kendall tau 0.1 higher than all comparing methods, even for 10% of wrong comparisons in simulated data matching our data model, and leading in accuracy if data is generated according to the Bradley-Terry model, in both cases faster by one order of magnitude, in seconds. In real data, PD-Rank requires less computational time to achieve the same Kendall tau than active learning methods.


Approximate Ranking from Pairwise Comparisons

A common problem in machine learning is to rank a set of n items based o...

A Nearly Instance Optimal Algorithm for Top-k Ranking under the Multinomial Logit Model

We study the active learning problem of top-k ranking from multi-wise co...

Bayesian Decision Process for Cost-Efficient Dynamic Ranking via Crowdsourcing

Rank aggregation based on pairwise comparisons over a set of items has a...

Optimal Data Collection For Informative Rankings Expose Well-Connected Graphs

Given a graph where vertices represent alternatives and arcs represent p...

Optimal Full Ranking from Pairwise Comparisons

We consider the problem of ranking n players from partial pairwise compa...

Just Sort It! A Simple and Effective Approach to Active Preference Learning

We address the problem of learning a ranking by using adaptively chosen ...

Confidence Ranking for CTR Prediction

Model evolution and constant availability of data are two common phenome...

Please sign up or login with your details

Forgot password? Click here to reset