A Margin-based MLE for Crowdsourced Partial Ranking

07/29/2018
by   Qianqian Xu, et al.
0

A preference order or ranking aggregated from pairwise comparison data is commonly understood as a strict total order. However, in real-world scenarios, some items are intrinsically ambiguous in comparisons, which may very well be an inherent uncertainty of the data. In this case, the conventional total order ranking can not capture such uncertainty with mere global ranking or utility scores. In this paper, we are specifically interested in the recent surge in crowdsourcing applications to predict partial but more accurate (i.e., making less incorrect statements) orders rather than complete ones. To do so, we propose a novel framework to learn some probabilistic models of partial orders as a margin-based Maximum Likelihood Estimate (MLE) method. We prove that the induced MLE is a joint convex optimization problem with respect to all the parameters, including the global ranking scores and margin parameter. Moreover, three kinds of generalized linear models are studied, including the basic uniform model, Bradley-Terry model, and Thurstone-Mosteller model, equipped with some theoretical analysis on FDR and Power control for the proposed methods. The validity of these models are supported by experiments with both simulated and real-world datasets, which shows that the proposed models exhibit improvements compared with traditional state-of-the-art algorithms.

READ FULL TEXT

page 1

page 7

research
10/14/2019

iSplit LBI: Individualized Partial Ranking with Ties via Split LBI

Due to the inherent uncertainty of data, the problem of predicting parti...
research
03/08/2018

From Social to Individuals: a Parsimonious Path of Multi-level Models for Crowdsourced Preference Aggregation

In crowdsourced preference aggregation, it is often assumed that all the...
research
07/07/2021

Uncertainty in Ranking

Ranks estimated from data are uncertain and this poses a challenge in ma...
research
04/26/2023

Diffsurv: Differentiable sorting for censored time-to-event data

Survival analysis is a crucial semi-supervised task in machine learning ...
research
09/15/2014

Crowdsourcing Pareto-Optimal Object Finding by Pairwise Comparisons

This is the first study on crowdsourcing Pareto-optimal object finding, ...
research
12/23/2017

Angle-Based Models for Ranking Data

A new class of general exponential ranking models is introduced which we...
research
06/13/2018

Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings

Cardinal scores (numeric ratings) collected from people are well known t...

Please sign up or login with your details

Forgot password? Click here to reset