Analysis of Crowdsourced Sampling Strategies for HodgeRank with Sparse Random Graphs

02/28/2015
by   Braxton Osting, et al.
0

Crowdsourcing platforms are now extensively used for conducting subjective pairwise comparison studies. In this setting, a pairwise comparison dataset is typically gathered via random sampling, either with or without replacement. In this paper, we use tools from random graph theory to analyze these two random sampling methods for the HodgeRank estimator. Using the Fiedler value of the graph as a measurement for estimator stability (informativeness), we provide a new estimate of the Fiedler value for these two random graph models. In the asymptotic limit as the number of vertices tends to infinity, we prove the validity of the estimate. Based on our findings, for a small number of items to be compared, we recommend a two-stage sampling strategy where a greedy sampling method is used initially and random sampling without replacement is used in the second stage. When a large number of items is to be compared, we recommend random sampling with replacement as this is computationally inexpensive and trivially parallelizable. Experiments on synthetic and real-world datasets support our analysis.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/01/2019

Parallel Weighted Random Sampling

Data structures for efficient sampling from a set of weighted items are ...
research
11/16/2020

On systems of quotas based on bankruptcy with a priori unions: estimating random arrival-style rules

This paper addresses a sampling procedure for estimating extensions of t...
research
08/29/2018

Consistent Sampling with Replacement

We describe a very simple method for `consistent sampling' that allows f...
research
11/04/2022

Significance improvement by randomized test in random sampling without replacement

This paper studies one-sided hypothesis testing under random sampling wi...
research
05/31/2023

Crowdsourcing subjective annotations using pairwise comparisons reduces bias and error compared to the majority-vote method

How to better reduce measurement variability and bias introduced by subj...
research
06/05/2023

A unified analysis of likelihood-based estimators in the Plackett–Luce model

The Plackett–Luce model is a popular approach for ranking data analysis,...
research
01/22/2021

New randomized response technique for estimating the population total of a quantitative variable

In this paper, a new randomized response technique aimed at protecting r...

Please sign up or login with your details

Forgot password? Click here to reset