A General Early-Stopping Module for Crowdsourced Ranking

by   Caihua Shan, et al.

Crowdsourcing can be used to determine a total order for an object set (e.g., the top-10 NBA players) based on crowd opinions. This ranking problem is often decomposed into a set of microtasks (e.g., pairwise comparisons). These microtasks are passed to a large number of workers and their answers are aggregated to infer the ranking. The number of microtasks depends on the budget allocated for the problem. Intuitively, the higher the number of microtask answers, the more accurate the ranking becomes. However, it is often hard to decide the budget required for an accurate ranking. We study how a ranking process can be terminated early, and yet achieve a high-quality ranking and great savings in the budget. We use statistical tools to estimate the quality of the ranking result at any stage of the crowdsourcing process and terminate the process as soon as the desired quality is achieved. Our proposed early-stopping module can be seamlessly integrated with most existing inference algorithms and task assignment methods. We conduct extensive experiments and show that our early-stopping module is better than other existing general stopping criteria. We also implement a prototype system to demonstrate the usability and effectiveness of our approach in practice.


page 1

page 2

page 3

page 4


Bayesian Decision Process for Cost-Efficient Dynamic Ranking via Crowdsourcing

Rank aggregation based on pairwise comparisons over a set of items has a...

How Many Workers to Ask? Adaptive Exploration for Collecting High Quality Labels

Crowdsourcing has been part of the IR toolbox as a cheap and fast mechan...

Searching for a higher power in the human evaluation of MT

In MT evaluation, pairwise comparisons are conducted to identify the bet...

HodgeRank with Information Maximization for Crowdsourced Pairwise Ranking Aggregation

Recently, crowdsourcing has emerged as an effective paradigm for human-p...

Fairness-driven Skilled Task Assignment with Extra Budget in Spatial Crowdsourcing

With the prevalence of mobile devices and ubiquitous wireless networks, ...

Query-level Early Exit for Additive Learning-to-Rank Ensembles

Search engine ranking pipelines are commonly based on large ensembles of...

Please sign up or login with your details

Forgot password? Click here to reset