How Many Workers to Ask? Adaptive Exploration for Collecting High Quality Labels

by   Ittai Abraham, et al.

Crowdsourcing has been part of the IR toolbox as a cheap and fast mechanism to obtain labels for system development and evaluation. Successful deployment of crowdsourcing at scale involves adjusting many variables, a very important one being the number of workers needed per human intelligence task (HIT). We consider the crowdsourcing task of learning the answer to simple multiple-choice HITs, which are representative of many relevance experiments. In order to provide statistically significant results, one often needs to ask multiple workers to answer the same HIT. A stopping rule is an algorithm that, given a HIT, decides for any given set of worker answers if the system should stop and output an answer or iterate and ask one more worker. Knowing the historic performance of a worker in the form of a quality score can be beneficial in such a scenario. In this paper we investigate how to devise better stopping rules given such quality scores. We also suggest adaptive exploration as a promising approach for scalable and automatic creation of ground truth. We conduct a data analysis on an industrial crowdsourcing platform, and use the observations from this analysis to design new stopping rules that use the workers' quality scores in a non-trivial manner. We then perform a simulation based on a real-world workload, showing that our algorithm performs better than the more naive approaches.



There are no comments yet.


page 4


Millionaire: A Hint-guided Approach for Crowdsourcing

Modern machine learning is migrating to the era of complex models, which...

CrowdMOT: Crowdsourcing Strategies for Tracking Multiple Objects in Videos

Crowdsourcing is a valuable approach for tracking objects in videos in a...

A General Early-Stopping Module for Crowdsourced Ranking

Crowdsourcing can be used to determine a total order for an object set (...

Truth Inference on Sparse Crowdsourcing Data with Local Differential Privacy

Crowdsourcing has arisen as a new problem-solving paradigm for tasks tha...

Cheaper and Better: Selecting Good Workers for Crowdsourcing

Crowdsourcing provides a popular paradigm for data collection at scale. ...

Toward Effective Automated Content Analysis via Crowdsourcing

Many computer scientists use the aggregated answers of online workers to...

Achieving Budget-optimality with Adaptive Schemes in Crowdsourcing

Crowdsourcing platforms provide marketplaces where task requesters can p...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.