Approval Voting and Incentives in Crowdsourcing

02/19/2015
by   Nihar B. Shah, et al.
0

The growing need for labeled training data has made crowdsourcing an important part of machine learning. The quality of crowdsourced labels is, however, adversely affected by three factors: (1) the workers are not experts; (2) the incentives of the workers are not aligned with those of the requesters; and (3) the interface does not allow workers to convey their knowledge accurately, by forcing them to make a single choice among a set of options. In this paper, we address these issues by introducing approval voting to utilize the expertise of workers who have partial knowledge of the true answer, and coupling it with a ("strictly proper") incentive-compatible compensation mechanism. We show rigorous theoretical guarantees of optimality of our mechanism together with a simple axiomatic characterization. We also conduct preliminary empirical studies on Amazon Mechanical Turk which validate our approach.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/26/2018

Millionaire: A Hint-guided Approach for Crowdsourcing

Modern machine learning is migrating to the era of complex models, which...
research
09/30/2016

Characterization of experts in crowdsourcing platforms

Crowdsourcing platforms enable to propose simple human intelligence task...
research
06/08/2019

Doubly Robust Crowdsourcing

Large-scale labeled datasets are the indispensable fuel that ignites the...
research
11/09/2021

A Survey of NLP-Related Crowdsourcing HITs: what works and what does not

Crowdsourcing requesters on Amazon Mechanical Turk (AMT) have raised que...
research
01/02/2019

Ethically Aligned Opportunistic Scheduling for Productive Laziness

In artificial intelligence (AI) mediated workforce management systems (e...
research
11/11/2021

Full Characterization of Adaptively Strong Majority Voting in Crowdsourcing

A commonly used technique for quality control in crowdsourcing is to tas...
research
09/12/2017

Certified Computation in Crowdsourcing

A wide range of learning tasks require human input in labeling massive d...

Please sign up or login with your details

Forgot password? Click here to reset