Exact Exponent in Optimal Rates for Crowdsourcing

05/25/2016
by   Chao Gao, et al.
0

In many machine learning applications, crowdsourcing has become the primary means for label collection. In this paper, we study the optimal error rate for aggregating labels provided by a set of non-expert workers. Under the classic Dawid-Skene model, we establish matching upper and lower bounds with an exact exponent mI(π) in which m is the number of workers and I(π) the average Chernoff information that characterizes the workers' collective ability. Such an exact characterization of the error exponent allows us to state a precise sample size requirement m>1/I(π)1/ϵ in order to achieve an ϵ misclassification error. In addition, our results imply the optimality of various EM algorithms for crowdsourcing initialized by consistent estimators.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/05/2022

Unsupervised Crowdsourcing with Accuracy and Cost Guarantees

We consider the problem of cost-optimal utilization of a crowdsourcing p...
research
02/03/2015

Cheaper and Better: Selecting Good Workers for Crowdsourcing

Crowdsourcing provides a popular paradigm for data collection at scale. ...
research
12/20/2019

Large Deviations Behavior of the Logarithmic Error Probability of Random Codes

This work studies the deviations of the error exponent of the constant c...
research
10/28/2019

Finite Sample-Size Regime of Testing Against Independence with Communication Constraints

The central problem of Hypothesis Testing (HT) consists in determining t...
research
06/01/2016

A Minimax Optimal Algorithm for Crowdsourcing

We consider the problem of accurately estimating the reliability of work...
research
02/14/2016

Embracing Error to Enable Rapid Crowdsourcing

Microtask crowdsourcing has enabled dataset advances in social science a...
research
04/05/2023

Maxflow-Based Bounds for Low-Rate Information Propagation over Noisy Networks

We study error exponents for the problem of low-rate communication over ...

Please sign up or login with your details

Forgot password? Click here to reset