Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing

11/15/2014
by   Hongwei Li, et al.
0

Crowdsourcing has become an effective and popular tool for human-powered computation to label large datasets. Since the workers can be unreliable, it is common in crowdsourcing to assign multiple workers to one task, and to aggregate the labels in order to obtain results of high quality. In this paper, we provide finite-sample exponential bounds on the error rate (in probability and in expectation) of general aggregation rules under the Dawid-Skene crowdsourcing model. The bounds are derived for multi-class labeling, and can be used to analyze many aggregation methods, including majority voting, weighted majority voting and the oracle Maximum A Posteriori (MAP) rule. We show that the oracle MAP rule approximately optimizes our upper bound on the mean error rate of weighted majority voting in certain setting. We propose an iterative weighted majority voting (IWMV) method that optimizes the error rate bound and approximates the oracle MAP rule. Its one step version has a provable theoretical guarantee on the error rate. The IWMV method is intuitive and computationally simple. Experimental results on simulated and real data show that IWMV performs at least on par with the state-of-the-art methods, and it has a much lower computational cost (around one hundred times faster) than the state-of-the-art methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/10/2013

Error Rate Bounds in Crowdsourcing Models

Crowdsourcing is an effective tool for human-powered computation on many...
research
09/18/2023

New Bounds on the Accuracy of Majority Voting for Multi-Class Classification

Majority voting is a simple mathematical function that returns the value...
research
02/23/2016

A Streaming Algorithm for Crowdsourced Data Classification

We propose a streaming algorithm for the binary classification of data b...
research
02/13/2018

Analysis of Minimax Error Rate for Crowdsourcing and Its Application to Worker Clustering Model

While crowdsourcing has become an important means to label data, crowdwo...
research
01/17/2022

Adjudication with Rational Jurors

We analyze a mechanism for adjudication involving majority voting and ra...
research
09/29/2021

Error rate control for classification rules in multiclass mixture models

In the context of finite mixture models one considers the problem of cla...
research
12/02/2013

Consistency of weighted majority votes

We revisit the classical decision-theoretic problem of weighted expert v...

Please sign up or login with your details

Forgot password? Click here to reset