Crowdsourcing Control: Moving Beyond Multiple Choice

by   Christopher H. Lin, et al.

To ensure quality results from crowdsourced tasks, requesters often aggregate worker responses and use one of a plethora of strategies to infer the correct answer from the set of noisy responses. However, all current models assume prior knowledge of all possible outcomes of the task. While not an unreasonable assumption for tasks that can be posited as multiple-choice questions (e.g. n-ary classification), we observe that many tasks do not naturally fit this paradigm, but instead demand a free-response formulation where the outcome space is of infinite size (e.g. audio transcription). We model such tasks with a novel probabilistic graphical model, and design and implement LazySusan, a decision-theoretic controller that dynamically requests responses as necessary in order to infer answers to these tasks. We also design an EM algorithm to jointly learn the parameters of our model while inferring the correct answers to multiple tasks at a time. Live experiments on Amazon Mechanical Turk demonstrate the superiority of LazySusan at solving SAT Math questions, eliminating 83.2 the state-ofthe-art strategy, majority-voting. We also show in live experiments that our EM algorithm outperforms majority-voting on a visualization task that we design.



page 1

page 2

page 3

page 4


How To Grade a Test Without Knowing the Answers --- A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing

We propose a new probabilistic graphical model that jointly models the d...

Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems

Crowdsourcing systems, in which numerous tasks are electronically distri...

When majority voting fails: Comparing quality assurance methods for noisy human computation environment

Quality assurance remains a key topic in human computation research. Pri...

Eliciting Thinking Hierarchy without Prior

A key challenge in crowdsourcing is that majority may make systematic mi...

A statistical model for aggregating judgments by incorporating peer predictions

We propose a probabilistic model to aggregate the answers of respondents...

Unifying Framework for Crowd-sourcing via Graphon Estimation

We consider the question of inferring true answers associated with tasks...

Investigating Crowdsourcing to Generate Distractors for Multiple-Choice Assessments

We present and analyze results from a pilot study that explores how crow...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.