Tuning Crowdsourced Human Computation

10/14/2016
by   Chen Cao, et al.
0

As the use of crowdsourcing increases, it is important to think about performance optimization. For this purpose, it is possible to think about each worker as a HPU(Human Processing Unit), and to draw inspiration from performance optimization on traditional computers or cloud nodes with CPUs. However, as we characterize HPUs in detail for this purpose, we find that there are important differences between CPUs and HPUs, leading to the need for completely new optimization algorithms. In this paper, we study the specific optimization problem of obtaining results fastest for a crowd sourced job with a fixed total budget. In crowdsourcing, jobs are usually broken down into sets of small tasks, which are assigned to workers one at a time. We consider three scenarios of increasing complexity: Identical Round Homogeneous tasks, Multiplex Round Homogeneous tasks, and Multiple Round Heterogeneous tasks. For each scenario, we analyze the stochastic behavior of the HPU clock-rate as a function of the remuneration offered. After that, we develop an optimum Budget Allocation strategy to minimize the latency for job completion. We validate our results through extensive simulations and experiments on Amazon Mechanical Turk.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/12/2016

A Truthful Mechanism with Biparameter Learning for Online Crowdsourcing

We study a problem of allocating divisible jobs, arriving online, to wor...
research
01/15/2016

It's about time: Online Macrotask Sequencing in Expert Crowdsourcing

We introduce the problem of Task Assignment and Sequencing (TAS), which ...
research
09/01/2017

Adversarial Task Allocation

The problem of allocating tasks to workers is of long standing fundament...
research
03/15/2020

Recommending Deployment Strategies for Collaborative Tasks

Our work contributes to aiding requesters in deploying collaborative tas...
research
03/12/2014

Statistical Decision Making for Optimal Budget Allocation in Crowd Labeling

In crowd labeling, a large amount of unlabeled data instances are outsou...
research
01/30/2017

Dynamic Task Allocation for Crowdsourcing Settings

We consider the problem of optimal budget allocation for crowdsourcing p...
research
06/16/2023

Evaluating hardware differences for crowdsourcing and traditional recruiting methods

The most frequently used method to collect research data online is crowd...

Please sign up or login with your details

Forgot password? Click here to reset