Beyond monetary incentives: experiments in paid microtask contests modelled as continuous-time markov chains

01/17/2019
by   Oluwaseyi Feyisetan, et al.
0

In this paper, we aim to gain a better understanding into how paid microtask crowdsourcing could leverage its appeal and scaling power by using contests to boost crowd performance and engagement. We introduce our microtask-based annotation platform Wordsmith, which features incentives such as points, leaderboards and badges on top of financial remuneration. Our analysis focuses on a particular type of incentive, contests, as a means to apply crowdsourcing in near-real-time scenarios, in which requesters need labels quickly. We model crowdsourcing contests as a continuous-time Markov chain with the objective to maximise the output of the crowd workers, while varying a parameter which determines whether a worker is eligible for a reward based on their present rank on the leaderboard. We conduct empirical experiments in which crowd workers recruited from CrowdFlower carry out annotation microtasks on Wordsmith - in our case, to identify named entities in a stream of Twitter posts. In the experimental conditions, we test different reward spreads and record the total number of annotations received. We compare the results against a control condition in which the same annotation task was completed on CrowdFlower without a time or contest constraint. The experiments show that rewarding only the best contributors in a live contest could be a viable model to deliver results faster, though quality might suffer for particular types of annotation tasks. Increasing the reward spread leads to more work being completed, especially by the top contestants. Overall, the experiments shed light on possible design improvements of paid microtasks platforms to boost task performance and speed, and make the overall experience more fair and interesting for crowd workers.

READ FULL TEXT

page 13

page 16

page 17

research
11/07/2021

Crowdsourcing with Meta-Workers: A New Way to Save the Budget

Due to the unreliability of Internet workers, it's difficult to complete...
research
03/27/2023

ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

Many NLP applications require manual data annotations for a variety of t...
research
05/26/2021

Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing

Extensive work has argued in favour of paying crowd workers a wage that ...
research
09/17/2021

A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python

Quality control is a crux of crowdsourcing. While most means for quality...
research
12/20/2020

Exploring Effectiveness of Inter-Microtask Qualification Tests in Crowdsourcing

Qualification tests in crowdsourcing are often used to pre-filter worker...
research
12/14/2018

Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics

Micro-task crowdsourcing is an international phenomenon that has emerged...
research
03/07/2023

Crowdsourcing in Precision Healthcare: Short Review

The age of deep learning has brought high-performing diagnostic models f...

Please sign up or login with your details

Forgot password? Click here to reset