DeepAI AI Chat
Log In Sign Up

Beyond monetary incentives: experiments in paid microtask contests modelled as continuous-time markov chains

by   Oluwaseyi Feyisetan, et al.

In this paper, we aim to gain a better understanding into how paid microtask crowdsourcing could leverage its appeal and scaling power by using contests to boost crowd performance and engagement. We introduce our microtask-based annotation platform Wordsmith, which features incentives such as points, leaderboards and badges on top of financial remuneration. Our analysis focuses on a particular type of incentive, contests, as a means to apply crowdsourcing in near-real-time scenarios, in which requesters need labels quickly. We model crowdsourcing contests as a continuous-time Markov chain with the objective to maximise the output of the crowd workers, while varying a parameter which determines whether a worker is eligible for a reward based on their present rank on the leaderboard. We conduct empirical experiments in which crowd workers recruited from CrowdFlower carry out annotation microtasks on Wordsmith - in our case, to identify named entities in a stream of Twitter posts. In the experimental conditions, we test different reward spreads and record the total number of annotations received. We compare the results against a control condition in which the same annotation task was completed on CrowdFlower without a time or contest constraint. The experiments show that rewarding only the best contributors in a live contest could be a viable model to deliver results faster, though quality might suffer for particular types of annotation tasks. Increasing the reward spread leads to more work being completed, especially by the top contestants. Overall, the experiments shed light on possible design improvements of paid microtasks platforms to boost task performance and speed, and make the overall experience more fair and interesting for crowd workers.


page 13

page 16

page 17


Crowdsourcing with Meta-Workers: A New Way to Save the Budget

Due to the unreliability of Internet workers, it's difficult to complete...

ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

Many NLP applications require manual data annotations for a variety of t...

Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing

Extensive work has argued in favour of paying crowd workers a wage that ...

A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python

Quality control is a crux of crowdsourcing. While most means for quality...

Exploring Effectiveness of Inter-Microtask Qualification Tests in Crowdsourcing

Qualification tests in crowdsourcing are often used to pre-filter worker...

Multimedia Crowdsourcing with Bounded Rationality: A Cognitive Hierarchy Prospective

In multimedia crowdsourcing, the requester's quality requirements and re...

Evaluating Complex Task through Crowdsourcing: Multiple Views Approach

With the popularity of massive open online courses, grading through crow...