DeepAI AI Chat
Log In Sign Up

Truth Serums for Massively Crowdsourced Evaluation Tasks

by   Vijay Kamble, et al.
Stanford University

A major challenge in crowdsourcing evaluation tasks like labeling objects, grading assignments in online courses, etc., is that of eliciting truthful responses from agents in the absence of verifiability. In this paper, we propose new reward mechanisms for such settings that, unlike many previously studied mechanisms, impose minimal assumptions on the structure and knowledge of the underlying generating model, can account for heterogeneity in the agents' abilities, require no extraneous elicitation from them, and furthermore allow their beliefs to be (almost) arbitrary. These mechanisms have the simple and intuitive structure of an output agreement mechanism: an agent gets a reward if her evaluation matches that of her peer, but unlike the classic output agreement mechanism, this reward is not the same across evaluations, but is inversely proportional to an appropriately defined popularity index of each evaluation. The popularity indices are computed by leveraging the existence of a large number of similar tasks, which is a typical characteristic of these settings. Experiments performed on MTurk workers demonstrate higher efficacy (with a p-value of 0.02) of these mechanisms in inducing truthful behavior compared to the state of the art.


page 1

page 2

page 3

page 4


Partial Truthfulness in Minimal Peer Prediction Mechanisms with Limited Knowledge

We study minimal single-task peer prediction mechanisms that have limite...

Collusion-proof And Sybil-proof Reward Mechanisms For Query Incentive Networks

This paper explores reward mechanisms for a query incentive network in w...

Information Signal Design for Incentivizing Team Formation

We study the use of Bayesian persuasion (i.e., strategic use of informat...

REFORM: Reputation Based Fair and Temporal Reward Framework for Crowdsourcing

Crowdsourcing is an effective method to collect data by employing distri...

Dominantly Truthful Multi-task Peer Prediction with a Constant Number of Tasks

In the setting where participants are asked multiple similar possibly su...

Civic Crowdfunding for Agents with Negative Valuations and Agents with Asymmetric Beliefs

In the last decade, civic crowdfunding has proved to be effective in gen...