DeepAI AI Chat
Log In Sign Up

Octopus: A Framework for Cost-Quality-Time Optimization in Crowdsourcing

by   Karan Goel, et al.
Indian Institute of Technology Delhi

We present Octopus, an AI agent to jointly balance three conflicting task objectives on a micro-crowdsourcing marketplace - the quality of work, total cost incurred, and time to completion. Previous control agents have mostly focused on cost-quality, or cost-time tradeoffs, but not on directly controlling all three in concert. A naive formulation of three-objective optimization is intractable; Octopus takes a hierarchical POMDP approach, with three different components responsible for setting the pay per task, selecting the next task, and controlling task-level quality. We demonstrate that Octopus significantly outperforms existing state-of-the-art approaches on real experiments. We also deploy Octopus on Amazon Mechanical Turk, showing its ability to manage tasks in a real-world dynamic setting.


page 1

page 2

page 3

page 4


On-the-Job Learning with Bayesian Decision Theory

Our goal is to deploy a high-accuracy system starting with zero training...

Towards AI-Empowered Crowdsourcing

Crowdsourcing, in which human intelligence and productivity is dynamical...

Milimili. Collecting Parallel Data via Crowdsourcing

We present a methodology for gathering a parallel corpus through crowdso...

It's about time: Online Macrotask Sequencing in Expert Crowdsourcing

We introduce the problem of Task Assignment and Sequencing (TAS), which ...

Distributional Ground Truth: Non-Redundant Crowdsourcing Data Quality Control in UI Labeling Tasks

HCI increasingly employs Machine Learning and Image Recognition, in part...

Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning

This paper presents a Discriminative Deep Dyna-Q (D3Q) approach to impro...

CrowdSim: A Hybrid Simulation Model for Failure Prediction in Crowdsourced Software Development

A typical crowdsourcing software development(CSD) marketplace consists o...