DeepAI AI Chat
Log In Sign Up

Octopus: A Framework for Cost-Quality-Time Optimization in Crowdsourcing

02/12/2017
by   Karan Goel, et al.
Indian Institute of Technology Delhi
0

We present Octopus, an AI agent to jointly balance three conflicting task objectives on a micro-crowdsourcing marketplace - the quality of work, total cost incurred, and time to completion. Previous control agents have mostly focused on cost-quality, or cost-time tradeoffs, but not on directly controlling all three in concert. A naive formulation of three-objective optimization is intractable; Octopus takes a hierarchical POMDP approach, with three different components responsible for setting the pay per task, selecting the next task, and controlling task-level quality. We demonstrate that Octopus significantly outperforms existing state-of-the-art approaches on real experiments. We also deploy Octopus on Amazon Mechanical Turk, showing its ability to manage tasks in a real-world dynamic setting.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/10/2015

On-the-Job Learning with Bayesian Decision Theory

Our goal is to deploy a high-accuracy system starting with zero training...
12/28/2022

Towards AI-Empowered Crowdsourcing

Crowdsourcing, in which human intelligence and productivity is dynamical...
07/23/2023

Milimili. Collecting Parallel Data via Crowdsourcing

We present a methodology for gathering a parallel corpus through crowdso...
01/15/2016

It's about time: Online Macrotask Sequencing in Expert Crowdsourcing

We introduce the problem of Task Assignment and Sequencing (TAS), which ...
12/25/2020

Distributional Ground Truth: Non-Redundant Crowdsourcing Data Quality Control in UI Labeling Tasks

HCI increasingly employs Machine Learning and Image Recognition, in part...
08/28/2018

Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning

This paper presents a Discriminative Deep Dyna-Q (D3Q) approach to impro...
03/17/2021

CrowdSim: A Hybrid Simulation Model for Failure Prediction in Crowdsourced Software Development

A typical crowdsourcing software development(CSD) marketplace consists o...