Combining Crowd and Machines for Multi-predicate Item Screening

04/01/2019
by   Evgeny Krivosheev, et al.
0

This paper discusses how crowd and machine classifiers can be efficiently combined to screen items that satisfy a set of predicates. We show that this is a recurring problem in many domains, present machine-human (hybrid) algorithms that screen items efficiently and estimate the gain over human-only or machine-only screening in terms of performance and cost. We further show how, given a new classification problem and a set of classifiers of unknown accuracy for the problem at hand, we can identify how to manage the cost-accuracy trade off by progressively determining if we should spend budget to obtain test data (to assess the accuracy of the given classifiers), or to train an ensemble of classifiers, or whether we should leverage the existing machine classifiers with the crowd, and in this case how to efficiently combine them based on their estimated characteristics to obtain the classification. We demonstrate that the techniques we propose obtain significant cost/accuracy improvements with respect to the leading classification algorithms.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/21/2018

Crowd-Machine Collaboration for Item Screening

In this paper we describe how crowd and machine classifier can be effici...
research
01/21/2021

Active Hybrid Classification

Hybrid crowd-machine classifiers can achieve superior performance by com...
research
11/11/2020

Active Learning from Crowd in Document Screening

In this paper, we explore how to efficiently combine crowdsourcing and m...
research
05/31/2018

CrowdRev: A platform for Crowd-based Screening of Literature Reviews

In this paper and demo we present a crowd and crowd+AI based system, cal...
research
11/05/2020

On the impact of predicate complexity in crowdsourced classification tasks

This paper explores and offers guidance on a specific and relevant probl...
research
10/27/2021

Cascaded Classifier for Pareto-Optimal Accuracy-Cost Trade-Off Using off-the-Shelf ANNs

Machine-learning classifiers provide high quality of service in classifi...
research
11/14/2020

Cost-Sensitive Machine Learning Classification for Mass Tuberculosis Verbal Screening

Score-based algorithms for tuberculosis (TB) verbal screening perform po...

Please sign up or login with your details

Forgot password? Click here to reset