A Full Probabilistic Model for Yes/No Type Crowdsourcing in Multi-Class Classification

01/02/2019
by   Belen Saldias-Fuentes, et al.
0

Crowdsourcing has become widely used in supervised scenarios where training sets are scarce and hard to obtain. Most crowdsourcing models in literature assume labelers can provide answers for full questions. In classification contexts, full questions mean that a labeler is asked to discern among all the possible classes. Unfortunately, that discernment is not always easy in realistic scenarios. Labelers may not be experts in differentiating all the classes. In this work, we provide a full probabilistic model for a shorter type of queries. Our shorter queries just required a 'yes' or 'no' response. Our model estimates a joint posterior distribution of matrices related to the labelers confusions and the posterior probability of the class of every object. We develop an approximate inference approach using Monte Carlo Sampling and Black Box Variational Inference, where we provide the derivation of the necessary gradients. We build two realistic crowdsourcing scenarios to test our model. The first scenario queries for irregular astronomical time-series. The second scenario relies on animal's image classification. Results show that we can achieve comparable results with full query crowdsourcing. Furthermore, we show that modeling the labelers failures plays an important role in estimating the true classes. Finally, we provide the community with two real datasets obtained from our crowdsourcing experiments. All our code is publicly available (Available at: revealed as soon as the paper gets published.)

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/07/2016

Measuring the reliability of MCMC inference with bidirectional Monte Carlo

Markov chain Monte Carlo (MCMC) is one of the main workhorses of probabi...
research
05/23/2022

Quasi Black-Box Variational Inference with Natural Gradients for Bayesian Learning

We develop an optimization algorithm suitable for Bayesian learning in c...
research
11/01/2016

Variational Inference via χ-Upper Bound Minimization

Variational inference (VI) is widely used as an efficient alternative to...
research
06/27/2012

How To Grade a Test Without Knowing the Answers --- A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing

We propose a new probabilistic graphical model that jointly models the d...
research
06/11/2019

Subspace Attack: Exploiting Promising Subspaces for Query-Efficient Black-box Attacks

Unlike the white-box counterparts that are widely studied and readily ac...
research
06/10/2015

On-the-Job Learning with Bayesian Decision Theory

Our goal is to deploy a high-accuracy system starting with zero training...
research
05/17/2023

Model-based Validation as Probabilistic Inference

Estimating the distribution over failures is a key step in validating au...

Please sign up or login with your details

Forgot password? Click here to reset