How To Grade a Test Without Knowing the Answers --- A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing

06/27/2012
by   Yoram Bachrach, et al.
0

We propose a new probabilistic graphical model that jointly models the difficulties of questions, the abilities of participants and the correct answers to questions in aptitude testing and crowdsourcing settings. We devise an active learning/adaptive testing scheme based on a greedy minimization of expected model entropy, which allows a more efficient resource allocation by dynamically choosing the next question to be asked based on the previous responses. We present experimental results that confirm the ability of our model to infer the required parameters and demonstrate that the adaptive testing scheme requires fewer questions to obtain the same accuracy as a static test scenario.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/16/2012

Crowdsourcing Control: Moving Beyond Multiple Choice

To ensure quality results from crowdsourced tasks, requesters often aggr...
research
06/18/2023

Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective

Large language models (LLMs), like ChatGPT, have shown some human-like c...
research
01/30/2017

Dynamic Task Allocation for Crowdsourcing Settings

We consider the problem of optimal budget allocation for crowdsourcing p...
research
09/11/2018

Reducing Uncertainty of Schema Matching via Crowdsourcing with Accuracy Rates

Schema matching is a central challenge for data integration systems. Ins...
research
02/10/2016

Achieving Budget-optimality with Adaptive Schemes in Crowdsourcing

Crowdsourcing platforms provide marketplaces where task requesters can p...
research
07/16/2014

Probabilistic Group Testing under Sum Observations: A Parallelizable 2-Approximation for Entropy Loss

We consider the problem of group testing with sum observations and noise...
research
01/02/2019

A Full Probabilistic Model for Yes/No Type Crowdsourcing in Multi-Class Classification

Crowdsourcing has become widely used in supervised scenarios where train...

Please sign up or login with your details

Forgot password? Click here to reset