Weighting Experts with Inaccurate Judges

by   Ben Abramowitz, et al.

We consider the problem of aggregating binary votes from an ensemble of experts to reveal an underlying binary ground truth where each expert votes correctly with some independent probability. We focus on settings where the number of agents is too small for asymptotic results to apply, many experts may vote correctly with low probability, and there is no central authority who knows the experts' competences, or their probabilities of voting correctly. Our approach is to designate a second type of agent – a judge – to weight the experts to improve overall accuracy. The catch is that the judge has imperfect competence just like the experts. We demonstrate that having a single minimally competent judge is often better than having none at all. Using an ensemble of judges to weight the experts can provide a better weighting than any single judge; even the optimal weighting under the right conditions. As our results show, the ability of the judge(s) to distinguish between competent and incompetent experts is paramount. Lastly, given a fixed set of agents with unknown competences drawn i.i.d. from a common distribution, we show how the optimal split of the agents between judges and experts depends on the distribution.


page 1

page 2

page 3

page 4


Towards Group Learning: Distributed Weighting of Experts

Aggregating signals from a collection of noisy sources is a fundamental ...

Prediction with Advice of Unknown Number of Experts

In the framework of prediction with expert advice, we consider a recentl...

Everyone Knows that Everyone Knows: Gossip Protocols for Super Experts

A gossip protocol is a procedure for sharing secrets in a network. The b...

Interpretable collective intelligence of non-rational human agents

We outline how to create a mechanism that provides an optimal way to eli...

Unsupervised Ensemble Regression

Consider a regression problem where there is no labeled data and the onl...

Probabilistic elicitation of expert knowledge through assessment of computer simulations

We present a new method for probabilistic elicitation of expert knowledg...

A Multiple-Expert Binarization Framework for Multispectral Images

In this work, a multiple-expert binarization framework for multispectral...

Please sign up or login with your details

Forgot password? Click here to reset