Online Learning Using Only Peer Assessment

10/10/2019
by   Yang Liu, et al.
0

This paper considers a variant of the classical online learning problem with expert predictions. Our model's differences and challenges are due to lacking any direct feedback on the loss each expert incurs at each time step t. We propose an approach that uses peer assessment and identify conditions where it succeeds. Our techniques revolve around a carefully designed peer score function s() that scores experts' predictions based on the peer consensus. We show a sufficient condition, that we call peer calibration, under which standard online learning algorithms using loss feedback computed by the carefully crafted s() have bounded regret with respect to the unrevealed ground truth values. We then demonstrate how suitable s() functions can be derived for different assumptions and models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/30/2018

Online Learning with an Almost Perfect Expert

We study the online learning problem where a forecaster makes a sequence...
research
03/07/2018

An Application of HodgeRank to Online Peer Assessment

Bias and heterogeneity in peer assessment can lead to the issue of unfai...
research
11/04/2021

Improving Peer Assessment with Graph Convolutional Networks

Peer assessment systems are emerging in many social and multi-agent sett...
research
03/09/2023

On the Value of Stochastic Side Information in Online Learning

We study the effectiveness of stochastic side information in determinist...
research
06/15/2021

Online Learning with Uncertain Feedback Graphs

Online learning with expert advice is widely used in various machine lea...
research
12/04/2017

Mining Supervisor Evaluation and Peer Feedback in Performance Appraisals

Performance appraisal (PA) is an important HR process to periodically me...
research
07/09/2015

Fast rates in statistical and online learning

The speed with which a learning algorithm converges as it is presented w...

Please sign up or login with your details

Forgot password? Click here to reset