Measurement Integrity in Peer Prediction: A Peer Assessment Case Study

08/12/2021
by   Noah Burrell, et al.
0

We propose measurement integrity, a property related to ex post reward fairness, as a novel desideratum for peer prediction mechanisms in many applications, including peer assessment. We operationalize this notion to evaluate the measurement integrity of different mechanisms in computational experiments. Our evaluations simulate the application of peer prediction mechanisms to peer assessment—a setting in which realistic models have been validated on real data and in which ex post fairness concerns are quite salient. We find that peer prediction mechanisms, as proposed in the literature, largely fail to demonstrate measurement integrity in our experiments. However, we also find that certain mechanisms can be supplemented with realistic parametric statistical models to improve their measurement integrity. In the same setting, we also evaluate an empirical notion of robustness against strategic behavior to complement the theoretical analyses of robustness against strategic behavior that have been the main focus of the peer prediction literature. In this dimension of analysis, we again find that supplementing certain mechanisms with parametric statistical models can improve their empirical performance. Even so, though, we find that theoretical guarantees of robustness against strategic behavior are somewhat noisy predictors of empirical robustness. As a whole, our empirical methodology for quantifying desirable mechanism properties facilitates a more nuanced comparison between mechanisms than theoretical analysis alone. Ultimately, we find there is a trade-off between our two dimensions of analysis. The best performing mechanisms for measurement integrity are highly susceptible to strategic behavior. On the other hand, certain parametric peer prediction mechanisms are robust against all the strategic manipulations we consider while still achieving reasonable measurement integrity.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/08/2020

Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment

We consider the issue of strategic behaviour in various peer-assessment ...
research
11/04/2021

Improving Peer Assessment with Graph Convolutional Networks

Peer assessment systems are emerging in many social and multi-agent sett...
research
06/06/2021

The Limits of Multi-task Peer Prediction

Recent advances in multi-task peer prediction have greatly expanded our ...
research
07/31/2018

Truthful Peer Grading with Limited Effort from Teaching Staff

Massive open online courses pose a massive challenge for grading the ans...
research
08/05/2019

Discovery of Bias and Strategic Behavior in Crowdsourced Performance Assessment

With the industry trend of shifting from a traditional hierarchical appr...
research
01/16/2019

A System Dynamics Analysis of National R D Performance Measurement System in Korea

Peer review is one of useful and powerful performance measurement proces...
research
08/14/2023

Sustainable Cooperation in Peer-To-Peer Networks

Traditionally, peer-to-peer systems have relied on altruism and reciproc...

Please sign up or login with your details

Forgot password? Click here to reset