PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review

06/16/2018
by   Ivan Stelmakh, et al.
0

We consider the problem of automated assignment of papers to reviewers in conference peer review, with a focus on fairness and statistical accuracy. Our fairness objective is to maximize the review quality of the most disadvantaged paper, in contrast to the commonly used objective of maximizing the total quality over all papers. We design an assignment algorithm based on an incremental max-flow procedure that we prove is near-optimally fair. Our statistical accuracy objective is to ensure correct recovery of the papers that should be accepted. We provide a sharp minimax analysis of the accuracy of the peer-review process for a popular objective-score model as well as for a novel subjective-score model that we propose in the paper. Our analysis proves that our proposed assignment algorithm also leads to a near-optimal statistical accuracy. Finally, we design a novel experiment that allows for an objective comparison of various assignment algorithms, and overcomes the inherent difficulty posed by the absence of a ground truth in experiments on peer-review. The results of this experiment corroborate the theoretical guarantees of our algorithm.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/04/2021

I Will Have Order! Optimizing Orders for Fair Reviewer Assignment

Scientific advancement requires effective peer review. Papers should be ...
research
01/25/2023

Into the Unknown: Assigning Reviewers to Papers with Uncertain Affinities

Peer review cannot work unless qualified and interested reviewers are as...
research
06/27/2020

A SUPER* Algorithm to Optimize Paper Bidding in Peer Review

A number of applications involve sequential arrival of users, and requir...
research
09/20/2021

Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment

In this paper we revisit the 2014 NeurIPS experiment that examined incon...
research
08/13/2021

Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design

Many scientific conferences employ a two-phase paper review process, whe...
research
05/27/2023

Counterfactual Evaluation of Peer-Review Assignment Policies

Peer review assignment algorithms aim to match research papers to suitab...
research
05/28/2019

Paper Matching with Local Fairness Constraints

Automatically matching reviewers to papers is a crucial step of the peer...

Please sign up or login with your details

Forgot password? Click here to reset