Into the Unknown: Assigning Reviewers to Papers with Uncertain Affinities

by   Cyrus Cousins, et al.
University of Massachusetts Amherst

Peer review cannot work unless qualified and interested reviewers are assigned to each paper. Nearly all automated reviewer assignment approaches estimate real-valued affinity scores for each paper-reviewer pair that act as proxies for the predicted quality of a future review; conferences then assign reviewers to maximize the sum of these values. This procedure does not account for noise in affinity score computation – reviewers can only bid on a small number of papers, and textual similarity models are inherently probabilistic estimators. In this work, we assume paper-reviewer affinity scores are estimated using a probabilistic model. Using these probabilistic estimates, we bound the scores with high probability and maximize the worst-case sum of scores for a reviewer allocation. Although we do not directly recommend any particular method for estimation of probabilistic affinity scores, we demonstrate how to robustly maximize the sum of scores across multiple different models. Our general approach can be used to integrate a large variety of probabilistic paper-reviewer affinity models into reviewer assignment, opening the door to a much more robust peer review process.


page 1

page 2

page 3

page 4


PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review

We consider the problem of automated assignment of papers to reviewers i...

Counterfactual Evaluation of Peer-Review Assignment Policies

Peer review assignment algorithms aim to match research papers to suitab...

A Framework for Optimizing Paper Matching

At the heart of many scientific conferences is the problem of matching s...

Integrating Rankings into Quantized Scores in Peer Review

In peer review, reviewers are usually asked to provide scores for the pa...

Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators' Expertise

Strategic behavior is a fundamental problem in a variety of real-world a...

A Dataset on Malicious Paper Bidding in Peer Review

In conference peer review, reviewers are often asked to provide "bids" o...

"It's a Match!" – A Benchmark of Task Affinity Scores for Joint Learning

While the promises of Multi-Task Learning (MTL) are attractive, characte...

Please sign up or login with your details

Forgot password? Click here to reset