Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design

08/13/2021
by   Steven Jecmen, et al.
0

Many scientific conferences employ a two-phase paper review process, where some papers are assigned additional reviewers after the initial reviews are submitted. Many conferences also design and run experiments on their paper review process, where some papers are assigned reviewers who provide reviews under an experimental condition. In this paper, we consider the question: how should reviewers be divided between phases or conditions in order to maximize total assignment similarity? We make several contributions towards answering this question. First, we prove that when the set of papers requiring additional review is unknown, a simplified variant of this problem is NP-hard. Second, we empirically show that across several datasets pertaining to real conference data, dividing reviewers between phases/conditions uniformly at random allows an assignment that is nearly as good as the oracle optimal assignment. This uniformly random choice is practical for both the two-phase and conference experiment design settings. Third, we provide explanations of this phenomenon by providing theoretical bounds on the suboptimality of this random strategy under certain natural conditions. From these easily-interpretable conditions, we provide actionable insights to conference program chairs about whether a random reviewer split is suitable for their conference.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/29/2020

Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments

We consider three important challenges in conference peer review: (i) re...
06/16/2018

PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review

We consider the problem of automated assignment of papers to reviewers i...
12/14/2021

Combating Collusion Rings is Hard but Possible

A recent report of Littmann [Commun. ACM '21] outlines the existence and...
09/28/2021

Generating Summaries for Scientific Paper Review

The review process is essential to ensure the quality of publications. R...
06/24/2022

A Dataset on Malicious Paper Bidding in Peer Review

In conference peer review, reviewers are often asked to provide "bids" o...
07/22/2022

Tradeoffs in Preventing Manipulation in Paper Bidding for Reviewer Assignment

Many conferences rely on paper bidding as a key component of their revie...
11/20/2018

Minimum Guesswork with an Unreliable Oracle

We study a guessing game where Alice holds a discrete random variable X,...

Code Repositories