DeepAI AI Chat
Log In Sign Up

Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review

by   Ivan Stelmakh, et al.

Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review as the number of competent reviewers is growing at a much slower rate. To curb this trend and reduce the burden on reviewers, several conferences have started encouraging or even requiring authors to declare the previous submission history of their papers. Such initiatives have been met with skepticism among authors, who raise the concern about a potential bias in reviewers' recommendations induced by this information. In this work, we investigate whether reviewers exhibit a bias caused by the knowledge that the submission under review was previously rejected at a similar venue, focusing on a population of novice reviewers who constitute a large fraction of the reviewer pool in leading machine learning and computer science conferences. We design and conduct a randomized controlled trial closely replicating the relevant components of the peer-review pipeline with 133 reviewers (master's, junior PhD students, and recent graduates of top US universities) writing reviews for 19 papers. The analysis reveals that reviewers indeed become negatively biased when they receive a signal about paper being a resubmission, giving almost 1 point lower overall score on a 10-point Likert item (Δ = -0.78, 95% CI = [-1.30, -0.24]) than reviewers who do not receive such a signal. Looking at specific criteria scores (originality, quality, clarity and significance), we observe that novice reviewers tend to underrate quality the most.


page 1

page 2

page 3

page 4


Does double-blind peer-review reduce bias? Evidence from a top computer science conference

Peer review is widely regarded as essential for advancing scientific res...

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

Conference peer review constitutes a human-computation process whose imp...

Testing for Reviewer Anchoring in Peer Review: A Randomized Controlled Trial

Peer review frequently follows a process where reviewers first provide i...

What makes a successful rebuttal in computer science conferences? : A perspective on social interaction

With an exponential increase in submissions to top-tier Computer Science...

Cite-seeing and Reviewing: A Study on Citation Bias in Peer Review

Citations play an important role in researchers' careers as a key factor...

A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions

Peer review is the backbone of academia and humans constitute a cornerst...

An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process

Mainstream machine learning conferences have seen a dramatic increase in...