DeepAI AI Chat
Log In Sign Up

An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process

by   David Tran, et al.

Mainstream machine learning conferences have seen a dramatic increase in the number of participants, along with a growing range of perspectives, in recent years. Members of the machine learning community are likely to overhear allegations ranging from randomness of acceptance decisions to institutional bias. In this work, we critically analyze the review process through a comprehensive study of papers submitted to ICLR between 2017 and 2020. We quantify reproducibility/randomness in review scores and acceptance decisions, and examine whether scores correlate with paper impact. Our findings suggest strong institutional bias in accept/reject decisions, even after controlling for paper quality. Furthermore, we find evidence for a gender gap, with female authors receiving lower scores, lower acceptance rates, and fewer citations per paper than their male counterparts. We conclude our work with recommendations for future conference organizers.


page 6

page 12

page 13

page 15


Analyzing the Machine Learning Conference Review Process

Mainstream machine learning conferences have seen a dramatic increase in...

De-anonymization of authors through arXiv submissions during double-blind review

In this paper, we investigate the effects of releasing arXiv preprints o...

Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review

Modern machine learning and computer science conferences are experiencin...

A System-Level Analysis of Conference Peer Review

The conference peer review process involves three constituencies with di...

Some Ethical Issues in the Review Process of Machine Learning Conferences

Recent successes in the Machine Learning community have led to a steep i...

Design and Analysis of the NIPS 2016 Review Process

Neural Information Processing Systems (NIPS) is a top-tier annual confer...

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

Conference peer review constitutes a human-computation process whose imp...