How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?

by   Charvi Rastogi, et al.

How do author perceptions match up to the outcomes of the peer-review process and perceptions of others? In a top-tier computer science conference (NeurIPS 2021) with more than 23,000 submitting authors and 9,000 submitted papers, we survey the authors on three questions: (i) their predicted probability of acceptance for each of their papers, (ii) their perceived ranking of their own papers based on scientific contribution, and (iii) the change in their perception about their own papers after seeing the reviews. The salient results are: (1) Authors have roughly a three-fold overestimate of the acceptance probability of their papers: The median prediction is 70 25 (statistically significant) miscalibration than male authors; predictions of authors invited to serve as meta-reviewers or reviewers are similarly calibrated, but better than authors who were not invited to review. (3) Authors' relative ranking of scientific contribution of two submissions they made generally agree (93 there is a notable 7 face a worse outcome. (4) The author-provided rankings disagreed with the peer-review decisions about a third of the time; when co-authors ranked their jointly authored papers, co-authors disagreed at a similar rate – about a third of the time. (5) At least 30 rejected papers said that their perception of their own paper improved after the review process. The stakeholders in peer review should take these findings into account in setting their expectations from peer review.


Does double-blind peer-review reduce bias? Evidence from a top computer science conference

Peer review is widely regarded as essential for advancing scientific res...

The Role of Author Identities in Peer Review

There is widespread debate on whether to anonymize author identities in ...

Decontamination of the scientific literature

Research misconduct and frauds pollute the scientific literature. Honest...

A System-Level Analysis of Conference Peer Review

The conference peer review process involves three constituencies with di...

Combating Collusion Rings is Hard but Possible

A recent report of Littmann [Commun. ACM '21] outlines the existence and...

What makes a successful rebuttal in computer science conferences? : A perspective on social interaction

With an exponential increase in submissions to top-tier Computer Science...

Deep Paper Gestalt

Recent years have witnessed a significant increase in the number of pape...

Please sign up or login with your details

Forgot password? Click here to reset