Understanding Peer Review of Software Engineering Papers

by   Neil A. Ernst, et al.

Peer review is a key activity intended to preserve the quality and integrity of scientific publications. However, in practice it is far from perfect. We aim at understanding how reviewers, including those who have won awards for reviewing, perform their reviews of software engineering papers to identify both what makes a good reviewing approach and what makes a good paper. We first conducted a series of in-person interviews with well-respected reviewers in the software engineering field. Then, we used the results of those interviews to develop a questionnaire used in an online survey and sent out to reviewers from well-respected venues covering a number of software engineering disciplines, some of whom had won awards for their reviewing efforts. We analyzed the responses from the interviews and from 175 reviewers who completed the online survey (including both reviewers who had won awards and those who had not). We report on several descriptive results, including: 45 award-winners are reviewing 20+ conference papers a year, while 28 non-award winners conduct that many. 88 hours on journal reviews. We also report on qualitative results. To write a good review, the important criteria were it should be factual and helpful, ranked above others such as being detailed or kind. The most important features of papers that result in positive reviews are clear and supported validation, an interesting problem, and novelty. Conversely, negative reviews tend to result from papers that have a mismatch between the method and the claims and from those with overly grandiose claims. The main recommendation for authors is to make the contribution of the work very clear in their paper. In addition, reviewers viewed data availability and its consistency as being important.


A Community's Perspective on the Status and Future of Peer Review in Software Engineering

Context: Pre-publication peer review of scientific articles is considere...

A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering

Context: Systematic literature reviews (SLRs) have become standard pract...

Empirical Standards for Software Engineering Research

Empirical Standards are natural-language models of a scientific communit...

The General Index of Software Engineering Papers

We introduce the General Index of Software Engineering Papers, a dataset...

Choosing How to Choose Papers

It is common to see a handful of reviewers reject a highly novel paper, ...

Consensus in Software Engineering: A Cognitive Mapping Study

Background: Philosophers of science including Collins, Feyerabend, Kuhn ...

Combating Collusion Rings is Hard but Possible

A recent report of Littmann [Commun. ACM '21] outlines the existence and...