Understanding Peer Review of Software Engineering Papers

09/02/2020
by   Neil A. Ernst, et al.
0

Peer review is a key activity intended to preserve the quality and integrity of scientific publications. However, in practice it is far from perfect. We aim at understanding how reviewers, including those who have won awards for reviewing, perform their reviews of software engineering papers to identify both what makes a good reviewing approach and what makes a good paper. We first conducted a series of in-person interviews with well-respected reviewers in the software engineering field. Then, we used the results of those interviews to develop a questionnaire used in an online survey and sent out to reviewers from well-respected venues covering a number of software engineering disciplines, some of whom had won awards for their reviewing efforts. We analyzed the responses from the interviews and from 175 reviewers who completed the online survey (including both reviewers who had won awards and those who had not). We report on several descriptive results, including: 45 award-winners are reviewing 20+ conference papers a year, while 28 non-award winners conduct that many. 88 hours on journal reviews. We also report on qualitative results. To write a good review, the important criteria were it should be factual and helpful, ranked above others such as being detailed or kind. The most important features of papers that result in positive reviews are clear and supported validation, an interesting problem, and novelty. Conversely, negative reviews tend to result from papers that have a mismatch between the method and the claims and from those with overly grandiose claims. The main recommendation for authors is to make the contribution of the work very clear in their paper. In addition, reviewers viewed data availability and its consistency as being important.

READ FULL TEXT
research
06/22/2017

A Community's Perspective on the Status and Future of Peer Review in Software Engineering

Context: Pre-publication peer review of scientific articles is considere...
research
09/22/2021

Developers Perception of Peer Code Review in Research Software Development

Background: Research software is software developed by and/or used by re...
research
09/21/2021

A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering

Context: Systematic literature reviews (SLRs) have become standard pract...
research
10/07/2020

Empirical Standards for Software Engineering Research

Empirical Standards are natural-language models of a scientific communit...
research
08/27/2018

Choosing How to Choose Papers

It is common to see a handful of reviewers reject a highly novel paper, ...
research
09/09/2022

Pitfalls and Guidelines for Using Time-Based Git Data

Many software engineering research papers rely on time-based data (e.g.,...
research
09/20/2023

Architecture Knowledge Representation and Communication Industry Survey

Background: The literature offers various methods for capturing software...

Please sign up or login with your details

Forgot password? Click here to reset