First Come First Served: The Impact of File Position on Code Review

by   Enrico Fregnan, et al.

The most popular code review tools (e.g., Gerrit and GitHub) present the files to review sorted in alphabetical order. Could this choice or, more generally, the relative position in which a file is presented bias the outcome of code reviews? We investigate this hypothesis by triangulating complementary evidence in a two-step study. First, we observe developers' code review activity. We analyze the review comments pertaining to 219,476 Pull Requests (PRs) from 138 popular Java projects on GitHub. We found files shown earlier in a PR to receive more comments than files shown later, also when controlling for possible confounding factors: e.g., the presence of discussion threads or the lines added in a file. Second, we measure the impact of file position on defect finding in code review. Recruiting 106 participants, we conduct an online controlled experiment in which we measure participants' performance in detecting two unrelated defects seeded into two different files. Participants are assigned to one of two treatments in which the position of the defective files is switched. For one type of defect, participants are not affected by its file's position; for the other, they have 64 opposed to first. Overall, our findings provide evidence that the relative position in which files are presented has an impact on code reviews' outcome; we discuss these results and implications for tool design and code review. Data and materials:


page 6

page 8


Assessing the Impact of File Ordering Strategies on Code Review Process

Popular modern code review tools (e.g. Gerrit and GitHub) sort files in ...

On The Effect Of Code Review On Code Smells

Code smells are symptoms of poor design quality. Since code review is a ...

The Upper Bound of Information Diffusion in Code Review

Background: Code review, the discussion around a code change among human...

An Empirical Study on Refactoring-Inducing Pull Requests

Background: Pull-based development has shaped the practice of Modern Cod...

RAID: Tool Support for Refactoring-Aware Code Reviews

Code review is a key development practice that contributes to improve so...

An Exploratory Study of Bot Commits

Background: Bots help automate many of the tasks performed by software d...

Automatic Patch Linkage Detection in Code Review Using TextualContent and File Location Features

Context: Contemporary code review tools are a popular choice for softwar...

Please sign up or login with your details

Forgot password? Click here to reset