A Large-Scale Study on Source Code Reviewer Recommendation

by   Jakub Lipcak, et al.
Masarykova univerzita

Context: Software code reviews are an important part of the development process, leading to better software quality and reduced overall costs. However, finding appropriate code reviewers is a complex and time-consuming task. Goals: In this paper, we propose a large-scale study to compare performance of two main source code reviewer recommendation algorithms (RevFinder and a Naive Bayes-based approach) in identifying the best code reviewers for opened pull requests. Method: We mined data from Github and Gerrit repositories, building a large dataset of 51 projects, with more than 293K pull requests analyzed, 180K owners and 157K reviewers. Results: Based on the large analysis, we can state that i) no model can be generalized as best for all projects, ii) the usage of a different repository (Gerrit, GitHub) can have impact on the the recommendation results, iii) exploiting sub-projects information available in Gerrit can improve the recommendation results.



page 1

page 2

page 3

page 4


CORRECT: Code Reviewer Recommendation in GitHub Based on Cross-Project and Technology Experience

Peer code review locates common coding rule violations and simple logica...

DeSkew-LSH based Code-to-Code Recommendation Engine

Machine learning on source code (MLOnCode) is a popular research field t...

Repo2Vec: A Comprehensive Embedding Approach for Determining Repository Similarity

How can we identify similar repositories and clusters among a large onli...

Using Source Code Density to Improve the Accuracy of Automatic Commit Classification into Maintenance Activities

Source code is changed for a reason, e.g., to adapt, correct, or adapt i...

Recommendation of Move Method Refactoring Using Path-Based Representation of Code

Software refactoring plays an important role in increasing code quality....

To test, or not to test: A proactive approach for deciding complete performance test initiation

Software performance testing requires a set of inputs that exercise diff...

More Accurate Recommendations for Method-Level Changes

During the life span of large software projects, developers often apply ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

The source code review process was formalized in 1976 by M. E. Fagan, as a highly structured process based on line-by-line group reviews in the form of inspections [1]. The code review practices have rapidly changed in recent years [2], and the modern code review process has become informal (in contrast to Fagan’s definition [1]), tool-based [3], more light-weight, continuous and asynchronous [4].

Software code reviews play nowadays a major role in improving the quality of software in terms of defects identified [5, 6]. However, code reviews are a time-consuming process, with significant amount of human effort involved [7]. Managers might have concerns that code reviews will slow the projects down [6]. Code reviews could also lead to negative moods in teams if team members fear public criticism caused by others reviewing their source code [8].

At a process level, source code review processes in companies nowadays are similar to the processes adopted in open source projects [6, 9, 10], with a lot of variation in the process steps [6, 11]. Furthermore, source code reviews can be expensive: knowledgeable understanding of large code changes by reviewers is a time-consuming process, and finding the most knowledgeable code reviewers for the source code parts to be reviewed can also be very labor-intensive for developers. Thongtanunam et al. [12] examined comments from more than 1,400 representative review samples of four open source projects. Authors discovered that 4%-30% of reviews have a code reviewer assignment problem, taking approximately 12 days longer to approve a code change with a code reviewer assignment problem, increasing projects costs.

The reduction of such costs was the main reason for the emergence of source code reviewer recommendation approaches [13, 12, 14, 15, 16, 17, 18] and for the appearance of automation in software engineering, beneficial also for industry [19, 20]. The automated recommendation of code reviewers can reduce the efforts necessary to find the best code reviewers, cutting time spent by code reviewers on understanding large code changes [13]. This can improve the effectiveness of the pull-based model, with source code reviewers assigned to the pull requests immediately after their creation.

In previous studies about source code reviewers recommendation algorithms, the compared projects were limited in size and number, with typically 8K-45K pull requests considered ( [21, 15, 12, 22, 23]

). In this paper, we look into a large-scale evaluation (293K pull requests, 51 projects) based on Gerrit and Github repositories, that can give more insights about the performance of reviewers recommendation approaches. Our goal is to study the impact of different repository type used for the source code recommendation. For this reason we chose two main baseline algorithms: RevFinder—heuristic model based on file paths

[12] and a Naïve Bayes-based approach (NB)—probabilistic model based on additional features such as owners. More advanced algorithms exist, however they require additional features to be built (e.g., collaboration networks between developers), making the mining process more complex. We have two main contributions in this paper:

  • provision of a large dataset of 51 projects mined from Gerrit (14) and Github (37) with 293,337 total pull requests analyzed, considering 180,111 owners and 157,885 reviewers. The dataset is available on Figshare [24].

  • comparison of the results in the two repositories (Gerrit, GitHub) using both RevFinder and Naïve Bayes-based approaches in the context of the 51-projects dataset.

The article is structured as follows. In section II, we propose the background on several algorithms for code reviewer recommendation. In section III we have the experimental study design, with research questions, context, data analysis and replicability information. In section IV, we answer the research questions with discussions and threats to validity. Section V proposes the related works and section VI the conclusions.

Ii Background

Over the years, many approaches for the automation of source code reviewers recommendation emerged [12, 14, 15, 16, 17, 18]

. We categorize the most relevant existing recommendation algorithms into four groups based on the main features they process and the main techniques they use to recommend code reviewers: i) Heuristic-based approaches, ii) Machine Learning-based, iii) Social Networks-based, iv) Hybrid approaches.

Ii-a Heuristic-based Approaches

Traditional recommendation approaches process historical project review data and use heuristic-based algorithms to find the most relevant code reviewers. Main algorithms are ReviewBot [17], RevFinder [12] (that we will explain later, as part of our empirical analysis), and CORRECT [14].

ReviewBot is a technique proposed by Balachandran [17]. It is a code reviewer recommendation approach based on the assumption that lines of code changed in the pull request should be reviewed by the same code reviewers who had previously discussed or modified the same lines of code (familiarity assumption).

CORRECT (Code Reviewer Recommendation based on Cross-project and Technology experience) is another code reviewers recommendation technique [14]. The baseline idea is that if a previous pull request used some similar external library or technology to the current pull request, then the reviewers of the past pull request are also good candidates for the current one (expertise assumption).

Ii-B Machine Learning

Algorithms in this group use different Machine Learning techniques for the recommendation of code reviewers. They are mainly different from the previous group, as they first need to build a model based on a training set. One typical approach is by Jeoung et al. [18].

Predicting Reviewers and Acceptance of Patches is an approach by Jeoung et al. [18]

that uses the Bayesian Network technique to predict reviewers and patch acceptance based on a series of features such as patch meta-data, patch content and bug report information.

Ii-C Social Networks

Social networks have also been used to determine similarities in communication between developers, suggesting more similar candidates for source code reviews.

The Comment Network (CN) approach was proposed by Yu et al. [16] as a code reviewer recommendation approach by analyzing social relations between contributors and developers. CN-based recommendation is based on the idea that the interest of developers can be extracted from their commenting interaction. Developers sharing common interests with the originator of a pull request are considered to be appropriate code reviewers [16].

Ii-D Hybrid Approaches

Algorithms in this group use different approaches (machine learning, social network analysis) for the recommendation of code reviewers. An example is CoreDevRec [15].

Automatic Core Member Recommendation for Contribution Evaluation (CoreDevRec)

builds a prediction model from historical pull requests using file paths similarities, developers GitHub relationship features, and activeness features, using Support Vector Machine (SVM) for prediction


Table I shows a summary of all the features utilized by the reviewed algorithms, while Table II summarizes the availability of source code, datasets, and metrics used for empirical validation of the proposed algorithms.


ReviewBot [17]

RevFinder [12]


CoreDevRec [15]

Pred. Rev. [18]

Comm. Net. [16]


File paths 2
Social interactions 2
Line change history 1
Reviewer expertise 1
Activeness of reviewers 1
Patch meta-data 1
Patch content 1
Bug report information 1
Total 1 1 1 3 3 1

TABLE I: Summary of features used in most common source code reviewers recommendation algorithms. v=used, x=not used
Paper Source code available Dataset available Metrics





F - Measure

ReviewBot [17]
RevFinder [12]
CoreDevRec [15]
Pred. Rev. [18]
Comm. Net. [16]
TABLE II: Source code & datasets availability, metrics used in the empirical studies.v=used, x=not used

Iii Empirical Study Design

The goal of this paper is to analyze source code recommendation algorithms, with the purpose of investigating i) performance on a large set of projects, ii) the impact of different repository type used for the source code recommendation. We focus on two main algorithms: RevFinder and a novel Naïve Bayes-based implementation, as these can give a good baseline for comparison. We have three main research questions:

  • RQ1: What is the comparison between RevFinder and NB algorithms in terms of performance on both Gerrit and Github Repositories? In this research question, we investigate two major algorithms for source code recommendation on 51 projects, based on top-k accuracy and Mean Reciprocal Rank (MRR) metrics.

  • RQ2: What is the impact of using data from GitHub or Gerrit for source code reviewers recommendation? In this research question, we look at the impact of different repositories and their characteristics in source code recommendation results.

  • RQ3: Is there any performance improvement when considering the additional sub-projects feature (Gerrit-specific)? In this research question, we look at exploiting the additional sub-projects feature available in the Gerrit repositories, looking at whether such feature can be useful to improve source code recommendation performance.

Iii-a Context

For the empirical evaluation, we compare two alternative approaches, RevFinder [12] based on file paths and a Naïve Bayes implementation, based on two main features: file paths and owners of reviews.

Iii-A1 RevFinder

RevFinder [12] is an approach based on the location of files included in pull requests. The idea of this method is that files located in similar file paths contain similar functionality and therefore should be reviewed by similar code reviewers.

The first part of RevFinder approach is the Code Reviewers Ranking Algorithm. It compares file paths included in a new pull request with all previously reviewed file paths, by using four string comparison techniques (Longest Common Prefix, Longest Common Suffix, Longest Common Substring and Longest Common Subsequence).

Code reviewer candidates are assigned points in this step. The more similar the file paths are, the higher is the number of points assigned to the code reviewers who previously reviewed them. The second important part of the RevFinder approach is the combination technique. The results of each of the four string comparison techniques are combined using the Borda count combination method. A sorted list of candidates with their scores is then returned as the output of the RevFinder algorithm [12].

We reimplemented the algorithm based on the information that was available in the research paper [12] and in the GitHub repository for the string comparison part111https://github.com/patanamon/revfinder.

Iii-A2 Naïve Bayes Recommendation

The pseudo-code of our code reviewer recommendation algorithm is shown in Fig. 1. The algorithm takes a new review as input (). This review has to contain information about all modified File paths, the Project name and about the Owner of the review. A sorted list of recommended code reviewer candidates () is returned as the output. The prediction model has to be built at the beginning from all previously closed reviews (lines 5 and 6).

Lines 8 to 18 describe the main loop where code reviewer candidates are recommended. It iterates over all files modified in the review. Feature probabilities are computed separately for every file and therefore code reviewers are recommended separately. Lines 14 to 17 calculate scores achieved by code reviewers. Every code reviewer is assigned a whole number based on his position in the recommendation list. All the code reviewers who reviewed at least one change request in the past will appear in the recommendation list thanks to

probability smoothing [25]. Score calculation is done for every file and the achieved scores are added together.

Finally, retired reviewers are concatenated to the end of the result list by the removeRetiredReviewers function (line 19). This function iterates over all code reviewers and moves down those who have not done any code reviews in recent n months. Finally (line 20), code reviewers are sorted by their scores and a sorted list of reviewer candidates is returned as the result.

2 : A new review
4: Sorted list of code reviewer candidates
5   List all previously closed reviews
8for  do
9    # Get sorted list of recommended reviewers
10    # for every file to be reviewed
12    # Assign points to reviewers
14    for  do
17    end for
18end for
Fig. 1: Naïve Bayes-based Code Reviewer Recommendation Algorithm.
Repository # Projects pull requests reviewers owners
avg max min tot avg max min tot avg max min tot
GitHub 37 4,323 29,807 431 159,953 583 1,949 85 21,599 1,061 2,970 313 39,267
Gerrit 14 9,530 31,582 344 133,424 204 651 30 4,558 325 766 27 4,558
Overall 51 5,752 31,582 344 293,377 479 1,949 30 24,461 859 1,061 27 43,825
TABLE III: Mined GitHub and Gerrit Repositories

Iii-A3 RevFinder+ and NB+

As pull requests in the same subproject are often reviewed by similar code reviewers, we added the sub-project name as a feature of RevFinder and NB algorithms for the recommendation process: we named these versions RevFinder+ and NB+. Unfortunately, this information is only available in the Gerrit repository, so we could only apply it in the context of the projects hosted in Gerrit.

Iii-B Approach

For the experiments, we gathered data from 51 open source software projects through the Gerrit and GitHub systems APIs (Table III, which provides information about the number of pull requests, reviewers, and owners in the dataset). We chose 14 large and active Gerrit projects and 37 GitHub projects, which belonged to the most popular GitHub repositories according to the gained stars.

Fig. 2: Data mining process.

The REST API provided by the Gerrit system was used to collect data from 14 projects involving 133,424 pull requests (Fig. 2, data mining process). Although the API slightly differs between different versions of Gerrit, it was always possible to mine the data necessary for our experiments. For each project, we extracted the data from all pull requests with the status ’merged’, where all the users who marked the pull request with positive Code-Review label were considered as reviewers of the given pull request.

The GitHub REST API was used to mine data from 37 projects having 159,953 pull requests in total (Fig. 2, data mining process). We only processed pull requests in a closed state with those users considered as reviewers, who created at least one Issue Comment or Pull Request Review Comment for the given pull request.

The final dataset contains 293,377 pull requests stored in the form of JSON files. Each JSON file contains the list of pull requests from one project with the following features: sub-project (always empty string for GitHub projects), change id, change number, timestamp, owner of the pull request, list of reviewers and list of modified files. The owner of a review is the developer that submitted the code review request, while reviewers represent developers that participated in the review process. If the owner of the pull request also appeared as the reviewer of the pull request, he/she was removed from the list of reviewers, because such cases do not need any recommendation. Pull requests with an empty list of reviewers or an empty list of files were removed from the dataset as well.

Evaluation Metrics.

Top-k Accuracy, Mean Reciprocal Rank (MRR), Precision and Recall (see also back to Table

II) are used for the evaluation of source code reviewers recommendations. However, we do not use Precision and Recall (and combined F-Measure) as they evaluate the intersection of top-k recommended code reviewers with all actual code reviewers, so they might not be appropriate for datasets containing reviews where the size of the set of actual code reviewers is smaller than the value [12].

Iii-B1 Top-k Accuracy

The result of top-k accuracy is a percentage describing how many code reviewers were correctly recommended within top k reviewers of the result list.


where: R is a set of reviews, isCorrect(r, k) is a function returning 1 if at least one code reviewer recommended within top k reviewers approved the review r, otherwise, 0 is returned.

Iii-B2 Mean Reciprocal Rank

Mean Reciprocal Rank (MRR) is a value describing an average position of actual code reviewer in the list returned by recommendation algorithm.


where: R is a set of reviews, recommend(r) is a function returning a sorted list of code reviewers recommended for review r, rank(r, l) is a function returning the rank of the code reviewer who approved review r in a sorted list l. The value of will be if there is no such code reviewer in the list l.

Iii-C Data Analysis

Assignment of code reviewers to pull requests is time dependent. Therefore, we had to use a setup which ensures that only data from past pull requests will be used to recommend code reviewers for future pull requests. For testing the Naïve Bayes-based approach we chose 11-fold validation inspired by Bettenburg et al. [26] and Jeong et al. [18], dividing the training set into 11 equally sized folds. The experiments are run in 10 iterations using different folds as training and test sets (Fig. 3). The results of all iterations are then averaged.

Fig. 3: Eleven folds experimental set-up

RevFinder algorithm always considers all pull requests from the past for recommendation, without needs to build a model for training folds. As we wanted to evaluate the accuracy of both algorithms for the same set of pull requests, we only executed RevFinder recommendation for pull requests in folds 2-11.

To compare differences between groups to answer the research questions, we used different statistical tests. For RQ1 and RQ2 we used the Wilcoxon Signed-Rank Test [27, 28], a paired difference test to evaluate the mean ranks differences that we applied to the MRR metric. For RQ3, we used the Mann-Whitney U Test (also Wilcoxon Rank-Sum Test, or just Wilcoxon-Mann-Whitney Test) [29, 28] for MRR, due to the fact that our samples for this RQ are not paired, coming from different repositories: Gerrit(14), GitHub (37). We use the more conservative assumption of two-tailed tests, as we do not have any a priori hypothesis that one of the compared approaches is better—furthermore, all tests are non-parametric, not assuming normal data distribution [30]. While the -value can show us if the effect is statistically significant, we also report the effect size, that gives an indication of the size of the observed effect, independently from the sample size [31, 32]. For Wilcoxon-Mann-Whitney tests, we calculate effect size as , to take into account non-independent paired samples [33], using Cohen’s definition to discriminate between small (), medium (), and large effects () [31].

Replicability. Throughout the study, we followed the concept of reproducible research [34, 35]. We implemented a Spring Boot application. We used the MySql relational database as a data storage and the Hibernate framework for object-relational mapping. We used the Jayes library for all the computations involving Naïve Bayes. The dataset is available on Figshare [24], the source code is available at https://github.com/XLipcak/rev-rec.

Iv Empirical Results

In this section, we discuss the study results based on the three central research questions that were set. Information about all projects considered is available in the appendix Table V.

Iv-a RQ1: What is the comparison between RevFinder and NB algorithms in terms of performance on both Gerrit and Github Repositories?

In this research question, we investigate RevFinder and NB on all 51 projects, based on top-k and MRR metrics.

In terms of average MRR from both the algorithms, results are similar (MRRRF=60.6, MRRNB=59.1). However, running a paired comparison with a Wilcoxon Signed-Rank Test, paired difference test to evaluate the mean ranks differences for MRR, showed significant results (-value 0.0466. The results are significant at , two-tailed, with small effect size ()).

Fig. 4: RQ1. Delta MRR between RevFinder and NB. Each bar represents one project (positive = gain of RevFinder, negative = gain by NB).

Such differences can be seen also when plotting the delta MRR (RevFinder-NB) for each of the 51 projects (Fig. 4). We observed that for projects Eclipse (+31.1), Lineageos (+28.8) and Qt (+21.8), the NB-approach is consistently better, while RevFinder is, in general, better in the prediction results. We did not find a clear explanation about the difference in the results for these three projects: one possibility is that considering owners for these projects is more relevant than file paths (used in RevFinder) for the choice of the final reviewer. Another explanation is that, as we will discuss in the limitations, all these three projects are from Gerrit, and we could not remove bots in such repository as there is no automated way for identification. So in all these projects, the presence of bots might have made the task of NB easier. Another explanation could be issues in the mining process, Qt was for example removed from the dataset provided in Yang et al. [36] due to issues in the official Gerrit API.

Interpretation of the results. While the two distributions show similar results, looking at paired differences between predictions on the same projects showed significant results. While generally RevFinder behaves better, three projects showed substantial difference in results in favor of NB—which brings main concerns about the selection of projects in smaller scale comparisons. Effect size is small, meaning that observed differences between RevFinder and NB are small and significance can be due to the sample size used in this study.

Findings. Within the 51 projects (Gerrit+GitHub), generally RevFinder is better, but effect size is small and differences might be trivial. A simple algorithm based on file paths (RevFinder) without building a model, reaches 60% MRR.

Iv-B RQ2: What is the impact of using data from GitHub or Gerrit for source code reviewers recommendation?

In this research question, we look at the impact of different repositories and their characteristics in source code recommendation results.

We run Wilcoxon U-Test for MRR, top-1, and top-k measures comparing GitHub and Gerrit repositories (Table IV). With six comparisons, we applied Bonferroni correction (), which requires to be significant. Results showed significant results for RevFinder for two out of three measures, while they were not significant for NB. In all cases, we can report a small effect size.

RevFinder NB
MRR , ** ,
Top-1 , ,
Top-3 , ** ,
TABLE IV: Wilcoxon U-Test performance on GitHub vs Gerrit. **=significant after Bonferroni correction
Fig. 5: RQ2. Comparison of Gerrit and Github datasets for % top-k and MRR: RF= RevFinder, NB=Naïve Bayes, GE= Gerrit, GH=GitHub.

If we look at the performance on the different repositories, we notice that results on GitHub are better than those on Gerrit (Fig. 5). Independently from the algorithm used, just by analyzing a GitHub project can bring better results than in the case of Gerrit (MRRmean = 53.87 vs 63.03, which is a quite notable difference). If we look for single algorithms, RevFinder reaches MRRmean = 50.41 on Gerrit and 55 on GitHub, while NB has MRRmean = 57.32 on Gerrit and 61.62 on GitHub.

Interpretation of the results. As seen, there are differences comparing results obtained on Gerrit or GitHub repositories in terms of recommendation performance. Such variations are statistically significant in the case of RevFinder. Comparisons of different studies need to take this aspect into account, as the application on one repository can get some gains, just for this reason. One interpretation we can give to the results (section IV-D looks to inspect specific pull requests) is about the differences between the mined repositories. GitHub repositories have a higher number of reviewers per pull request, lower number of files per pull request (Fig. 6). Aspects in which the difference is consistent are a much lower number of pull requests per reviewer and pull requests per owner. Our interpretation is that these factors can have an impact on the final recommendation results. Also in this case, effect size is small, showing that differences can be limited in the practical context.

Fig. 6: RQ2. Comparison of Gerrit and Github datasets (log-scale): GE= Gerrit, GH=GitHub, RevsPR= Reviewers per Pull Request, FilesPR= Files per Pull Request, PRRevs=Pull Requests per Reviewers, PROwns=Pull Requests per Owner.

Findings. Using different repositories for data analysis can have an impact on the recommendation results in case of the RevFinder algorithm. Running on GitHub can bring better results than on Gerrit projects. However, effect size is small. Our interpretation is that this difference is mainly due to the divergence of the two communities: more general GitHub, more specialized Gerrit.

Iv-C RQ3: Is there any performance improvement when considering the additional sub-projects feature (Gerrit-specific)?

In this research question, we look at exploiting the sub-project information feature available in the Gerrit repository.

Fig. 7: RQ3. Using Gerrit additional features for recommendation (RevFinder vs RevFinder+, NB vs NB+).

We show MRR and top-k performance for both RevFinder, RevFinder+, NB and NB+. The analysis is run on the 14 Gerrit projects (Fig. 7). Running a Wilcoxon Signed-Rank Test, a paired difference test to evaluate the mean ranks differences for MRR results, showed non-significant results for RevFinder+ improvements (-value 0.167. The results are not significant at (), two-tailed, with small effect size ()). The same test for NB+ improvements was instead statistically significant (-value 0.004. The results are significant at (), two-tailed, with medium effect size ()).

Interpretation of the results. There is only one project in which NB+ worsens the results slightly compared to NB: in all other projects, considering the sub-projects information with NB improves the outcomes (Fig. 8). The main suggestion that derives from this analysis is that large distributed projects that are hosted on repositories that support sub-project information can benefit from recommendation automation by using this as a feature. As source code reviewers recommendation algorithms are more useful in case of large distributed system development, such information should be available for improvement of the recommendation results. Medium effect size shows that the differences can be quite noticeable.

Fig. 8: RQ3. Delta MRR between RevFinder+ vs RevFinder and NB+ vs NB

Findings. Considering the 14 Gerrit projects, sub-project information brings significant improvements to the recommendation results for the NB-based approach, with medium effect size.

Iv-D Evaluation by Inspection

To understand the differences between the review process in Gerrit and GitHub, we also manually checked several pull requests to get an overall sense of the whole process. The most significant differences can be seen in the number of people involved in the review and development process of projects hosted in these systems (Table III). Although the average size of our datasets mined from GitHub is lower than the size of Gerrit datasets, the average number of reviews, reviewers and owners is much higher for projects mined from GitHub. There are usually fewer people involved in Gerrit projects, but these people tend to be more active. Whereas an average reviewer of our GitHub dataset reviewed only 7.36 pull requests and an average owner created 3.68 pull requests, 74.91 pull requests were reviewed and 41.1 pull requests were created by an average reviewer/owner in Gerrit.

The pull request number 2810222https://github.com/vuejs/vue/pull/2810 shows the typical example of GitHub review process. User tejitak found some issue in the vue project and fixed it. He created a new pull request with his solution for this project, which was then discussed by two other community members and was finally merged to the vuejs:next branch. It was his first and only pull request in vue project and we were able to find many similar cases.

On the other hand, pull request number 71010333https://go-review.googlesource.com/c/go/+/71010 shows an example of the typical review process in the Gerrit system. It was created for the project go by the user Rob Pike, who has many other contributions to this project. His pull request was reviewed by the user Emmanuel Odeke, who has also already created or reviewed many other pull requests in this project. Such cases are more common in projects hosted by Gerrit, where each project is hosted on a different server with a much smaller and more specialized community compared to GitHub.

Iv-E Threats to Validity

External validity: Threats to external validity are related to the generalizability of our study [37, 38]. Our empirical evaluation was executed on 51 open source projects, which use both Gerrit and GitHub systems for code reviews. We cannot claim that the same results would be achieved for other projects. Smaller open source projects, commercial projects or projects using other systems for code reviews might show different results.

Internal validity: Threats to internal validity are related to experimental errors and biases [37, 38]

. The main internal validity threats are about the implementation and the data collection process. We manually inspected results that looked as outliers to fix eventual issues. The dataset only contains pull requests with at least one reviewer. About bots, no steps were done to reduce them in Gerrit, as there is no automated way for identification. In GitHub, bots usually contain the string ”bot” in their name, so those were automatically removed while data mining. The RevFinder implementation was based on the information available in the article and software repository of Thongtanunam et al.

[12]. Still, we might have introduced unintentionally some slight variation. However, experimental results were consistent with previous research results.

A limitation of the GitHub REST API that should be taken into account is that responses include a maximum of 300 files per pull request. Pull requests with more than 300 pull requests appear rarely and thus we consider this as a minor limitation. Furthermore, we considered active only developers that had at least one review performed in the last 12 months.

Construct validity: Threats to construct validity relate to how much the measures used represent what researchers aim to investigate [38, 39].

Our threats are mainly related to the suitability of our evaluation metrics [37]. We used top-k accuracy and MRR for empirical evaluation, which are widely used metrics in the relevant literature and should not cause threats to construct validity [12, 14, 15]. In our experiments, we evaluate whether one of the top-k recommended reviewers actually evaluated the pull request. However, we do not know whether this reviewer was the best candidate for the specific task. Historical data might include wrong reviewers choices.

Source code reviewer recommendation algorithms are primarily expected to be used for large projects where code reviewer assignment problems are more likely to be present [15], utilization of sub-project information is consistent with this view of larger distributed systems. In a real environment, we would also have to consider some other aspects as well. The most active code reviewers would be probably assigned to a considerable number of pull requests which could lead to overburdening of certain reviewers. Therefore, workload balancing should be also taken into account [12].

V Related Works

In this work, we focus on automation of source code recommendation algorithms ([13, 12, 17, 15]), with models often based on the knowledge gathered from previous empirical studies.

The scale of our review is larger than previous studies, in which the maximum number of pull requests considered were ten-times lower orders of magnitude (293K in our case, versus 8K [21], 18K [15], 42K [12], 42K [22], 45K [23]). Only comparable study to ours is Yu et al. [16], in which 84 GitHub projects were analyzed, a subset of 650K pull-requests (919 projects, based on the GHTorrent project [40]).

Due to the scale of the dataset, a comparison with related works concerning MRR and top-k measures is appealing, to see the how the results in previous papers are generalizable. In Thongtanunam et al. [12], RevFinder was able to reach top-10 recommendation of 79% on four projects (Android, Qt, OpenStack, LibreOffice) mined from Gerrit and Rietveld. In our case, on a larger dataset, RevFinder had a top-10 accuracy of 87.39% (51 projects). However—by findings of the current paper—this dataset also included GitHub projects, thus better recommendation results. If we look into RevFinder just in 14 Gerrit projects, we get a top-10 accuracy of 79.40%— very similar to the one in the original paper, while RevFinder+ (considering sub-project information) reached 82.17%.

CoreDevRec [15] tested on five GitHub projects provides a top-5 average accuracy of 87.82%, in our case RevFinder gets 83.08% on 37 GitHub projects. The CORRECT approach [14] is reported on an industrial project with top-5 accuracy higher than 90%. However, tests on six GitHub projects report an accuracy of 85.20%, not far from 83.08% reported.

The cHRev approach based on building an expertise model on historical past code changes [21] was tested on Android (MRR=0.65), Eclipse (MRR=0.63), Mylyn (MRR=0.72), and a private dataset (MRR=0.70). In our case, Android (MRRNB+=0.68) was mined from a different period.

The RevRec hybrid approach was tested on Android (MRR=0.69), OpenStack (MRR=0.63), and Qt (MRR=0.54) projects [23]. These projects were also included in our dataset, and while authors report large improvements over RevFinder, in our case NB+ tested on the same projects yielded similar results: Android (MRRNB+=0.68), OpenStack (MRRNB+=0.57), and Qt (MRRNB+=0.54).

It is challenging to compare the results in other studies. Jeong et al. [18] report 51%-80% top-5 accuracy on Firefox and Mozilla Core projects, however, these constitute a small sample. Yu et al. [16] use precision, recall and f-measure for different top-k levels, making difficult the comparison. Finally, the ReviewBot approach based on the familiarity of developers about lines of code developed/reviewed [17], was applied experimentally to two proprietary projects (top-5=80.85%, top-5=92.31%) making direct comparison difficult. However, as reported in RevFinder comparison, ReviewBot was found to provide sub-optimal performance [12] and is generally considered a more limited approach in terms of results [15].

Vi Conclusions

Software code reviews are a key part of modern development processes, to increase the overall software quality and reduce overall costs [5, 6]. The main issue in large distributed systems development is to find the appropriate code reviewers, as idle time / wrong assignments can be costly for projects [12]. In this paper, we proposed a large-scale study to examine the performance of different source code reviewer recommendation algorithms (RevFinder, Naïve Bayes-based) to identify the best code reviewers for opened pull requests. We mined and compared data from Github and Gerrit repositories, building a large dataset of 51 projects, with more than 293K pull requests analyzed, 180K owners and 157K reviewers.

Overall, we found that there is generally no one of the two algorithms implemented that performs better in reviewers recommendation for all projects. However, a key finding is that the performance of the models is very different on the two types of repositories analyzed (Gerrit, GitHub). Evaluation by manual inspection of the pull requests showed that difference might be due to the characteristics of Gerrit and GitHub communities: while Gerrit is a more specialized community, GitHub is more based on reviewers posting one-time source code reviews. On Gerrit each reviewer has on the average a much higher number of pull requests he/she contributed to. Another finding is that exploiting additional sub-project details allowed to improve the recommendation results, bringing statistically significant results for the Naïve Bayes-based approach. However, such information is only available for Gerrit projects, making difficult to have a generalized approach for all repository types.


The work was supported from European Regional Development Fund Project CERIT Scientific Cloud (No. CZ.02.1.01/0.0/0.0/16_013/0001802). Access to the CERIT-SC computing and storage facilities provided by the CERIT-SC Center, provided under the programme ”Projects of Large Research, Development, and Innovations Infrastructures” (CERIT Scientific Cloud LM2015085), is greatly appreciated.


  • [1] M. Fagan, “Design and code inspections to reduce errors in program development,” in Software pioneers.   Springer, 2002, pp. 575–607.
  • [2] T. Baum, O. Liskin, K. Niklas, and K. Schneider, “A faceted classification scheme for change-based industrial code review processes,” in Software Quality, Reliability and Security (QRS), 2016 IEEE International Conference on.   IEEE, 2016, pp. 74–85.
  • [3] A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges of modern code review,” in Proceedings of the 2013 international conference on software engineering.   IEEE Press, 2013, pp. 712–721.
  • [4] P. C. Rigby and C. Bird, “Convergent contemporary software peer review practices,” in Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering.   ACM, 2013, pp. 202–212.
  • [5] C. F. Kemerer and M. C. Paulk, “The impact of design and code reviews on software quality: An empirical study based on psp data,” IEEE transactions on software engineering, vol. 35, no. 4, pp. 534–550, 2009.
  • [6] T. Baum, O. Liskin, K. Niklas, and K. Schneider, “Factors influencing code review processes in industry,” in Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering.   ACM, 2016, pp. 85–96.
  • [7] P. C. Rigby and M.-A. Storey, “Understanding broadcast based peer review on open source software projects,” in Proceedings of the 33rd International Conference on Software Engineering.   ACM, 2011, pp. 541–550.
  • [8] K. E. Wiegers, Peer reviews in software: A practical guide.   Addison-Wesley Boston, 2002.
  • [9] B. Rossi, B. Russo, and G. Succi, “Analysis of open source software development iterations by means of burst detection techniques,” in IFIP International Conference on Open Source Systems.   Springer, 2009, pp. 83–93.
  • [10] ——, “Modelling failures occurrences of open source software with reliability growth,” in IFIP International Conference on Open Source Systems.   Springer, 2010, pp. 268–280.
  • [11] L. Harjumaa, I. Tervonen, and A. Huttunen, “Peer reviews in real life-motivators and demotivators,” in Quality Software, 2005.(QSIC 2005). Fifth International Conference on.   IEEE, 2005, pp. 29–36.
  • [12] P. Thongtanunam, C. Tantithamthavorn, R. G. Kula, N. Yoshida, H. Iida, and K.-i. Matsumoto, “Who should review my code? a file location-based code-reviewer recommendation approach for modern code review,” in Software Analysis, Evolution and Reengineering (SANER), 2015 IEEE 22nd International Conference on.   IEEE, 2015, pp. 141–150.
  • [13] T. Baum and K. Schneider, “On the need for a new generation of code review tools,” in Product-Focused Software Process Improvement: 17th International Conference, PROFES 2016, Trondheim, Norway, November 22-24, 2016, Proceedings 17.   Springer, 2016, pp. 301–308.
  • [14] M. M. Rahman, C. K. Roy, and J. A. Collins, “Correct: code reviewer recommendation in github based on cross-project and technology experience,” in Software Engineering Companion (ICSE-C), IEEE/ACM International Conference on.   IEEE, 2016, pp. 222–231.
  • [15] J. Jiang, J.-H. He, and X.-Y. Chen, “Coredevrec: Automatic core member recommendation for contribution evaluation,” Journal of Computer Science and Technology, vol. 30, no. 5, pp. 998–1016, 2015.
  • [16] Y. Yu, H. Wang, G. Yin, and T. Wang, “Reviewer recommendation for pull-requests in github: What can we learn from code review and bug assignment?” Information and Software Technology, vol. 74, pp. 204–218, 2016.
  • [17] V. Balachandran, “Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation,” in Software Engineering (ICSE), 2013 35th International Conference on.   IEEE, 2013, pp. 931–940.
  • [18] G. Jeong, S. Kim, T. Zimmermann, and K. Yi, “Improving code review by predicting reviewers and acceptance of patches,” Research on Software Analysis for Error-free Computing Center Tech-Memo (ROSAEC MEMO 2009-006), pp. 1–18, 2009.
  • [19] V. Dedík and B. Rossi, “Automated bug triaging in an industrial context,” in Software Engineering and Advanced Applications (SEAA), 2016 42th Euromicro Conference on.   IEEE, 2016, pp. 363–367.
  • [20] N. K. S. Roy and B. Rossi, “Towards an improvement of bug severity classification,” in Software Engineering and Advanced Applications (SEAA), 2014 40th EUROMICRO Conference on.   IEEE, 2014, pp. 269–276.
  • [21] M. B. Zanjani, H. Kagdi, and C. Bird, “Automatically recommending peer reviewers in modern code review,” IEEE Transactions on Software Engineering, vol. 42, no. 6, pp. 530–543, 2016.
  • [22] X. Xia, D. Lo, X. Wang, and X. Yang, “Who should review this change?: Putting text and file location analyses together for more accurate recommendations,” in Software Maintenance and Evolution (ICSME), 2015 IEEE International Conference on.   IEEE, 2015, pp. 261–270.
  • [23] A. Ouni, R. G. Kula, and K. Inoue, “Search-based peer reviewers recommendation in modern code review,” in Software Maintenance and Evolution (ICSME), 2016 IEEE International Conference on.   IEEE, 2016, pp. 367–377.
  • [24] J. Lipčák and B. Rossi, “Rev-rec Source - Code Reviews Dataset,” 2018. [Online]. Available: https://doi.org/10.6084/m9.figshare.6462380.v1
  • [25] D. Hiemstra, “Probability smoothing,” in Encyclopedia of Database Systems.   Springer, 2009, pp. 2169–2170.
  • [26] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim, “Duplicate bug reports considered harmful… really?” in Software maintenance, 2008. ICSM 2008. IEEE international conference on.   IEEE, 2008, pp. 337–345.
  • [27] D. Rey and M. Neuhäuser, “Wilcoxon-signed-rank test,” in International encyclopedia of statistical science.   Springer, 2011, pp. 1658–1659.
  • [28] M. Kraska-Miller, Nonparametric statistics for social and behavioral sciences.   CRC Press, 2013.
  • [29] T. W. MacFarland and J. M. Yates, “Mann–whitney u test,” in Introduction to Nonparametric Statistics for the Biological Sciences Using R.   Springer, 2016, pp. 103–132.
  • [30] B. Kitchenham, L. Madeyski, D. Budgen, J. Keung, P. Brereton, S. Charters, S. Gibbs, and A. Pohthong, “Robust statistical methods for empirical software engineering,” Empirical Software Engineering, vol. 22, no. 2, pp. 579–630, 2017.
  • [31] J. Cohen, “A power primer.” Psychological bulletin, vol. 112, no. 1, p. 155, 1992.
  • [32] G. M. Sullivan and R. Feinn, “Using effect size–or why the p value is not enough,” Journal of graduate medical education, vol. 4, no. 3, pp. 279–282, 2012.
  • [33] J. Pallant, SPSS survival manual.   McGraw-Hill Education (UK), 2013.
  • [34] L. Madeyski and B. Kitchenham, “Reproducible research–what, why and how,” Wroclaw University of Technology, PRE W, vol. 8, 2015.
  • [35] ——, “Would wider adoption of reproducible research be beneficial for empirical software engineering research?” Journal of Intelligent & Fuzzy Systems, vol. 32, no. 2, pp. 1509–1521, 2017.
  • [36] X. Yang, R. G. Kula, N. Yoshida, and H. Iida, “Mining the modern code review repositories: A dataset of people, process and product,” in Proceedings of the 13th International Conference on Mining Software Repositories, 2016, pp. 460–463.
  • [37] Y. Tian, D. Lo, and J. Lawall, “Automated construction of a software-specific word similarity database,” in Software Maintenance, Reengineering and Reverse Engineering (CSMR-WCRE), 2014 Software Evolution Week-IEEE Conference on.   IEEE, 2014, pp. 44–53.
  • [38] P. Runeson and M. Höst, “Guidelines for conducting and reporting case study research in software engineering,” Empirical software engineering, vol. 14, no. 2, p. 131, 2009.
  • [39] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in software engineering.   Springer Science & Business Media, 2012.
  • [40] G. Gousios, B. Vasilescu, A. Serebrenik, and A. Zaidman, “Lean ghtorrent: Github data on demand,” in Proceedings of the 11th working conference on mining software repositories.   ACM, 2014, pp. 384–387.