Pull Request Latency Explained: An Empirical Overview

08/23/2021
by   Xunhui Zhang, et al.
University of Groningen
NetEase, Inc
0

Pull request latency evaluation is an essential application of effort evaluation in the pull-based development scenario. It can help the reviewers sort the pull request queue, remind developers about the review processing time, speed up the review process and accelerate software development. There is a lack of work that systematically organizes the factors that affect pull request latency. Also, there is no related work discussing the differences and variations in characteristics in different scenarios and contexts. In this paper, we collected relevant factors through a literature review approach. Then we assessed their relative importance in five scenarios and six different contexts using the mixed-effects linear regression model. We find that the relative importance of factors differs in different scenarios, e.g., the first response time of the reviewer is most important when there exist comments. Meanwhile, the number of commits in a pull request has a more significant impact on pull request latency when closing than submitting due to changes in contributions brought about by the review process.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

05/28/2021

Pull Request Decision Explained: An Empirical Overview

Context: Pull-based development model is widely used in open source, lea...
11/25/2020

Nudge: Accelerating Overdue Pull Requests Towards Completion

Pull requests are a key part of the collaborative software development a...
06/21/2020

Helping Software Developers through Offline Repository Based API Searching in Data MiningIntegrated Environment

Software development is getting changed so rapidly. It will be highly be...
07/08/2020

Effect of Technical and Social Factors on Pull Request Quality for the NPM Ecosystem

Pull request (PR) based development, which is a norm for the social codi...
03/10/2021

Identifying bot activity in GitHub pull request and issue comments

Development bots are used on Github to automate repetitive activities. S...
03/22/2021

Evaluating a bot detection model on git commit messages

Detecting the presence of bots in distributed software development activ...
04/13/2020

Detecting Latency Degradation Patterns in Service-based Systems

Performance in heterogeneous service-based systems shows non-determistic...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

As an important paradigm of distributed software development, pull-based development is widely used in social coding communities including GitHub and GitLab. Because of the temporal and spatial asynchrony of the project participants in this model, the latency of pull requests is an important issue. For reviewers, understanding the latency of pull requests can help them optimize the order of pull request reviews. For contributors, the perception of latency can allow them to actively participate in open source contributions and avoid abandonment behaviour (

Li et al. (2021)). For the pull-based paradigm, latency covers the entire pull request lifecycle and is an important research area to grasp the pull-based development model as a whole.

Considering the entire lifetime (see Figure 1), the pull-based development model mainly consists of the following stages.

Figure 1: Workflow of a pull request
  1. Contributors (also known as requesters or submitters) first submit the code changes of their cloned repository to reviewers for inspection in the form of pull requests.

  2. With/Without automatic continuous integration (CI) builds, reviewers manually inspect the code changes and discuss them in comments.

  3. The integrator (also called the closer or the merger) evaluates the code changes based on the review process information and then decides to merge or reject the pull request.

  4. After rejecting the pull request, the contributor or project manager may also reopen the pull request and make further changes for the final merge.

A pull request has multiple states throughout the process, including submit, review, close, merge, and reopen. Due to CI tools, the review state consists of two sub-states, namely CI construction and manual review state.

It can be seen from the above states that the lifetime of a pull request is complex, and there is a lot of research on each stage of the entire lifetime. Research in the submitting state focuses on the automatic generation of pull request descriptions (Liu et al. (2019)), reviewer recommendations (Jiang et al. (2017); Yu et al. (2014)), duplication assessment (Wang et al. (2019); Yu et al. (2018)), etc. For the review state, research focuses on the interplay between continuous integration builds and pull requests (Zampetti et al. (2019)) and the influence of static analysis tools on code review effort (Singh et al. (2017)). For the manual review state, research focuses on the prioritization method of review (v. d. Veen et al. (2015)) and commenter recommendation (Jing et al. (2017)). For the reject or merge stage, research focuses on the prediction of pull request decisions (Gousios et al. (2014)). Reopen state focuses on the evaluation and analysis of the impact of reopening (Jiang et al. (2019)). For the entire lifetime, there are studies of factors influencing pull request decisions (Tsay et al. (2014); Zhang et al. (2021)) and latency (Yu et al. (2015)), etc.

Therefore, we can see that exploring the factors across the whole lifetime of a pull request is helpful to understand the entire process and various states of the pull request. We have previously studied the factors that influence the final merge decision of pull request throughout its lifetime Zhang et al. (2021). As a critical research area in the entire lifetime of pull requests, the study of pull request latency also covers all the states of pull requests. Meanwhile, as part of the effort evaluation (Maddila et al. (2019)), pull request latency analysis and prediction can also help reviewers save review time, improve review efficiency, and optimize pull request review priority. Therefore we would like to build on our previous work (Zhang et al. (2021)) and explore the impact of various factors on pull request latency.

Although related work has explored the factors that affect the latency of pull requests, there is no systematic analysis of the influencing factors and the exploration of factors’ influence in different situations and contexts. Therefore, building on a large-scale and diverse dataset, this paper conducts an empirical study on the impact of factors in different situations and contexts on the latency of pull requests. Notably, we explore the following two research questions:

  1. [start=1,label=RQ0, leftmargin = 3em]

  2. How do factors influence pull request latency?

  3. Do the factors influencing pull request latency change with a change in context?

To answer the above research questions, we first collect a comprehensive list of factors influencing pull request latency by conducting a systematic literature review. Then, we create a large-scale and diverse dataset of these factors. Finally, we build models (mixed-effect linear regression model) for different scenarios (e.g., pull request using CI) and contexts (e.g., pull request closing time).

This paper makes the following contributions:

  1. We collect a dataset of 11,230 projects on GitHub with 63 factors and 3,347,937 pull requests relating to the latency of pull requests.

  2. We present a synthesis of the factors identified in the literature, indicating their significance and direction in inferring pull request latency.

  3. We show the importance of these factors in explaining the latency of pull requests in different scenarios and how they change within different contexts.

The rest of the paper is organized as follows. We explain our research design in Section 2. In Section 3, we show the results. We present the discussion of this paper in Section 4 and threats in Section 5. We present the conclusion in Section 6.

2 Study Design

Figure 2 shows the framework of our study. There are mainly four parts. First, we gather all the papers explaining pull request latency (systematic literature review (SLR)). Second, we collect data for factors extracted in the above step from diverse GitHub projects (data collecting (DC)). Third, we preprocess the data for further analysis (data preprocessing (DP)). Finally, we do exploratory analysis and build models to answer the research questions (statistical modelling (SM)).

Figure 2: Framework of this paper

2.1 Systematic literature review (SLR)

To gather papers related to pull request latency and collect measurable factors mentioned in previous studies, we conduct SLR following the guideline of Kitchenham and Charters (2007). There are mainly three steps, i.e., paper query, paper selection and data extraction.

2.1.1 Paper query (SLR.1)

Our previous study (Zhang et al. (2021)) found that the keyword “pull based” can be related to the “peer to peer” related studies. Therefore, to limit the search scope, we use keyword “pull based development” and “pull based model” instead of “pull based”. For the time related keyword, we choose “lifetime” (Maddila et al. (2019)) and “latency” (Yu et al. (2016)). Finally, we defined the boolean search string as follows:

(“pull based model” OR “pull based development” OR “pull request”) AND (“latency” OR “lifetime”)

We conducted the query on May 15th, 2020. We got 1,273 papers, including 164 from ACM Digital Library, 1 from IEEExplore, 5 from Web of Science, 5 from Ei Compendex and 1,098 from Google Scholar.

2.1.2 Paper selection (SLR.2)

The first author manually checked the keywords, title and abstract of each paper and excluded papers for the following reasons:

  • not written in English (4 papers)

  • have duplications (241 papers)

  • have an extension (3 papers)

  • not related to pull request latency (1,004 papers)

  • explain factors not applicable to GitHub (1 paper), i.e., factors based on mailing list (Jiang et al. (2013)).

  • not generalizable to a wider range of projects on GitHub (2 papers), i.e., Microsoft and C# related factors (Maddila et al. (2019)).

  • not measurable quantitatively (1 paper), i.e., the features relating to the pull request latency found in a qualitative study (Gousios et al. (2015)).

To avoid the missing of related papers, we conducted a one-round backward snowballing process (Jalali and Wohlin (2012)). However, we found no new articles related to the pull request latency. All in all, we found 17 papers presenting the factors related to the pull request latency (before May 2020).

2.1.3 Data extraction (SLR.3)

In total, we extract 45 features from the previous selected 17 studies together with the description of measurement, the influence direction and significance. To clearly show the extraction results, we list the symbolic representations of the factors in Table 1 and the factor’s relationship with pull request latency in Table LABEL:factor_table.

Factor Description Factor Description
Developer Characteristics
first pr first pull request? yes/no prior review num # of previous reviews in a project
core member core member? yes/no first response time # of minutes from pull request creation to the reviewer’s first response
contrib gender gender? male or female contrib affiliation contributor affiliation
same affiliation same affiliation contributor/integrator? yes/no inte affiliation integrator affiliation
social strength fraction of team members interacted with in the last three months prev pullreqs # of previous pull requests
followers # of followers at pull request creation time same user same contributor and integrator? yes/no
Project Characteristics
sloc executable lines of code team size # of active core team members in the last three months
project age # of months from project to pull request creation open pr num # of open pull requests
integrator availability latest activity of the two most active integrators test lines per kloc # of test lines per 1K lines of code
test cases per kloc # of test cases per 1K lines of code asserts per kloc # of assertions per 1K lines of code
perc external contribs % of external pull request contributions requester succ rate past pull request success rate
Pull Request Characteristics
bug fix fixes a bug? yes/no description length length of pull request description
hash tag “#” tag exists? yes/no num participants # of participants in pull request comments
ci exists uses CI? yes/no part num code # of participants in pull request and commit comments
ci latency # of minutes from pull request creation to the first CI build finish time num code comments # of code comments
reopen or not pull request is reopened? yes/no friday effect pull request submitted on a Friday? yes/no
has comments pull request has a comment? yes/no num comments # of comments
num comments con # of contributor comments at tag “@” tag exists? yes/no
num code comments con # of contributor’s code comments ci test passed all CI builds passed? yes/no
comment conflict keyword “conflict” exists in comments? yes/no
1-4[0.8pt/2pt] num commits open # of commits at pull request open time num commits close/change # of commits at close time/changed compared to open time
src churn open # of lines changed (added + deleted) at pull request open time src churn close/change # of lines changed (added + deleted) at close time/changed compared to open time
files changed open # of files touched at pull request open time files changed close/change # of files touched at close time/changed compared to open time
commits on files touched open # of commits on files touched at pull request open time commits on files touched close/change # of commits on files touched at close time/changed compared to open time
churn addition open # of added lines of code at pull request open time churn addition close/change # of added lines of code at close time/changed compared to open time
churn deletion open # of deleted lines of code at pull request open time churn deletion close/change # of deleted lines of code at close time/changed compared to open time
test churn open # of lines of test code changed (added + deleted) at pull request open time test churn close/change # of lines of test code changed (added + deleted) at close time/changed compared to open time
test inclusion open test case existing at pull request open time? yes/no test inclusion close/change test case existing at close time/changed compared to open time? yes/no
NOTE: Factors marked as are dynamic factors that can change during the review process. Here we collect their values at two different snapshots (pull request submission and close). Factors marked as are those make sense when creating a pull request.
All metrics are relative to a referenced pull request in a project.
Factors that change over time (e.g., core team) are measured using the previous three months of development activities in a project.
Table 1: Comprehensive list of the factors known to influence pull request latency on GitHub

2.2 Data collecting (DC)

According to the described measurement in related work, we collect the extracted factors from SLR. The data collection process is based on GHTorrent MySQL111http://ghtorrent-downloads.ewi.tudelft.nl/mysql/mysql-2019-06-01.tar.gz and Mongo data dump on June 1st, 2019. Also, we directly extract factors based on GitHub API and the cloned source repository.

Some pull request-related factors change dynamically throughout the pull request lifetime (e.g., the number of commits contained in a pull request changes when modifying the pull request during the review process). We refer to these factors as dynamic factors, which are shown in Table 3. Based on our previous dataset (Zhang et al. (2021)), we calculated the values of these dynamic factors at the time of pull request submission and count the changes of these factors at the time of pull request submission and closing. By adding these factors, we can determine whether the importance of these dynamic factors changes under different snapshots (pull request submission and close) during the whole lifetime of the pull request.

The final dataset offers a total of 3,347,937 pull requests from 11,230 projects. The diversity of the dataset is reflected in 6 programming languages, different projects sizes and different project activities (Zhang et al. (2021)).

2.3 Data preprocessing (DP)

There are mainly two parts for data preprocessing, i.e. unreliable data removal (DP.1) and special factor discovery (DP.2).

2.3.1 Unreliable data removal (DP.1)

There are some unexpected values for factors, including first_response_time, ci_latency and project_age. We need to fix the problem for future reliable analysis. Detailed reasoning for the unexpected problems can be seen in our technical report (Zhang et al. (2020b)).

  • first_response_time has negative values since our metric considers the comments under the related code, and some comments exist before pull request creation (0.4% pull requests have negative values).

  • ci_latency has negative values when commits exist before pull request creation, and the time of the first CI build is earlier than the creation time of a pull request (1.5%).

  • project_age has negative values where the creation time of a user account on GHTorrent is different from that on Github API (0.1%).

  • We also remove bug_fix, which has 99.3% empty values, to avoid its impact on further analysis.

2.3.2 Special factor discovery (DP.2)

Some factors would not make sense unless a precondition were met (see Table 2). E.g., factor ci_latency only make sense when ci_exists was true. Therefore, ci_exists presents a precondition contingent on which ci_latency is meaningful, also referred to as a postconditional factor. Other factors which have no dependencies will not be affected by conditions.

  • first_response_time, it only make sense when there exists comment (has_comments=1).

  • ci_test_passed and ci_latency depend on the existence of CI tools (ci_exists=1).

  • same_affiliation, it only makes sense when the contributor and integrator are different persons (same_user=0).

postconditional factor preconditional factor
first_response_time has_comments
ci_test_passed ci_exists
ci_latency
same_affiliation same_user
Table 2: Factors with dependency

Some pull request related factors are related to the review process and can change from the creation time to the close time of pull request (see Table 3). To comprehensively understand factors’ influence on pull request latency, we consider these factors when submitting (opening) and closing a pull request separately.

factor name split by process
churn_addition churn_addition_open
churn_addition_close
churn_deletion churn_deletion_open
churn_deletion_open
commits_on_files_touched commits_on_files_touched_open
commits_on_files_touched_close
files_changed files_changed_open
files_changed_close
num_commits num_commits_open
num_commits_close
src_churn src_churn_open
src_churn_close
test_churn test_churn_open
test_churn_close
test_inclusion test_inclusion_open
test_inclusion_close
Table 3: List of dynamic factors

2.4 Statistical modeling (SM)

To analyze the influence of factors on pull request latency, we first discover the relationship between factors and solve the collinearity and multicollinearity problems (see SM.1 and SM.2 in Figure 2).222https://quantifyinghealth.com/correlation-collinearity-multicollinearity/ Then we transform the data for further interpretation (SM.3). Finally, we build models in different scenarios (SM.4) and models within different contexts (SM.5).

2.4.1 Discover factor relationship

To ensure the reliability for the building of further analysis models, we removed the collinearity and multicollinearity problems via two steps.

Collinearity removal (SM.1). We first calculated the correlations among all the factors. For continuous factors, we used the Spearman correlation coefficient () (Gousios et al. (2014)) and marked as a strong correlation (Gousios et al. (2014)). For categorical factors, we used Cramér’s V value ()333https://www.statstest.com/cramers-v-2/, where 444http://www.real-statistics.com/chi-square-and-f-distributions/effect-size-chi-square/ was considered as a strong correlation (Cohen (1969)). For the correlation between continuous and categorical factors, we used the partial Eta-squared value () (Jones (2019)) and marked as a strong correlation (Cohen (1969)). The heatmap of correlation results can be seen in the technical report.555https://github.com/zhangxunhui/ESE_pull_request_latency Then we extracted the strongly correlated factors and created strong correlation networks, which are also shown in the technical report.

After calculating factor correlation, we removed the strongly correlated factors by keeping popular factors while removing factors strongly correlated with many others. The description of the workflow666https://github.com/zhangxunhui/ESE_pull_request_latency/blob/main/report.pdf and the code777https://github.com/zhangxunhui/ESE_pull_request_latency/blob/main/ese_latency_factor_selection.py are open source.

Multicollinearity removal (SM.2)

For the multicollinearity problem, we excluded factors with variance inflation factor (VIF) values

5, as such values could inflate variance, measured using the vif function of the car package in R (Cohen et al. (2013)). In this way, we removed num_comments.

2.4.2 Ease of interpretation (SM.3)

We stabilize the variance in features log-transforming continuous variables after adding “0.5” (continuous factors) (Hall et al. (2009)

). Then we transformed the features into a comparable scale with a mean value of “0” and a standard deviation of “1”.

2.4.3 Factors’ influence on pull request latency

Figure 3: Various situations of model construction

To explain the influence on pull request latency of all the factors, we need to consider that different factors only make sense in different situations (e.g., factor ci_latency only makes sense when a comment exists). Figure 3 presents all the situations of model construction in this paper, where situation I-V are used to explain factors’ influence. In these situations, we include as many factors and relevant pull requests as possible to explore which of all potential factors are more important in different cases.

  1. [label=, itemsep=0pt, topsep=0pt]

  2. Pull requests at submission time. We can learn how factors that can be measured at the pull request submission time influence latency in this situation. Here, factors marked by (see Table 1) are included when building up the model.

  3. Pull requests at close time. Comparing to situation I, factors merged during review process (e.g., has_comments) are included. Both close time and submission time are just snapshots of the entire lifetime of pull requests, which can help us understand the impact of different factors on pull request latency throughout the lifetime.

  4. Pull requests with comments (has_comments=1). The first response latency of a pull request (first_response_time) only makes sense when there exists comment.

  5. Pull requests using CI tools (ci_exists=1). The latency of CI builds (ci_latency) and the build results (ci_test_passed) only make sense when pull requests use CI tools.

  6. Pull requests submitted and integrated by different users (same_user=0). Whether contributor and integrator of a pull request are affiliated to the same organization (same_affiliation) only makes sense when this pull request is submitted and integrated by different people.

2.4.4 Influence change within different contexts

To explore the relevance of context in explaining pull request latency, we studied six scenarios (see context VI-XI in Figure 3). Context VI-X inherit from our previous study Zhang et al. (2021). We use the same set of factors in each context but different pull requests to build our model for studying change of factor’s importance.

  1. [label=, itemsep=0pt, topsep=0pt]

  2. Time context: the period before June 1st, 2016, June 1st, 2016 and June 1st, 2018, and the period after June 1st, 2018.

  3. Project context: team size small (), medium (), and large ().

  4. Pull request context: has_comments=1 and has_comments=0.

  5. Tool context: ci_exists=1 and ci_exists=0.

  6. Developer characteristic: same_user=1 and same_user=0.

  7. Process context. We select the dynamic factors that can change during the review process (refer to factors marked by in Table 1). We study whether the importance of these factors changes after the review process. We can understand whether the impact of factors changes under different snapshots in the pull request lifetime through this context. It is also possible to understand the effect of pull request modifications on pull request latency during the review process.

2.5 Model interpretation

The resulting mixed-effect linear regression models explain the influence of factors in models and their relative relevance. Section 3 presents the findings from these mixed-effect linear regression models. We report the regression coefficients together with their p-values. The term p-value indicates the statistical significance of a factor, which was indicated by asterisks: . Meanwhile, we present the explained variance of each factor derived from ANOVA type-II analysis (Langsrud (2003)). Finally, the percentage of explained variance (calculated by ) was used as a proxy for the relative importance of a factor (Overney et al. (2020)). For the goodness of fit of each model, we report both marginal and conditional (Nakagawa and Schielzeth (2013)), which considers the variance of the fixed effects and both fixed and random effects, respectively.

3 Results

In this section, we first present how factors influence pull request latency (answering RQ1). It includes models for pull requests at open time, pull requests at close time, pull requests with comments, pull requests using CI tools, and pull requests submitted and integrated by different users (see Section 2.4.3). After that, we present how factors influence pull request latency change with context change (answering RQ2), consisting of models for six contexts, namely time, project, pull request, tool, developer, and process contexts (see Section 2.4.4).

3.1 RQ1: How do factors influence pull request latency?

3.1.1 Open time

Columns 2 and 3 in Table 7 show the influence of factors at pull request open time, where description_length (row 2), src_churn_open (row 3), prev_pullreqs (row 4), integrator_availability (row 5) and open_pr_num (row 6) are the top 5 important factors. They together explain 88.7% of the total variance and are all significantly important regarding the latency of pull requests.

Among all the factors, the length of pull request description (description_length) contributes the most to the latency of pull request (46.3% variance), which positively correlates with the latency. The pull request text description’s length may indicate the pull request’s complexity, and it’s likely that more complex or challenging to understand pull requests take longer to review.

The amount of source code change (src_churn_open) ranks the second (20.4% variance), which is also positively correlated with pull request latency. One possible explanation is that the amount of source code changed affects the amount of time the maintainer has to check the correctness of the software.

In addition to these technical factors, the contributor’s experience (prev_pullreqs) also has a sizable effect, which explains 8.9% variance and negatively correlates with the latency of pull requests. Comparing to other developer characteristics, the contributor’s experience in the target project is more decisive regerding the latency at pull request open time.

For project characteristics, the integrator_availability and open_pr_num have sizable effects, explaining 6.7% and 6.5% variance, respectively. These results indicate that the top active reviewers and workload of projects influence the latency of pull requests, increasing the latency with the increase of description length.

When submitting a pull request, both social and technical factors (pull request, developer, and project characteristics) influence the latency of pull request with sizable effects, among which the length of pull request description and the size of the source code change are the most influential.

3.1.2 Close time

Columns 4 and 5 in Table 7 present the influence of factors at pull request close time, among which has_comments (row 21), same_user (row 22), description_length (row 2), num_code_comments (row 23) and num_commits_close (row 14) are the top 5 important factors, explaining 80.8% of the total variance and are all significantly important.

Factor has_comments ranks the first in explaining pull request latency (37.5% variance) with a positive correlation. Communication between reviewers and the contributor may indicate that reviewers ask for changes or ask the contributor relevant questions to understand the contribution content.

For factor same_user, it explains 16.9% of the total variance with a positive correlation. It indicates that when the contributor and integrator are not the same (same_user=0), the latency of a pull request is more likely to increase compared to the situation when the contributor and integrator are the same. A possible explanation is that when reviewing pull requests submitted by other people, reviewers need to spend more time making the final decision.

Among all the factors, num_code_comments has sizable positive effect, which explains 8.4% of the total variance. This indicates that the presence or absence of comments on the code during the review process has a moderate correlation with pull request latency. Moreover, comments on the code increase the decision time.

For the length of pull request description, its importance ranks the 3rd to explain pull request at close time (12.3% variance). It still has a sizable impact in increasing pull request latency even comparing with factors that occurred during the review process.

We see that factor num_commits_close has a sizable positive effect by explaining 5.7% of the total variance. This indicates that the number of commits in the pull request has a moderate impact on the pull request latency, as seen in the snapshot of the pull request closing. In Session 3.2.6, we will examine how the same factors differ in the interpretation of pull request latency in the two contexts of pull request submission and closing.

From an overall perspective, the close time model fits better than the submission model according to the R-square values. The result is likely to be related to adding factors that occurred during the review process (e.g., has_comments) and the change of dynamic factors (e.g., num_commits).

When closing a pull request, factors that occurred during the review process (e.g., has_comments) have a significant impact on the latency of pull request, which weakens the effect of factors measured at pull request submission time. Meanwhile, the overall fitness of the model becomes better by adding the factors that occurred during the review process.

3.1.3 Pull requests with comments

Columns 6 and 7 in Table 7 present the influence of factors when pull requests have comments (has_comments=1), where first_response_time (row 28), num_comments (row 29), num_code_comments (row 23), hash_tag (row 24) and prev_pullreqs (row 4) rank the top 5, explaining 95.4% of the total variance.

Among all the factors, we can see that the latency is highly dependent on the comments during the review process, where the importance of first_response_time, num_comments and num_code_comments ranks the 1st, 2nd and 3rd, explaining 58.7%, 31.4% and 2.6% variance respectively. However, for factor hash_tag and prev_pullreqs, which ranks the 4th and 5th, explaining only 1.8% and 0.9% variance respectively.

When there exist comments in pull requests, comment-related factors during the review process play a decisive role in the latency of pull requests. The latency of the first reviewer’s response is the most important.

3.1.4 Pull requests using CI tools

Columns 8 and 9 in Table 7 show the results of factors’ influence on pull request latency when pull requests use CI tools. The top 5 important factors are has_comments (row 21), ci_latency (row 30), ci_test_passed (row 31), num_code_comments (row 23) and same_user (row 22), with 64.7% variance explained in total. All of the five factors are factors that occurred during the review process. For factors that can be measured at the opening time of pull request, they also have a sizable effect although not ranking in the top 5. E.g., description_length (row 2) and integrator_availability (row 5), they explain 6.1% and 4.6% of the total variance respectively.

Factors ci_latency and ci_test_passed explain 18.2% and 10.4% of the total variance, respectively, which indicate that for pull requests using CI tools, the time used for making decisions is moderately relevant to the build time and build outcome of CI tools. For the influence direction, pull requests that need longer CI builds are more likely to take more time for review. Further, pull requests which passed the CI builds are more likely to be handled in a shorter time.

For other situations except for ci_exists=1 in Table 7, we can see that pull requests using CI tools are likely to take more time to be handled (row 8).

When pull requests use CI tools, both factors that occurred during the review process (e.g., has_comments) and factors that can be measured at pull request submission time (e.g., description_length) have a sizable effect on pull request latency. Meanwhile, we find that the usage of CI tools slows down the processing of pull requests.

3.1.5 Pull requests submitted and integrated by different people

Columns 10 and 11 in Table 7 present the influence of factors on pull request latency when pull requests are submitted and integrated by different people. Among all the factors, has_comments (row 21), num_code_comments (row 23), description_length (row 2), integrator_availability (row 5) and num_commits_close (row 14) rank the top 5, explaining 70.1% of the total variance.

Factor has_comments stands out, which explains 33.9% variance and is much higher than all the other factors. A possible explanation is that when the contributor and integrator are not the same, the review process highly depends on communication, which helps the reviewers understand the contribution and make the final decision.

When pull requests are submitted and integrated by different people, comments under a pull request play the most important role in influencing pull request latency. This reflects that pull request reviewers rely on communicating with contributors through comments for the standardised pull-based development, which is likely to bring time costs to the review process.

3.2 RQ2: Do the factors influencing pull request latency change with a change in context?

3.2.1 Time context

Columns 2-7 of Table 8 present the results of comparison among pull requests in different time contexts. Among all the factors, the variance explained by factor has_comments (row 2) and project_age (row 20) change more than 5%.

For factor project_age, its explained variance increases directly from 0.1% to 26.2% as projects evolve, with negative influence on pull request latency. Since the project age can reflect the project maturity to some extent, this result indicates that project maturity is increasingly important to distinguish the pull request latency as projects evolve. Meanwhile, the more mature the project, the faster the review of the pull request will be.

For factor has_comments, the explained variance decreases from 43.1% to 27.9% as projects evolve. It indicates that comment under a pull request cannot be a decisive factor in measuring its latency as the project evolves. We calculated the percentage of pull requests with comments for each period and found that the ratio decreases as projects evolve (62.7% for the period before 1st June, 2016; 61.3% for the period between 1st June, 2016-1st June, 2018; 57.7% for the period after 1st June, 2018). However, the percentage of pull requests using CI tools increases as projects evolve (65.7% for the period before 1st June, 2016; 83.9% for the period between 1st June, 2016-1st June, 2018; 86.0% for the period after 1st June, 2018). CI tools are likely replacing the manual review process as projects evolve, leading to decreased importance of comments.

As the project evolves, the maturity of the project becomes more and more important for determining pull request latency, while the comment existence no longer plays a decisive role.

3.2.2 Project context

Columns 2-7 of Table 9 present the results of comparison among projects with different team sizes when submitting pull requests. Among all the factors, the variance explained by factor num_code_comments (row 7) and same_user (row 3) change more than 5%.

For factor num_code_comments, the explained variance increases from 4.2% to 15.8% with the increase of team size. We calculate the median value of project workload (open_pr_num) with different team sizes and find that the workload increases a lot with the team size (small: 6, mid: 13, large: 77). Therefore, it’s likely that the increased workload distracts the reviewers, so if code comments exist, it may take longer to wait for the changes to be reviewed again.

For factor same_user, its explained variance decreases from 21.7% to 10.5% with the increase of team size. It’s likely that due to the standardization of the pull-based development process, pull requests need to be reviewed by others. We calculated the ratio of pull requests submitted and integrated by the same user for different team sizes (small: 45.3%, mid: 44.1%, large: 40.5%). It’s likely that the decrease of pull requests submitted and integrated by the same user lead to the decline in importance in explaining the pull request latency.

As the team size of projects become larger, the discussion under pull request code snippets becomes more and more important for explaining pull request latency, while the relationship between contributor and integrator becomes less important.

3.2.3 Pull request context

Columns 8-11 of Table 8 present the comparison results between pull requests with and without comments. Among all the factors, same_user (row 2), num_commits (row 6), src_churn (row 8) and prev_pullreqs (row 14) are outstanding, with explained variance change more than 5%.

For factor same_user, the explained variance increases from 8.8% (has_comments=true) to 50.8% (has_comments=false), which indicates that when there does not exist comment, whether submitted and integrated by the same user, plays a decisive role in the latency of pull request. Statistically, we calculated the latency (in minutes) of pull requests in different situations (see Table 4). It is clear to see that the although the median latency of has_comments=false situation is much smaller than has_comments=true situation, the discrimination for same_user is much stronger when no comment exists (with more than 10 times difference in pull request median latency). This indicates that contributors can easily decide according to the project standard by themselves. In contrast, for others, it takes some time to complete the review and make the final decision, even though no need to communicate with the contributor.

has_comments=true has_comments=false
same_user=true 1,348 30
same_user=false 2,881 311
Table 4: Pull request median latency (in minutes) for has_comments and same_user cross situations

For factor num_commits, the explained variance decreases from 28.1% (has_comments=true) to 5.9% (has_comments=false). A possible explanation is that when there exist comments, it’s likely that the pull request needs to be modified to meet project standards. Thus its latency is highly dependent on the code changes during the review process.

Likewise, for factor src_churn, it’s directly related to the code change during review process. Therefore, its explained variance decreases from 14.6% (has_comments=true) to 3.2% (has_comments=false).

For factor prev_pullreqs, the explained variance decreases from 8.2% (has_comments=true) to 0.1% (has_comments=false). It’s likely that when there exist comments, experienced contributors can modify their contributions easier and faster than the non-experienced.

Comparing to pull requests without comments, when there exists communication between the contributor and reviewers, the size of the contribution and the contributor’s experience become more important in determining the latency of pull requests. Without communication, the relation between contributor and integrator plays a decisive role in pull request latency.

3.2.4 Tool context

Columns 8-11 of Table 9 present the results of comparison between pull requests using CI tools and not using CI tools. Among all the factors, only same_user (row 3) has more than 5% change of explained variance, which increases from 14.1% (ci_exists=true) to 25.3% (ci_exists=false). Likewise, we calculate the latency (in minutes) in different situations (see Table 5). We can see from the result that using CI tools increases the latency of pull requests in an overall perspective. Contributors likely need to spend more time on modifying contributions to meet project standards. When not using CI tools, the latency of pull request is easier to distinguish using factor same_user (with more than 18 times difference in pull request median latency). A possible explanation is that pull request decisions can be made quickly for developers with merge access due to the lack of automatic feedback from CI tools. However, it takes relatively long for other people’s contributions to be reviewed.

ci_exists=true ci_exists=false
same_user=true 310 51
same_user=false 1,461 926
Table 5: Pull request median latency (in minutes) for ci_exists and same_user cross situations

The use of CI tools increases the overall latency of pull requests. The latency differs significantly for pull requests that do not use CI tools, considering the relationship between the submitter and the integrator.

3.2.5 Developer context

Columns 8-11 of Table 10 present the results of pull requests within different developer contexts (whether contributor and integrator are the same). Among all the factors, has_comments (row 21), description_length (row 2), integrator_availability (row 5) and open_pr_num (row 6) are outstanding, with explained variance change of more than 5% across different contexts.

For factor has_comments, the explained variance decreases from 50.3% (same_user=true) to 33.8% (same_user=false). According to Table 4, factor has_comments is more distinguishable for pull request latency when same_user=true. For factor description_length, the explained variance decreases from 17.6% (same_user=true) to 9.1% (same_user=false). Through the data statistics, we found that the description information was generally long for cases with different integrators and contributors (same_user=true median: 19, same_user=false median: 26). This illustrates that while self-integrated pull request description messages are generally shorter and in this case, it is easier to explain the latency of the pull request by the length of description.

For factor integrator_availability, the explained variance increases from 2.1% (same_user=true) to 8.8% (same_user=false). The result may be explained by the fact that only pull request reviewed by others highly depends on the availability of other integrators. Likewise, the workload (open_pr_num) of projects only have a sizable effect on the latency of pull request submitted and integrated by different people.

If the contributor and integrator are the same, pull request latency depends on the communication between the contributor and reviewers and the length of the pull request description. When the contributor and integrator are different, pull request latency depends on the project workload and availability of active integrators.

3.2.6 Process context

Columns 2-5 of Table 10 present the results of comparison for pull requests at submission time and close time. We study whether the dynamic factors perform differently regarding pull request latency (column 2-5) and whether the change of these factors influences the latency of pull requests (column 6-7).

Among all the factors, the explained variance of description_length decreases from 46.3% (open time) to 36.5% (close time). It indicates that the description length becomes less important in determining the latency of pull requests as the review process evolves. During the review process, pull requests can be modified, which may be the reason for the importance increase of factor num_commits (0.3% at submission time, 22.8% at close time). To verify the explanation, we built another model (process change) by replacing factor num_commits with its change during the review process (num_commit_change), and found that the change of commits is significantly important for pull request latency with a sizable effect (row 14, columns 6 and 7; 34.5% variance).

Surprisingly, we find that the explained variance of factor src_churn decreases (20.4% at submission time, 11.1% at close time). A possible explanation is that the change in the number of commits indicates the change in time compared to the change in lines of source code, which in turn can better explain the time consumption of the pull request review process.

The model fits better at the close time than the submission time. Meanwhile, we find that if we replace the dynamic factors with the amount of variation during the review process, the model fits better. It indicates that the factors explain the pull request latency better for snapshots that are closer to the merge/reject state. Also, the change of factors during the review process has a significant impact on pull request latency.

For dynamic factors that can change during the review process, the number of commits significantly influences the latency of pull requests. As pull requests approach the merge/reject state, the factors become more explanatory of the pull request delay.

4 Discussion

By studying the factors affecting pull request latency, we now have a deeper understanding of the entire lifetime of pull requests. To better understand the impact of the review process on pull request latency, we add dynamic factors to the data collection in the pull request submission and closure states compared to previous work. We analyze the impact of factors on pull request latency in different situations and contexts with a more comprehensive dataset.

4.1 Interpretation

RQ1 How do factors influence pull request latency? The impact of factors on pull request latency cannot be simply summarized in a holistic manner alone. For example, when submitting a pull request, the length of the pull request description, the number of source code changes, developer experience, project reviewer activity, and project workload are the five most important factors. However, when we consider the snapshot of when the pull request is closed, we find that all the factors are less important in explaining the pull request’s latency than those generated during the review process. Similarly, for pull requests with comment information, the latency of the first comment and the number of comments are most important. Pull requests that use CI tools depend on the CI builds latency and the build results. If the person who finally integrates the pull request is not the same as the submitter, it depends on comment information. This phenomenon indicates that in the future, when conducting research related to pull request latency or building prediction tools, we should consider the characteristics of pull requests in-depth and build different models.

RQ2 Do the factors influencing pull request latency change with a change in context? Yes. The effect of factors on pull request latency is subject to change with context, and we consider the impact of the same set of factors in different contexts. The project’s maturity is found to have a better explanatory role with pull request latency as it evolves. When the project team size increases, the code comments during the process become more important. Whether the pull request contains discussion information is correlated with whether its integrator and contributor are the same. If one ends up making decisions about own pull requests, this decision depends on the comments during the review process, which in turn affects the review latency. For the whole pull request review process, the closer a factor is to the final state, the better the effect of factors on the interpretation of the pull request, and the change of factors in the process has a significant impact on the latency of the pull request. These results illustrate that the importance of factors on the interpretation of pull request is changed with the change of pull request state in the lifetime of the pull request.

4.2 Implication

The findings of this paper have important implications for future research and practice. First of all, our results on different scenarios and contexts can guide future researchers to consider different factors according to their own situation when conducting research on pull request latency (e.g., when constructing a latency prediction model, different situations need to be considered differently). Secondly, we open-source the relevant dataset (Zhang et al. (2021)) and model-building scripts888https://github.com/zhangxunhui/ESE_pull_request_latency. Researchers can quickly replicate the results in different situations when considering the influence of new factors in the future. Finally, we can make the following actionable suggestions: When submitting a pull request, the contributor should write a clear and concise description to facilitate reviewers to understand the contribution and speed up the review process. Meanwhile, the code change should meet the project standards as much as possible before submission, reducing the influence of further modification on the latency. For code reviewers, quick response to contributions can greatly reduce the total time consumption of the review process.

5 Threats to validity

This work is based on our previous dataset (Zhang et al. (2020a)). The construction of the dataset relies on the measurement methods of related work and the GitHub API and GHTorrent dataset. Therefore, this paper inherits all the threats of the previous work.

At the same time, the paper also has the following limitations:

  1. Different algorithms have different ways of evaluating the importance of factors. Here we use the mixed-effect logistic regression model. The reason is that while explaining the relative importance of factors, we can determine their linear influence.

  2. When preprocessing the dataset, we deleted factor bug_fix, which had many missing values. Thus this paper lacks a discussion about this factor.

  3. Although we have discussed the six kinds of contexts, there are many more contexts, and we have not divided the contexts according to each factor. And because the dataset is just diverse to a certain extent, it can only guarantee a certain degree of diversity in each context.

  4. This paper only considers the factors that can be measured on GitHub and only considers the factors that can be quantitatively measured. We did not verify the generality of the results on other platforms.

6 Conclusion

This paper sorts out the influence of factors on pull request latency. Our study shows that the relatively importance of factors in different scenarios are different. E.g., when the pull request is closed, the process-related factors are more important than the pull request description. At the same time, the importance of factors varies with context. E.g., when considering the same set of factors, due to changes brought about by the review process, the number of commits in a pull request becomes more important when the pull request is closed. This paper can be helpful to follow-up research and practice. We created a dataset and open-sourced the script for model replication, which facilitates future research. Meanwhile, follow-up research can reuse our results and build up models according to different contexts and scenarios. Finally, we put forward actionable suggestions for pull request contributors and reviewers. E.g., contributors should refine the text description when submitting a pull request, and reviewers should respond to contributions as quickly as possible.

Acknowledgements.
This work is supported by National Grand R&D Plan (Grant No.2020AAA0103504).

References

  • O. Baysal, O. Kononenko, R. Holmes, and M. W. Godfrey (2012) The secret life of patches: a firefox case study. In 2012 19th Working Conference on Reverse Engineering, Vol. , pp. 447–455. External Links: Document, ISSN 1095-1350 Cited by: Table 6.
  • O. Baysal, O. Kononenko, R. Holmes, and M. W. Godfrey (2016) Investigating technical and non-technical factors influencing modern code review. Empirical Software Engineering 21 (3), pp. 932–959. External Links: Document, ISBN 1573-7616, Link Cited by: Table 6.
  • J. H. Bernardo, D. Alencar da Costa, and U. Kulesza (2018) Studying the impact of adopting continuous integration on the delivery time of pull requests. In 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR), Vol. , pp. 131–141. Cited by: Table 6.
  • A. Bosu and J. C. Carver (2014) Impact of developer reputation on code review outcomes in oss projects: an empirical investigation. In Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’14, New York, NY, USA. External Links: ISBN 9781450327749, Link, Document Cited by: Table 6.
  • J. Cohen, P. Cohen, S. G. West, and L. S. Aiken (2013) Applied multiple regression/correlation analysis for the behavioral sciences. Routledge. Cited by: §2.4.1.
  • J. Cohen (1969) Statistical power analysis for the behavioral sciences. Academic press. Cited by: §2.4.1.
  • G. Gousios, A. Zaidman, M. Storey, and A. v. Deursen (2015) Work practices and challenges in pull-based development: the integrator’s perspective. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1, pp. 358–368. Cited by: 7th item.
  • G. Gousios, M. Pinzger, and A. v. Deursen (2014) An exploratory study of the pull-based software development model. In Proceedings of the 36th International Conference on Software Engineering, ICSE 2014, New York, NY, USA, pp. 345–355. External Links: ISBN 9781450327565, Link, Document Cited by: §1, §2.4.1, Table 6.
  • Hall, Daniel, and B. (2009) Data analysis using regression and multilevel/hierarchical models.. Journal of the American Statistical Association. Cited by: §2.4.2.
  • C. Hechtl (2020) On the influence of developer coreness on patch acceptance: a survival analysis. . Cited by: Table 6.
  • M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig (2016) Usage, costs, and benefits of continuous integration in open-source projects. In 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), Vol. , pp. 426–437. Cited by: Table 6.
  • D. Hu, T. Wang, J. Chang, Y. Zhang, and G. Yin (2018) Bugs and features, do developers treat them differently?. pp. 250–255. Cited by: Table 6.
  • N. Imtiaz, J. Middleton, J. Chakraborty, N. Robson, G. Bai, and E. Murphy-Hill (2019) Investigating the effects of gender bias on github. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), Vol. , pp. 700–711. Cited by: Table 6.
  • S. Jalali and C. Wohlin (2012) Systematic literature studies: database searches vs. backward snowballing. In Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, Vol. , pp. 29–38. External Links: Document Cited by: §2.1.2.
  • J. Jiang, A. Mohamed, and L. Zhang (2019) What are the characteristics of reopened pull requests? a case study on open source projects in github. IEEE Access 7 (), pp. 102751–102761. Cited by: §1, Table 6.
  • J. Jiang, Y. Yang, J. He, X. Blanc, and L. Zhang (2017) Who should comment on this pull request? analyzing attributes for more accurate commenter recommendation in pull-based development. Information and Software Technology 84, pp. 48–62. Cited by: §1.
  • Y. Jiang, B. Adams, and D. M. German (2013) Will my patch make it? and how fast? case study on the linux kernel. In 2013 10th Working Conference on Mining Software Repositories (MSR), Vol. , pp. 101–110. External Links: Document, ISSN 2160-1852 Cited by: 5th item.
  • J. Jing, Y. Yun, J. He, X. Blanc, and Z. Li (2017) Who should comment on this pull request? analyzing attributes for more accurate commenter recommendation in pull-based development - sciencedirect. Information & Software Technology 84 (C), pp. 48–62. Cited by: §1.
  • J. S. Jones (2019) Learn to use the eta coefficient test in spss with data from the niosh quality of worklife survey (2014). Cited by: §2.4.1.
  • B. Kitchenham and S. Charters (2007) Guidelines for performing systematic literature reviews in software engineering. . Cited by: §2.1.
  • O. Kononenko, T. Rose, O. Baysal, M. Godfrey, D. Theisen, and B. de Water (2018) Studying pull request merges: a case study of shopify’s active merchant. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP ’18, New York, NY, USA, pp. 124–133. External Links: ISBN 9781450356596, Link, Document Cited by: Table 6.
  • O. Langsrud (2003) ANOVA for unbalanced data: use type ii instead of type iii sums of squares. Kluwer Academic Publishers. Cited by: §2.5.
  • A. Lee and J. C. Carver (2017) Are one-time contributors different? a comparison to core and periphery developers in floss repositories. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Vol. , pp. 1–10. Cited by: Table 6.
  • Z. Li, Y. Yu, T. Wang, G. Yin, and H. Wang (2021) Are you still working on this an empirical study on pull request abandonment. IEEE Transactions on Software Engineering PP (99), pp. 1–1. Cited by: §1.
  • Z. Liu, X. Xia, C. Treude, D. Lo, and S. Li (2019) Automatic generation of pull request descriptions. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 176–188. Cited by: §1.
  • C. Maddila, C. Bansal, and N. Nagappan (2019) Predicting pull request completion time: a case study on large scale cloud services. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 874–882. Cited by: §1, 6th item, §2.1.1.
  • S. Nakagawa and H. Schielzeth (2013) A general and simple method for obtaining r2 from generalized linear mixed-effects models. Methods in ecology and evolution 4 (2), pp. 133–142. Cited by: §2.5.
  • C. Overney, J. Meinicke, C. Kstner, and B. Vasilescu (2020) How to not get rich: an empirical study of donations in open source. In ICSE ’20: 42nd International Conference on Software Engineering, Cited by: §2.5.
  • G. Pinto, L. F. Dias, and I. Steinmacher (2018) Who gets a patch accepted first? comparing the contributions of employees and volunteers. In Proceedings of the 11th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE ’18, New York, NY, USA, pp. 110–113. External Links: ISBN 9781450357258, Link, Document Cited by: Table 6.
  • C. Sadowski, E. Söderberg, L. Church, M. Sipko, and A. Bacchelli (2018) Modern code review: a case study at google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP ’18, New York, NY, USA, pp. 181–190. External Links: ISBN 9781450356596, Link, Document Cited by: Table 6.
  • D. Singh, V. R. Sekar, K. T. Stolee, and B. Johnson (2017) Evaluating how static analysis tools can reduce code review effort. In 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), Cited by: §1.
  • J. Tsay, L. Dabbish, and J. Herbsleb (2014) Influence of social and technical factors for evaluating contribution in github. In Proceedings of the 36th International Conference on Software Engineering, ICSE 2014, New York, NY, USA, pp. 356–366. External Links: ISBN 9781450327565, Link, Document Cited by: §1.
  • E. v. d. Veen, G. Gousios, and A. Zaidman (2015) Automatically prioritizing pull requests. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, Vol. , pp. 357–361. Cited by: §1.
  • Q. Wang, B. Xu, X. Xia, T. Wang, and S. Li (2019) Duplicate pull request detection: when time matters. In Proceedings of the 11th Asia-Pacific Symposium on Internetware, pp. 1–10. Cited by: §1.
  • Y. Yu, H. Wang, V. Filkov, P. Devanbu, and B. Vasilescu (2015) Wait for it: determinants of pull request evaluation latency on github. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, Vol. , pp. 367–371. Cited by: §1, Table 6.
  • Y. Yu, H. Wang, G. Yin, and C. X. Ling (2014) Who should review this pull-request: reviewer recommendation to expedite crowd collaboration. In 2014 21st Asia-Pacific Software Engineering Conference, Vol. 1, pp. 335–342. External Links: Document, ISSN 1530-1362 Cited by: §1.
  • Y. Yu, Z. Li, G. Yin, T. Wang, and H. Wang (2018) A dataset of duplicate pull-requests in github. In Proceedings of the 15th International Conference on Mining Software Repositories, pp. 22–25. Cited by: §1.
  • Y. Yu, G. Yin, T. Wang, C. Yang, and H. Wang (2016) Determinants of pull-based development in the context of continuous integration. Science China Information Sciences 59 (8), pp. 080104. External Links: Document, ISBN 1869-1919, Link Cited by: §2.1.1, Table 6.
  • F. Zampetti, G. Bavota, G. Canfora, and M. D. Penta (2019) A study on the interplay between pull request review and continuous integration builds. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), Vol. , pp. 38–48. External Links: Document, ISSN 1534-5351 Cited by: §1.
  • X. Zhang, A. Rastogi, and Y. Yu (2020a) On the shoulders of giants: a new dataset for pull-based development research. In Proceedings of the 17th International Conference on Mining Software Repositories, pp. 543–547. Cited by: §5.
  • X. Zhang, A. Rastogi, and Y. Yu (2020b) Technical Report. Note: https://github.com/zhangxunhui/new_pullreq_msr2020/blob/master/technical_report.pdf[Online; accessed 3-March-2021] Cited by: §2.3.1.
  • X. Zhang, Y. Yu, G. Gousios, and A. Rastogi (2021) Pull request decision explained: an empirical overview. External Links: 2105.13970 Cited by: §1, §1, §2.1.1, §2.2, §2.2, §2.4.4.
  • X. Zhang, Y. Yu, T. Wang, A. Rastogi, and H. Wang (2021) Cited by: §4.2.
  • Y. Zhao, A. Serebrenik, Y. Zhou, V. Filkov, and B. Vasilescu (2017) The impact of continuous integration on other software development practices: a large-scale empirical study. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), Vol. , pp. 60–71. Cited by: Table 6.