DeepAI
Log In Sign Up

How does Working from Home Affect Developer Productivity? – A Case Study of Baidu During COVID-19 Pandemic

Nowadays, working from home (WFH) has become a popular work arrangement due to its many potential benefits for both companies and employees (e.g., increasing job satisfaction and retention of employees). Many previous studies have investigated the impact of working from home on the productivity of employees. However, most of these studies usually use a qualitative analysis method such as survey and interview, and the studied participants do not work from home for a long continuing time. Due to the outbreak of coronavirus disease 2019 (COVID-19), a large number of companies asked their employees to work from home, which provides us an opportunity to investigate whether working from home affects their productivity. In this study, to investigate the difference of developer productivity between working from home and working onsite, we conduct a quantitative analysis based on a dataset of developers' daily activities from Baidu Inc, one of the largest IT companies in China. In total, we collected approximately four thousand records of 139 developers' activities of 138 working days. Out of these records, 1,103 records are submitted when developers work from home due to COVID-19 pandemic. We find that WFH has both positive and negative impacts on developer productivity in terms of different metrics, e.g., the number of builds/commits/code reviews. We also notice that working from home has different impacts on projects with different characteristics including programming language, project type/age/size. For example, working from home has a negative impact on developer productivity for large projects. Additionally, we find that productivity varies for different developers. Based on these findings, we get some feedbacks from developers of Baidu and understand some reasons why WFH has different impacts on developer productivity.

READ FULL TEXT VIEW PDF

page 1

page 2

page 3

page 4

01/12/2021

The Daily Life of Software Engineers during the COVID-19 Pandemic

Following the onset of the COVID-19 pandemic and subsequent lockdowns, s...
01/22/2022

From 996 to 007: Challenges of Working from Home During the Epidemic in China

During the COVID-19 epidemic in China, millions of workers in tech compa...
07/13/2021

The Impact of Working From Home on the Success of Scrum Projects: A Multi-Method Study

The number of companies opting for remote working has been increasing ov...
08/25/2020

A Tale of Two Cities: Software Developers Working from Home During the COVID-19 Pandemic

The COVID-19 pandemic has shaken the world to its core and has provoked ...
08/26/2022

Half-Empty Offices in Flexible Work Arrangements: Why are Employees Not Returning?

Although the pandemic times of the world-wide forced working from home s...
04/04/2022

Optimal Workplace Occupancy Strategies during the COVID-19 Pandemic

During the COVID-19 pandemic, many organizations (e.g. businesses, compa...
02/14/2018

Do Programmers Work at Night or During the Weekend?

Abnormal working hours can reduce work health, general well-being, and p...

1. Introduction

Working from home (WFH) is a work arrangement in which employees do not need not work at a central place (e.g., office building, warehouse, or store). Working from home has various names, such as remote work, teleworking, or telecommuting. These terms are used differently and interchangeably from study to study (Shin et al., 2000; Spark, 2017; Hill et al., 2003). Nowadays, since WFH is facilitated by many tools such as virtual private network, could computing, and online meeting software, more and more companies allow their employees to work from home. A survey in 2018 from OWL labs show that 52% of employees work from home at least once a week and 56% of companies allow remote work111https://www.owllabs.com/state-of-remote-work/2018. WFH can offer some benefits to both companies and employees, for instance, when employees can work from home, they feel more trusted and are better able to balance work and life responsibilities, which can increase employee retention and make them happier and more productive.

WFH is also adopted by many IT companies, for instance, a recent mentioned that Twitter announced staff can continue working from home permanently 222https://www.bbc.com/news/technology-52628119. Developers can perform their daily tasks (e.g., writing code, debugging, build projects, and code review) as usual by remotely accessing resources of companies when working from home. WFH might have different impacts on productivity, which is a big concern of software developer organizations (Meyer et al., 2017a). Understanding the difference of developer productivity when working from home and the reasons behind it can help improve the management of companies and projects, increase the job satisfaction of developers, and make developers more productive.

The survey of OWL labs reported that employees who work remote at least once a month are 24% more likely to feel productive in their roles than those who do not or cannot work remotely. On the contrary, WFH might have a negative impact on productivity. For example, it would decrease the efficiency of developer communication, which plays an important role in software development (Wolf et al., 2009). There are many studies that investigate the impacts of WFH on productivity (Neufeld and Fang, 2004, 2005; Baker et al., 2007; Laihonen et al., 2012; Campbell, 2015). However, most of these studies use a qualitative approach (e.g., survey or interview) based on the feedbacks from general workers (not only developers). Additionally, the studied participants usually do not work from home for a long continuing time. In this study, we aim to investigate the impacts of developer productivity when working from home for a long time in a quantitative way.

Due to the outbreak of coronavirus disease 2019 (COVID-19)333https://www.who.int/health-topics/coronavirus, which is an infectious disease caused by a newly discovered coronavirus, a large number of IT companies ask their employees to work from home, which provides us an opportunity to investigate how their productivity is affected when working from home for a long continuing time. In this study, we collect the data from Baidu, Inc., China, which contains the development activities from 139 developers in 138 working days . There are a part of records in this dataset on which developers work from home due to COVID-19 pandemic. We compare developer productivity when working from home with working onsite in terms of multiple aggregated values such as mean, median of several metrics (e.g., the number of builds, commits, and code review.). We summarize our findings and contributions as follows:

  • To the best of our knowledge, we are the first to investigate the impacts of working from home on developer productivity based on developers’ daily activities. We find that WFH has both positive and negative impacts on developer productivity in terms of different metrics, such as the number of builds/commits/code reviews.

  • We investigate the impacts of WFH on the projects with different characteristics including program language and project type/age/size, and find that WFH has different impacts on different kinds of projects. For example, WFH has a negative impact on developer productivity for large projects. We also find that the productivity varies for different developers.

  • We conclude the reasons why developers have different productivity when working from home and provide implications based on our findings and the feedbacks from Baidu.

Paper Structure: The remainder of the paper is structured as follows. Section 2 describes the dataset and research questions in this study. Section 3 presents the results of analysis for the six research questions. Section 4 discusses implications and threats to validity. Section 5 briefly reviews related work. Section 6 concludes the paper and discusses future directions.

2. Case Study Setup

In this section, we first present the dataset from Baidu. Then, we describe the research questions and their corresponding motivations.

2.1. Dataset

Date Range #Working Days #Records
ONSITE_2019 2019/01/02 - 2019/03/10 68 1,565
ONSITE_2020 2019/12/23 - 2020/02/02 42 1,325
WFH 2020/02/03 - 2020/03/01 28 1,103
Total 138 3,993
Table 1. The overview of the dataset

We collected a dataset of developers’ daily activities from Baidu, Inc., which is the world’s largest Chinese language Internet search provider444https://www.baidu.com/. As the world’s largest Chinese language Internet search provider, Baidu responds to a huge amount of search queries from more than 100 countries and regions every day, serving as the most important way for netizens to access Chinese language information. With its mission to “make the complicated world simpler through technology”, Baidu promotes constant technological innovation, and is committed to being a world-leading technology company that understands its users and helps them grow.

, the largest knowledge and information centered Internet platform company in China, and a world-leading artificial intelligence (AI) company. The dataset we get from Baidu contains 139 developers’ daily activities from eight projects in 138 working days. Table 

1 presents the overview of the dataset. As shown in this table, the time of WFH is from 2020/02/03 to 2020/03/01 because Baidu asked all its employees to work from home after the outbreak of COVID-19 pandemic in China. There are another two date ranges during which developers work onsite in the dataset, i.e., from 2019/01/02 to 2019/03/10 (referred to as ONSITE_2019) and from 2019/12/23 to 2020/02/02 (referred to as ONSITE_2020). We also combine ONSITE_2019 and ONSITE_2020 into ONSITE_All, which is referred to as all developers’ daily activities when working onsite. This dataset has a total of 3,993 records. Among these records, there are 1,565, 1,325, and 1,103 records that are belong to ONSITE_2019, ONSITE_2020, and WFH, respectively.

Feature Description
date_partition The date of a record
username_e The encrypted user name of a developer
repo_name_e The encrypted repository name
commit_count The number of commits submitted by a developer
line_inserted The number of lines of code inserted by a developer
line_deleted The number of lines of code deleted by a developer
review_count The number of code review performed by a developer
job_status_build The status of build performed by a developer
build_count
The number of builds performed by a developer
A build refers to the process of continuous integration,
including compilation, test, deploy, etc.
job_status_release The status of release performed by a developer
release_count The number of releases performed by a developer
compile_count The number of compilations performed by a developer
Table 2. The metrics of each record of developers’ daily activities provided by Baidu.

Each record in the dataset has several metrics that represent the activities of a developer in one day. Table 2 presents the fields of a record. Each record has a date (data_partition) on which a developer’s activities are reported. The developer name and their project in the record are encrypted due to the security and privacy policy of Baidu. Each record has the following numeric metrics: commit_count, line_inserted, line_deleted, review_count, build_count, release_count, compile_count, which indicates the developer productivity in one day.

Due to the security policy of Baidu, the numeric metrics are standardized by the following formula:

where is the standardized value, is the real value of a metric in a record, is the mean of a metric in the dataset, and

is the standard deviation of a metric in the dataset. Thus, the standardized values do not affect the distribution of a metric and the findings in this study since the findings are based on the comparison between the values of metrics when developers work from home and work onsite. Moreover, the standardized values can be positive or negative. Additionally, there are two other fields, i.e.,

job_status_build and job_status_release, which represent the status of builds/releases performed by a developer. There are four possible values for the status of builds/releases, including success, failed, canceled, or NULL. Comparing to the other three status, a successful build or release means that a developer is more productive on that day.

Created Year Project Type Language #Developer
P1 2017 APP C++ 13
P2 2017 APP Java 13
P3 2018 SERVER Java 4
P4 2017 SERVER Java 25
P5 2018 SERVER C++ 53
P6 2018 SDK C++ 10
P7 2018 SERVER Java 7
P8 2017 SERVER C++ 14
Table 3. The project information.

Table 3 presents the information of the eight projects in the dataset. These projects are created in two different years, i.e., 2017 and 2018. There are three types of project, i.e., APP (application software), SERVER (server software such as web services, API libraries), and SDK (software development kits). Among these projects, four projects are written in C++ while the other four projects are written in Java. Additionally, we count the number of developers who have records of development activities in the dataset for each project. We also report these projects have different numbers of developers, for example, the project P5 has the most number of developers (i.e., 53) and P3 has the least number of developers (only 4 developers).

Figure 1. Violin plots for WFH and ONSITE_ALL in terms of the mean of numeric metrics grouped by day and build/release successful rate in one day.

2.2. Research Questions

In this section, we aim to answer the following six research questions:

RQ1. Are there any significant differences between the productivity of developers working from home and work onsite?

Motivation: In this RQ, we want to investigate whether working from home can affect developers’ productivity comparing to working onsite. Given the dataset from Baidu, we measure the overall productivity of all developers by combining their activities together, then compare the overall productivity when working from home and working onsite.

RQ2. Do different programming languages affect developer productivity when working from home?

Motivation: Previous studies have shown that programming languages have an important impact on developers’ activities, such as programming comprehension (Xia et al., 2017)

, and being a long-time contributors of open source projects 

(Von Krogh et al., 2003; Bao et al., 2019). Working from home might have different impacts on developers using different programming languages. For example, since C++ projects in Baidu are usually larger and require more computing resources than Java projects, developers often need build and debug these C++ projects on a powerful machine remotely. Meanwhile for most of Java projects, developers can write code and debug in their own computers at home. Thus, in this RQ, we want to investigate whether developers using different programming languages have different productivities when working from home.

RQ3. Do different project types affect developer productivity when working from home?

Motivation: As shown in Table 3, the eight projects in the dataset have three different types, i.e., APP, SERVER, and SDK. The projects with different project types could have different project management methods and styles of schedules, different development and communication tools, which might have a potential impact on developers’ productivity (Melo et al., 2011; Meyer et al., 2014). WFH changes the way of project management and development, which have different impacts on developer productivity for the projects with different types. For example, many APP projects develop mobile apps, which usually rely on some specific framework and have predefined code styles and specifications. While the software developed by SERVER projects are usually applied in much more complicated scenarios and depend on many different frameworks and programming languages. Thus, it might be more different for developers when working from home to build, test, and debug a SERVER project than a APP project. Thus, we want to investigate whether different project types have an impact on developer productivity when working from home.

RQ4. Do different project ages affect developer productivity who working from home?

Motivation: Different project ages might affect developers’ activities. For example, our previous study found that developers in older projects spend more time on program comprehension activities than those of projects in the beginning stage (Xia et al., 2017). Working from home might expand such effects caused by project age, for example, for a older project, developers need to read the source code and documents more frequently since such a project usually has more maintenance tasks; but they cannot access these resources and communicate with colleagues easily when working from home, which might lower their productivity. Thus, we want to investigate whether different project phases have an impact on the productivity of developers when working from home in this RQ.

RQ5. Do different project sizes affect developer productivity who work from home?

Motivation: Different project sizes (measured by the number of developers of a project in this study) might have an impact on developers’ productivity (Paiva et al., 2010). Due to different project sizes, projects might have different ways of project management and software development, which would be affected by working from home differently. For example, it might be more difficult to communicate with each other in a large project when working from home, which might decrease the productivity of the project.

RQ6. Do individual developers have different productivity when working from home?

Motivation: Comparing with working onsite, developers might have different productivity when working from home due to some personal factors, e.g., experience, personality, habit, and skills. For example, developers are much easier to be interrupted by some other stuff when working from home. Additionally, identifying developers who have different performance when working from home and the potential reasons behind it can help project leaders manage the projects. Thus, in this RQ we want to investigate whether individual developers have different productivity when working from home.

3. Case Study Results

In this section, we present the results of these six research questions one by one.

Metric mean median sum max min
p-value p-value p-value p-value p-value
build_count 0.037 0.219 1.27E-06 -0.560 0.029 0.233 1.93E-04 0.411 0.002 -0.245
commit_count 0.174 0.116 0.295 0.064 0.323 0.057 4.05E-04 0.434 0.313 -0.009
compile_count 0.008 -0.182 0.241 -0.018 8.58E-05 -0.461 0.241 -0.018 NA NA
lines_deleted 0.221 -0.095 0.129 -0.138 1.92E-05 -0.505 0.166 0.119 0.313 -0.009
lines_inserted 0.163 -0.121 0.031 -0.227 1.33E-05 -0.516 0.240 0.087 0.313 -0.009
release_count 0.019 0.253 NA NA 0.232 0.090 0.065 0.179 NA NA
review_count 0.131 0.138 0.224 0.084 0.131 0.138 0.005 0.317 NA NA
Table 4. P-value and Cliff’s delta () for WFH compared with ONSITE_ALL in terms of different aggregate values of each metric in one day.
Metric mean median sum max min
p-value p-value p-value p-value p-value
build_count 0.016 0.279 1.84E-07 -0.647 0.014 0.287 1.69E-04 0.467 0.003 -0.235
commit_count 0.335 0.056 0.351 -0.048 0.423 0.026 0.001 0.394 0.268 -0.015
compile_count 0.007 -0.191 0.268 -0.015 4.78E-06 -0.577 0.268 -0.015 NA NA
lines_deleted 0.209 -0.106 0.007 -0.317 2.48E-06 -0.596 0.103 0.165 0.268 -0.015
lines_inserted 0.067 -0.195 0.000 -0.449 1.09E-06 -0.618 0.332 0.057 0.268 -0.015
release_count 0.041 0.226 NA NA 0.377 0.041 0.153 0.129 NA NA
review_count 0.262 0.084 0.421 0.024 0.150 0.136 0.007 0.322 NA NA
Table 5. P-value and Cliff’s delta () for WFH compared with ONSITE_2019 in terms of different aggregate values of each metric in one day.
Metric mean median sum max min
p-value p-value p-value p-value p-value
build_count 0.196 0.122 0.001 -0.419 0.153 0.146 0.012 0.320 0.002 -0.262
commit_count 0.069 0.212 0.034 0.245 0.227 0.107 2.05E-04 0.500 NA NA
compile_count 0.012 -0.167 0.216 -0.024 0.027 -0.273 0.216 -0.024 NA NA
lines_deleted 0.297 -0.077 0.139 0.152 0.006 -0.359 0.378 0.045 NA NA
lines_inserted 0.498 0.000 0.175 0.132 0.007 -0.350 0.172 0.135 NA NA
release_count 0.019 0.296 NA NA 0.119 0.168 0.027 0.259 NA NA
review_count 0.056 0.226 0.075 0.182 0.161 0.141 0.015 0.308 NA NA
Table 6. P-value and Cliff’s delta () for WFH compared with ONSITE_2020 in terms of different aggregate values of each metric in one day.

3.1. RQ1. The Overall Developer Productivity when Working From Home

Methodology: To compare developer productivity when working from home with that when working onsite, we first group the records in the dataset by day, and compute several aggregate values including mean, median, sum, max, and min for each numeric metric, as shown in Table 3. Thus, we can know a more accurate distribution of developers’ productivity in terms of each metric. For example, although mean of the submitted commits (commit_count) in one day indicate the average workload of developers in one day, but sometimes a few experienced/core developers usually contribute more commits than junior/periphery developers, the mean of commits might still be very high. So, we calculate these aggregate values of each metric of all developers for each day. For the two other non-numeric feature job_status_build and job_status_release, we compute the success rate of build/release in one day, that is, the ratio of the number of times that a build/release is successful.

For each aggregation function of a metric, we have two groups, i.e., the days on which developers work from home and work onsite, respectively. The number of data points of a group is equal to the number of days of the corresponding group. Then, we apply the Wilcoxon rank-sum test (Wilcoxon, 1945) to investigate whether the difference is statistically significant in terms of one kind of aggregate values of each metric. We also compute Cliff’s delta (Cliff, 2014) to quantify the amount of difference between the two groups. Also, We repeat the above method using the different dataset of developers not working from home, i.e., ONSITE_All, ONSITE_2019 and ONSITE_2020. Cliff defines a delta of less than 0.147, between 0.147 to 0.33, between 0.33 and 0.474, and above 0.474 as negligible, small, medium, and large effect size, respectively.

Results: Since we only have standardized values for these metrics in the dataset, we use violin plots to show the distributions of these metrics when working from home and working onsite, as shown in Figure 1. From this figure, these metrics have less wider distributions when developers work from home than those when developers work onsite except the number of lines deleted everyday. For example, the range of the mean of build_count by day when working onsite is approximately from -0.1 to 0.3, while the range of the mean of build_count by day when working from home is approximately from -0.05 to 0.2. This indicates that developers might have a more stable productivity when working from home than that when they work onsite.

Table 45, and 6 present p-values and Cliff’s deltas () for WFH compared with ONSITE_ALL, ONSITE_2019 and ONSITE_2020 in terms of different aggregate values of each metric in one day, respectively. There are some cases whose values are “NA” in these tables, which is caused by the two compared lists are completely the same. We find that there are some cases in which developer productivity when working from home is significantly different from that when working onsite for ONSITE_ALL, ONSITE_2019 and ONSITE_2020. For example, the sum of compile_count, line_deleted and line_inserted when working from home are significantly less than those when working onsite, and all the effect sizes are not at the negligible level. This might indicates WFH has a negative effect on developer productivity in terms of the number of compilations, deleted lines, and inserted lines everyday.

Moreover, we find that WFH has different impacts on developer productivity in terms of different metrics. For example, in terms of build_count, its median values for developers when working from home is less than these when working onsite. On the contrary, the maximum value of build_count for developers when working from home is larger than these when working onsite. This might indicate that a small amount of developers perform more builds when working from home, while most of the developers perform less builds.

According to the feedbacks from Baidu, the workload of a project varies very quickly in different stages, and a project in the early stage has more workload than in the mature stage. Thus, we believe that there might be many other factors except WFH that can affect developer productivity in different times. In the remaining paper, we only use the records of 2020 (i.e., ONSITE_2020 and WFH) to avoid the influences of confound factors.

[width=8.5cm] Overall, WFH has different impacts on developer productivity in terms of different metrics.

Language Metric Positive Negative
C++ build_count sum, max min
commit_count mean, max
compile_count mean, sum
lines_deleted sum
lines_inserted sum
release_count mean
review_count mean, max
Java build_count mean, median, sum, min
commit_count sum
compile_count
lines_deleted
lines_inserted
release_count
review_count mean
Table 7. The cases in which values when working from home is significantly different for C++ and Java projects

3.2. RQ2. The Impact of Programming Language

Methodology: For both the records in WFH and ONSITE_2020, we divide them into two groups: those from the projects using C++ and Java. For each group, we use the same method in the RQ1 then investigate whether the difference between the productivity of developers working from home and working onsite is statistically significant in terms of the aggregate values of each metric.

Results: Table 7 presents the cases in which values when working from home are significantly different and the effect sizes are not negligible from these when working onsite for C++ and Java projects. The column ‘Positive’/‘Negative’ means that WFH has a positive/negative impact on developer productivity in terms of an aggregate value of a metric. “—” means there are no cases in which values when working from home are significantly different from these when working onsite for a metric.

We find that for C++ projects, there are both some positive and negative cases. For example, the mean of commit_count, release_count and review_count are belong to positive cases while the sum of compile_count, lines_deleted and lines_inserted are belong to negative cases. On the other hand, all the cases of Java projects are belong to negative cases. This indicates that WFH has more negative impacts on Java projects than C++ projects.

[width=8.5cm] WFH has both positive and negative impacts on developer productivity for C++ projects in terms of different metrics, but has a negative impact on developer productivity for Java projects.

Type Metric Positive Negative
APP build_count max mean, min
commit_count mean, median, sum, max
compile_count mean
lines_deleted sum
lines_inserted
release_count mean
review_count mean, median, sum, max
SDK build_count mean, max min
commit_count min
compile_count
lines_deleted min
lines_inserted min
release_count
review_count min
SERVER build_count median, min
commit_count
compile_count mean, sum
lines_deleted sum
lines_inserted sum
release_count mean
review_count
Table 8. The cases in which values when working from home is significantly different for APP, SDK and Server projects

3.3. RQ3. The Impact of Project Type

Methodology: We split the records in the dataset into three parts based on the project type, i.e., APP, SERVER, and SDK. For each part, we use the same approach as RQ1 and RQ2 to investigate the difference between the productivity of developers when working from home and working on site for projects with different types.

Results: Table 8 presents the cases in which values when working from home are significantly different from those when working onsite and and the effect sizes is not negligible for APP, SERVER, and SDK projects. For APP projects, there are some positive and negative cases, which might indicates that WFH has both positive and negative impacts on developer productivity. For example, in terms of mean, median, sum and max of commit_count and review_count, their values for developers when working from home are significantly larger than these when working onsite; on the contrary, in terms of mean of build_count and compile_count, their values for developers when working from home are significantly less than these when working onsite.

For SDK and SERVER projects, most of cases are negative, which indicates that WFH has a negative impact on developer productivity. This might be because SDK and SERVER projects usually has more components than APP projects, which requires more collaboration and communication with the other developers. According to the feedbacks from Baidu (see Section 4.1), WFH has a negative impact on collaboration and communication, which decreases developer productivity.

[width=8.5cm] WFH has both positive and negative impacts on developer productivity for APP projects, and decreases developer productivity for SDK and SERVER projects.

Year Metric Positive Negative
2017 build_count max median, min
commit_count mean, sum, max
compile_count mean
lines_deleted median, max
lines_inserted median, max sum
release_count mean
review_count mean, median, sum, max
2018 build_count median, min
commit_count
compile_count mean, sum
lines_deleted mean, sum
lines_inserted sum
release_count
review_count sum
Table 9. The cases in which values when working from home is significantly different for projects created on 2017 and 2018

3.4. RQ4. The Impact of Project Age

Methodology: As the projects in the dataset are created on two different years, i.e., 2017 and 2018, we split the records into two groups based on the year in which a project is created. Then, we investigate the difference between developer productivity when working from home and working onsite for the projects with different ages.

Results: Table 9 presents the cases in which values when working from home are significantly different from those when working onsite and the effect sizes are not negligible for projects created on 2017 and 2018. As shown in this table, for projects created on 2017, there are more positive cases than negative cases. For example, the mean of commit_count, release_count, and review_count are belong to positive cases while only the mean of compile_count is belong to negative cases. On the other hand, all the cases of projects created on 2018 are belong to negative cases. According to the feedbacks from Baidu, comparing to projects created on 2017, projects created on 2018 are newer and might has more tasks and schedules. It is not easy for a project to complete some kinds of tasks or schedules when working at home, for example, recruiting a new developer usually takes more time when working at home, but a newer project usually needs more new developers than these older projects.

[width=8.5cm] WFH has a positive effect on developer productivity for projects created on 2017 but a negative impact on developer productivity for projects created on 2018.

Size Metric Positive Negative
Small build_count max median, min
commit_count mean, sum, max
compile_count mean
lines_deleted median
lines_inserted median
release_count
review_count mean, median, sum, max
Large build_count sum median, min
commit_count
compile_count mean, sum
lines_deleted sum
lines_inserted sum
release_count mean, max
review_count
Table 10. The cases in which values when working from home is significantly different for small and large projects
Project Developer build_count commit_count compile_count lines_deleted lines_inserted release_count review_count
P1 D1 0.567 (large) 0.634 (large) 0.324 (small) 0.295 (small) 0.452 (medium)
P1 D2 -0.316 (small) -0.176 (small)
P1 D3 0.445 (medium)
P1 D4 -0.395 (medium) -0.358 (medium) 0.272 (small)
P1 D5 -0.083 (negligible) -0.335 (medium)
P1 D8 -0.196 (small) -0.197 (small) 0.148 (small)
P2 D1 -0.667 (large) -0.667 (large) 0.600 (large)
P5 D5 -0.427 (medium) -0.296 (small) -0.290 (small)
P5 D6 -0.293 (small)
P5 D7 0.585 (large) 0.530 (large) 0.367 (medium) 0.537 (large)
P5 D11 -0.368 (medium) -0.405 (medium) -0.402 (medium) -0.299 (small)
P5 D13 0.288 (small)
P5 D14 0.214 (small)
P5 D17 1.000 (large) 0.636 (large) 0.591 (large) -0.773 (large)
P5 D20 -0.167 (small) -0.451 (medium)
P5 D23 0.280 (small) 0.372 (medium)
P5 D24 0.351 (medium) 0.385 (medium) -0.368 (medium)
P5 D29 -0.688 (large)
P6 D2 -0.342 (medium)
P7 D2 -0.312 (small) -0.413 (medium) -0.264 (small)
P8 D6 0.833 (large)
  • * denotes p-value; ** denotes p-value; *** denotes p-value;.

Table 11. Developers whose productivity when working from home is significantly different from those when working onsite in terms of different features
Metric #Positive #Negative
build_count 4 2
commit_count 3 7
compile_count 0 2
line_deleted 3 6
line_inserted 3 6
release_count 1 1
review_count 7 4
Table 12. The number of developers whose productivity when working from home is significantly larger and smaller than that when working onsite in terms of each metric.

3.5. RQ5. The Impact of Project Size

Methodology: As shown in Table 3, these eight projects have different numbers of developers. We regard the project P4 and P5 whose number of developers are larger than 20 as large projects, and the other 6 projects as small projects. Then, we split the records into two groups based on project size. Then, we investigate the difference between developers’ productivity when working from home and working onsite for the projects with different sizes.

Results: Table 10 presents the cases in which values when working from home are significantly different from those when working onsite and the effect sizes are not negligible for small and large projects. We find that there are more positive cases than negative cases for small projects and most of cases in large projects are belong to negative cases except for the sum of build_count. The reason might be that it is more difficult for large project to adjust their structure and schedule after working from home and it is more difficult for large projects to collaborate and communicate with others when working from home.

[width=8.5cm] WFH has a larger impacts on developer productivity for large projects than small projects.

3.6. RQ6. The Productivity of Individual Developers When Working From Home

Methodology: For each developer in our dataset, we have two kinds of records, i.e, those when working from home and when working onsite. Then, we investigate whether their productivity when working from home is significantly different from that when working onsite in terms of each metric. We also compute Cliff’s delta (Cliff, 2014) to quantify the amount of difference. For the records when working onsite, we only use ONSITE_2020 according to the findings in RQ1.

Results: Table 11 presents the individual developers if there exists at least one case in which the values of an aggregation of a metric when working from home are significantly different from those when working onsite. The second column in this table is the index of a developer in their project. Out of 139 developers in the whole dataset, the productivity of 21 developers when working from home is different from that when working onsite. This indicate that the productivity of the majority of developers (84.8%) when working from home is similar to that when working onsite.

Table 12 presents the number of developers for who WFH has a positive or negative impact on their productivity in terms of a metric. As shown in this table, only in terms of compile_count, there is no developer whose productivity when working from home is significantly larger that that when working onsite. We also notice that the productivity of several developers when working from home is significantly larger than that when working onsite in terms of all metrics, e.g., D1 of the project P1 and D7 of the project P5. For these developers, the company should encourage allow them to work from home for more time since remote work can improve their productivity. On the contrary, some developers are less productive when working from home, e.g., D2 of the project P1 and D5 of the project P5. For these developers, remote work is not encouraged since their productivity decreases when working from home.

[width=8.5cm] The productivity of the majority of developers when working from home is similar to that when working onsite. For a small portion of developers, WFH has different impacts on their productivity.

4. Discussion

In this section, we first present the feedbacks from Baidu, then provide implications of our findings. At the end of this section, we discuss some threats to validity.

4.1. Feedbacks from Baidu

Based on our findings, we asked the developers in the studied projects to share their opinions on the impacts of WFH on their productivity. Many of them agreed that WFH can have both positive and negative impacts on developer productivity. Some also agree that there is no difference of productivity when working from home. The followings are some of the feedbacks we collected:

WFH improves developer productivity.

  • It’s the first time for some developers to work from home, so they feel very excited and have a lot of energy to do their work.

  • Developers can focus on their own work and not be disturbed by colleague.

  • After working from home, the company asked developers to write daily reports instead of weekly reports. Daily reports can help developers recall their daily work and push them to work harder in the second day if their tasks aren’t completed.

  • WFH decreases the cost of transportation and save a lot of time for developers.

  • WFH might increase developers’ working time because there is no switch between workplace and home and developers can work very early in the morning or very late in the evening.

  • WFH gives developers better work-life balance so that developers can work in better condition.

WFH decreases developer productivity.

  • There are many other stuff (e.g., looking after children or pets, cooking by themselves, etc) to interrupt developers’ work and take a lot of their time.

  • Some developers without self-discipline cannot focus on work when working from home. Unlikely onsite, they might be too relax at home since there is no colleague around them.

  • Although video conferencing tools or telephone are now very convenient for communication, the efficiency of collaboration still decreases due to working from home.

There is no difference of developer productivity when working from home.

  • There are no barriers for many developers to complete their daily tasks (e.g., writing code, build projects, code review, etc.) when working from home.

  • There is no difference of project schedule between working from home and working onsite since developers can know the schedule using an online project schedule tool.

  • Current video conferencing tools are very powerful, for example, they usually support screen sharing. So, there is very slight difference between meeting at a meeting room of the company and online.

4.2. Implications

WFH has different impacts on overall developer productivity. Many previous studies have shown that WFH has a positive effect on the productivity of workers (Neufeld and Fang, 2004, 2005; Baker et al., 2007; Laihonen et al., 2012; Coenen and Kok, 2014; Campbell, 2015; Kazekami, 2020). Some other studies also showed that WFH might have a negative impact on employee productivity. For example, Kazekami found that the long working time of WFH would decrease teleworker productivity (Kazekami, 2020). In our study, we use a quantitative analysis to show that WFH has different impacts on developer productivity. From the feedbacks of Baidu, the difference of developer productivity might be caused by many reasons. As shown in RQ6, since the productivity of developers are not affected by working from home, we recommend that WFH can be considered as a choice of work arrangement for employees because WFH offers many other benefits except productivity, such as saving costs for company and the flexibility of working time for developers.

A project needs prepare for WFH according to its own characteristics. In this study, we find that developer productivity might be associated with the characteristics of a project including programming language and project type/age/size. For example, WFH might have a positive effect on developer productivity for small project but do not affect developer productivity for large projects (see RQ2 in Section 3.5). Thus, we believe that adopting the WFH policy for a project should be based on its own characteristics, e.g., programming languages, and project size. When starting the WFH policy, a project need to prepare some resources to reduce the risks of decreasing developer productivity. For example, a large project should consider the communication cost of WFH and prepare the relevant tools to facilitate the communication of the team.

Using different strategies of WFH for individual developers. We find that the productivity of the majority of developers in this study does not change when working from home. Still there exist some developers whose productivity when working from home is different from these when working onsite. Thus, some approach based on some development metrics can be used to identify whether the productivity of a developer increases or decreases when working from home. Once the productivity of a developer decreases, the project manager needs to identify the reasons behind it. If the developer is not suitable to WFH, they should be asked to go to the company for work. On the other hand, if the productivity of a developer increases, the project team should allow them to continue working from home.

For researchers, to improve individual developers’ productivity when working from home, more empirical studies are required to investigate more factors that affect their productivity, e.g., personality, moods, and their working environment at home. Additionally, some machine learning models based on developers’ daily activities can be built to predict whether their productivity will change when working from home.

4.3. Threats to Validity

Threats to internal validity. First, there might exist errors in our code and experiment setting. We write a python script to process and analyze the dataset provided by Baidu. We double check our code, however, there may exist some errors that we do not notice. The second internal validity is that we use some metrics of software development (e.g., the number of builds and commits) in the dataset to measure developer productivity. We think these metrics can potentially indicate the productivity of developers and we also use multiple aggregate values (such as mean and median) of these metrics by day to measure the productivity. Third, there might be many other factors (e.g., the workload in different time) that affect developer productivity. It is difficult to exclude all other factors in the study. To minimize this threat, we only use the records of developers’ activities of WFH and ONSITE in the same year (i.e., 2020) for comparison. Finally, the metrics in the dataset are standardized due to the security policy and privacy of Baidu. But wo focus on the difference between the productivity of developers when working from home and working onsite. Thus, we believe that the comparison results using standardized values does not affect the findings in the study.

Threats to external validity relate to the generalizability of our findings. In this study, the dataset we used is from Baidu. Thus, it is unclear whether the same results still hold for other developers from other companies. However, since Baidu is one of the largest IT company in China, we believe that our findings in this study have typical and common characteristics to some extent. Additionally, we analyze 138 working days of software development activities from 139 developers. These developers are from eight different projects with different characteristics such as programming languages and project types. Another threats to external validity relate to the generalizability of the metrics used to measure the productivity of developers. These metrics used in this study are very general and often used in software development (Bao et al., 2019; Yan et al., 2019). In the future, to reduce these threats, we plan to investigate more developers from different companies and consider more metrics.

5. Related Work

In this section, we discuss related work in fields of working from home and developer productivity.

5.1. Working from Home

There are many studies in the literature that investigate the benefits and drawbacks of WFH (Nilles, 1994; Bailey and Kurland, 1999; Shin et al., 2000; Pérez et al., 2002; Felstead and Henseke, 2017; Aguilera et al., 2016). According to the literature, the main benefits of WFH for companies include saving costs of buildings and increasing productivity and job satisfaction of employees. For employees, WFH gives them more flexible working time and provides better work-life balance. WFH can also offer benefits to some special kinds of persons, such as the disabled (Spark, 2017) and transgender developers (Ford et al., 2019). The main disadvantages of WFH are the access to technology and the integration of telework with the company’s strategy and organisational structure, as well as the teleworkers motivation and control (Pérez et al., 2002). Felstead and Henseke also reported that telework makes employees it difficulty to insulate the world of work from other aspects of life when both worlds collide and overlap (Felstead and Henseke, 2017).

Several studies have investigated the impact of WFH on productivity (Neufeld and Fang, 2004, 2005; Baker et al., 2007; Laihonen et al., 2012; Coenen and Kok, 2014; Campbell, 2015; Kazekami, 2020). Among these studies, many of them reported that WFH has a positive impact on the productivity of teleworkers. For example, Coenen and Kok found that telework has a positive effect on the performance of new product development through enabling knowledge sharing, cross-functional cooperation and inter-organizational involvement (Campbell, 2015). On the contrary, WFH might have a negative impact on productivity. For instance, Kazekami found that appropriate telework hours increase labor productivity but when telework hours are too long, telework decreases labor productivity (Kazekami, 2020). WFH might decrease the efficiency of developer communication, which plays an important role in software development (Wolf et al., 2009).

However, most of previous studies for WFH are based on a qualitative analysis using survey or interview. The participants in these studies are general workers not only for developers, and they do not have the experience of working from home for a long continuing time. In this study, we perform a quantitative analysis based on a lot of activity data of developers when working from home during the time of COVID-19 pandemic. We focus on the impact of WFH on developer productivity and the potential factors affecting developer productivity.

5.2. Developer Productivity

A lot of studies use developers’ daily activities to investigate their productivity. For example, Perry et al. found that many developers spend a lot of time on the communication with colleague (Perry et al., 1994). Additionally, many studies reported that developers work is fragmented and frequently interrupted, which has an important impact on their productivity (Bailey et al., 2001; Chong and Siino, 2006; Czerwinski et al., 2004; Horvitz, 2001; Parnin and DeLine, 2010; van Solingen et al., 1998). For example, Sanchez et al. found that work fragmentation is correlated to lower observed productivity and longer activity switches seem to strengthen the effect (Sanchez et al., 2015).

Developer productivity is often measured by software artifacts produced by developers in a certain time, e.g., submitted lines of code (LOC) (Devanbu et al., 1996; Nguyen et al., 2011), function points (Albrecht, 1979), completed tasks (Minelli et al., 2015), and time to implement a requirement (Cataldo et al., 2008). Meyer et al. proposed a list of metrics to measure developer productivity and several ways to improve a developer’s productivity through tool support (Meyer et al., 2014). Some studies also investigate the factors affecting developer productivity, e.g., characteristics of workplace (e.g., privacy, noise) (DeMarco and Lister, 1985), programming languages and development tools (Boehm, 1987), project switching (Vasilescu et al., 2016), and developers’ mood (Khan et al., 2011). Additionally, personal factors might have an impact on productivity, for example, some developers feel more productive when communicating with others but some don’t like to be interrupted when working (Meyer et al., 2017b).

In this study, we focus on the difference of developer productivity between working from home and working onsite. We measure developer productivity by several metrics based on developers’ daily activities, which have been used in previous studies. We also investigate the factors affecting developer productivity when working from home, such as programming language, project type, project size, etc.

6. Conclusion

In this paper, we investigate the productivity of developers when working from home for a long time due to COVID-19 pandemic. We use a quantitative analysis based on a dataset of developers’ daily activities from Baidu. To compare developer productivity when working from home with that when working onsite, we use several metrics of software development in the dataset, such as the number of builds, commits, and inserted/deleted lines. We find that WFH has different impacts on developer productivity in terms of different metrics. Also, we investigate some factors affecting developers productivity when working from home including programming language and project type/age/size. Additionally, we find that a small amount of individual developers have different productivity when working from home. In the future, we plan extend our study by using more data from more developers and companies. We also want to build machine learning models to predict developers productivity based on developers’ daily activities.

References

  • A. Aguilera, V. Lethiais, A. Rallet, and L. Proulhac (2016) Home-based telework in france: characteristics, barriers and perspectives. Transportation Research Part A: Policy and Practice 92, pp. 1–11. Cited by: §5.1.
  • A. J. Albrecht (1979) Measuring application development productivity. In Proc. Joint Share, Guide, and IBM Application Development Symposium, 1979, Cited by: §5.2.
  • B. P. Bailey, J. A. Konstan, and J. V. Carlis (2001) The effects of interruptions on task performance, annoyance, and anxiety in the user interface.. In Interact, Vol. 1, pp. 593–601. Cited by: §5.2.
  • N. Bailey and N. B. Kurland (1999) The advantages and challenges of working here, there, anywhere, and anytime. Organizational dynamics 28 (2), pp. 53–68. Cited by: §5.1.
  • E. Baker, G. C. Avery, and J. Crawford (2007) Satisfaction and perceived productivity when professionals work from home. Research & Practice in Human Resource Management. Cited by: §1, §4.2, §5.1.
  • L. Bao, X. Xia, D. Lo, and G. C. Murphy (2019) A large scale study of long-time contributor prediction for github projects. IEEE Transactions on Software Engineering. Cited by: §2.2, §4.3.
  • B. W. Boehm (1987) Improving software productivity. Computer (9), pp. 43–57. Cited by: §5.2.
  • K. M. Campbell (2015) Flexible work schedules, virtual work programs, and employee productivity. Cited by: §1, §4.2, §5.1.
  • M. Cataldo, J. D. Herbsleb, and K. M. Carley (2008) Socio-technical congruence: a framework for assessing the impact of technical and work dependencies on software development productivity. In Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, pp. 2–11. Cited by: §5.2.
  • J. Chong and R. Siino (2006) Interruptions on software teams: a comparison of paired and solo programmers. In Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, pp. 29–38. Cited by: §5.2.
  • N. Cliff (2014) Ordinal methods for behavioral data analysis. Psychology Press. Cited by: §3.1, §3.6.
  • M. Coenen and R. A. Kok (2014) Workplace flexibility and new product development performance: the role of telework and flexible work schedules. European Management Journal 32 (4), pp. 564–576. Cited by: §4.2, §5.1.
  • M. Czerwinski, E. Horvitz, and S. Wilhite (2004) A diary study of task switching and interruptions. In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 175–182. Cited by: §5.2.
  • T. DeMarco and T. Lister (1985) Programmer performance and the effects of the workplace. In Proceedings of the 8th international conference on Software engineering, pp. 268–272. Cited by: §5.2.
  • P. Devanbu, S. Karstu, W. Melo, and W. Thomas (1996) Analytical and empirical evaluation of software reuse metrics. In Proceedings of IEEE 18th International Conference on Software Engineering, pp. 189–199. Cited by: §5.2.
  • A. Felstead and G. Henseke (2017) Assessing the growth of remote working and its consequences for effort, well-being and work-life balance. New Technology, Work and Employment 32 (3), pp. 195–212. Cited by: §5.1.
  • D. Ford, R. Milewicz, and A. Serebrenik (2019) How remote work can foster a more inclusive environment for transgender developers. In 2019 IEEE/ACM 2nd International Workshop on Gender Equality in Software Engineering (GE), pp. 9–12. Cited by: §5.1.
  • E. J. Hill, M. Ferris, and V. Märtinson (2003) Does it matter where you work? a comparison of how three work venues (traditional office, virtual office, and home office) influence aspects of work and personal/family life. Journal of Vocational Behavior 63 (2), pp. 220–241. Cited by: §1.
  • E. C. M. C. E. Horvitz (2001) Notification, disruption, and memory: effects of messaging interruptions on memory and performance. In Human-Computer Interaction: INTERACT, Vol. 1, pp. 263. Cited by: §5.2.
  • S. Kazekami (2020) Mechanisms to improve labor productivity by performing telework. Telecommunications Policy 44 (2), pp. 101868. Cited by: §4.2, §5.1.
  • I. A. Khan, W. Brinkman, and R. M. Hierons (2011) Do moods affect programmers’ debug performance?. Cognition, Technology & Work 13 (4), pp. 245–258. Cited by: §5.2.
  • H. Laihonen, A. Jääskeläinen, A. Lönnqvist, and J. Ruostela (2012) Measuring the productivity impacts of new ways of working. Journal of facilities management. Cited by: §1, §4.2, §5.1.
  • C. Melo, D. S. Cruzes, F. Kon, and R. Conradi (2011) Agile team perceptions of productivity factors. In 2011 Agile Conference, pp. 57–66. Cited by: §2.2.
  • A. N. Meyer, L. E. Barton, G. C. Murphy, T. Zimmermann, and T. Fritz (2017a) The work life of developers: activities, switches and perceived productivity. IEEE Transactions on Software Engineering 43 (12), pp. 1178–1193. Cited by: §1.
  • A. N. Meyer, T. Fritz, G. C. Murphy, and T. Zimmermann (2014) Software developers’ perceptions of productivity. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 19–29. Cited by: §2.2, §5.2.
  • A. N. Meyer, T. Zimmermann, and T. Fritz (2017b) Characterizing software developers by perceptions of productivity. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 105–110. Cited by: §5.2.
  • R. Minelli, A. Mocci, and M. Lanza (2015) I know what you did last summer-an investigation of how developers spend their time. In 2015 IEEE 23rd International Conference on Program Comprehension, pp. 25–35. Cited by: §5.2.
  • D. J. Neufeld and Y. Fang (2004) Predicting telecommuter productivity. In 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the, pp. 10–pp. Cited by: §1, §4.2, §5.1.
  • D. J. Neufeld and Y. Fang (2005) Individual, social and situational determinants of telecommuter productivity. Information & Management 42 (7), pp. 1037–1049. Cited by: §1, §4.2, §5.1.
  • V. Nguyen, L. Huang, and B. Boehm (2011) An analysis of trends in productivity and cost drivers over years. In Proceedings of the 7th International Conference on Predictive Models in Software Engineering, pp. 1–10. Cited by: §5.2.
  • J. M. Nilles (1994) Making telecommuting happen: a guide for telemanagers and telecommuters. Cited by: §5.1.
  • E. Paiva, D. Barbosa, R. Lima, and A. Albuquerque (2010) Factors that influence the productivity of software developers in a developer view. In Innovations in computing sciences and software engineering, pp. 99–104. Cited by: §2.2.
  • C. Parnin and R. DeLine (2010) Evaluating cues for resuming interrupted programming tasks. In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 93–102. Cited by: §5.2.
  • M. P. Pérez, A. M. Sánchez, and M. de Luis Carnicer (2002) Benefits and barriers of telework: perception differences of human resources managers according to company’s operations strategy. Technovation 22 (12), pp. 775–783. Cited by: §5.1.
  • D. E. Perry, N. A. Staudenmayer, and L. G. Votta (1994) People, organizations, and process improvement. IEEE Software 11 (4), pp. 36–45. Cited by: §5.2.
  • H. Sanchez, R. Robbes, and V. M. Gonzalez (2015) An empirical study of work fragmentation in software evolution tasks. In 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp. 251–260. Cited by: §5.2.
  • B. Shin, O. A. El Sawy, O. R. L. Sheng, and K. Higa (2000) Telework: existing research and future directions. Journal of organizational computing and electronic commerce 10 (2), pp. 85–101. Cited by: §1, §5.1.
  • R. Spark (2017) Accessibility to work from home for the disabled: the need for a shift in management style. In Proceedings of the 14th Web for All Conference on The Future of Accessible Work, pp. 1–4. Cited by: §1, §5.1.
  • R. van Solingen, E. Berghout, and F. van Latum (1998) Interrupts: just a minute never is. IEEE software 15 (5), pp. 97–103. Cited by: §5.2.
  • B. Vasilescu, K. Blincoe, Q. Xuan, C. Casalnuovo, D. Damian, P. Devanbu, and V. Filkov (2016) The sky is not the limit: multitasking across github projects. In Proceedings of the 38th International Conference on Software Engineering, pp. 994–1005. Cited by: §5.2.
  • G. Von Krogh, S. Spaeth, and K. R. Lakhani (2003) Community, joining, and specialization in open source software innovation: a case study. Research policy 32 (7), pp. 1217–1241. Cited by: §2.2.
  • F. Wilcoxon (1945) Individual comparisons by ranking methods. Biometrics bulletin 1 (6), pp. 80–83. Cited by: §3.1.
  • T. Wolf, A. Schroter, D. Damian, and T. Nguyen (2009) Predicting build failures using social network analysis on developer communication. In 2009 IEEE 31st International Conference on Software Engineering, pp. 1–11. Cited by: §1, §5.1.
  • X. Xia, L. Bao, D. Lo, Z. Xing, A. E. Hassan, and S. Li (2017) Measuring program comprehension: a large-scale field study with professionals. IEEE Transactions on Software Engineering 44 (10), pp. 951–976. Cited by: §2.2, §2.2.
  • M. Yan, X. Xia, D. Lo, A. E. Hassan, and S. Li (2019) Characterizing and identifying reverted commits. Empirical Software Engineering 24 (4), pp. 2171–2208. Cited by: §4.3.