An Exploratory Characterization of Bugs in COVID-19 Software Projects

05/31/2020 ∙ by Akond Rahman, et al. ∙ Tennessee Tech University NC State University 0

Context: The dire consequences of the COVID-19 pandemic has influenced development of COVID-19 software i.e., software used for analysis and mitigation of COVID-19. Bugs in COVID-19 software can be consequential, as COVID-19 software projects can impact public health policy and user data privacy. Objective: The goal of this paper is to help practitioners and researchers improve the quality of COVID-19 software through an empirical study of open source software projects related to COVID-19. Methodology: We use 129 open source COVID-19 software projects hosted on GitHub to conduct our empirical study. Next, we apply qualitative analysis on 550 bug reports from the collected projects to identify bug categories. Findings: We identify 8 bug categories, which include data bugs i.e., bugs that occur during mining and storage of COVID-19 data. The identified bug categories appear for 7 categories of software projects including (i) projects that use statistical modeling to perform predictions related to COVID-19, and (ii) medical equipment software that are used to design and implement medical equipment, such as ventilators. Conclusion: Based on our findings, we advocate for robust statistical model construction through better synergies between data science practitioners and public health experts. Existence of security bugs in user tracking software necessitates development of tools that will detect data privacy violations and security weaknesses.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The novel Coronavirus disease (COVID-19) is a worldwide pandemic that spreads through droplets generated from coughs or sneezes and by touching contaminated surfaces (John Hopkins University, 2020). As of May 31 2020, COVID-19 has caused 370,247 deaths across the world (John Hopkins University, 2020). Apart from causing thousands of deaths and creating long term health repercussions for vulnerable populations, COVID-19 has severely impacted the economic sector. According to a recent study (Erin Duffin, 2020), due to COVID-19 gross domestic product (GDP) will decrease from 3.0% to 2.4% worldwide. As of May 28 2020, nearly 41 million citizens reported unemployment in USA alone (Mitchell Hartman, 2020). More than 3.9 billion people around the world were under some form of stay at home order due to COVID-19 (Alasdair Sandford, 2020).

Health care professionals are at the frontline of combating COVID-19. Practitioners from other domains, such as software engineering have also joined forces to analyze and mitigate the negative consequences of COVID-19. For example, statistical modeling was used to build a software that identifies pneumonia caused by COVID-19 from lung scan images (Tom Simonite, 2020). The software was used in 34 Chinese hospitals (Tom Simonite, 2020). In response to the food insecurity caused by COVID-19, practitioners have created an interactive visualization software that displays free meal sites across USA (Why Hunger, 2020). The creators of the software envision in building a social movement to eradicate hunger and address economic inequalities. As another example, Apple and Google have jointly announced of creating a software framework that will help practitioners build tools to trace COVID-19 infection status of mobile app users (Apple, 2020). The above-mentioned examples show COVID-19 software i.e., software used for analysis and mitigation of COVID-19, to have near-term and long-term effects on public health and society.

Despite the above-mentioned advancements, COVID-19 software projects are susceptible to bugs. Let us consider Figure 1 in this regard. Figure 1 provides a snapshot of a bug report related to statistical modeling (Begley, 2020a). We observe when implementing a statistical model the practitioners did not consider the correlation between ICU bed availability and death rate prediction. Furthermore, the number of intensive care unit (ICU) beds is incorrectly assumed to be 40,000 instead of 1,000.

We hypothesize systematic analysis can reveal bug categories including statistical modeling bugs similar to Figure 1. In prior work researchers (Garcia et al., 2020; Rahman et al., 2020; Linares-Vásquez et al., 2017; Catolino et al., 2019; Thung et al., 2012; Wan et al., 2017) have documented the importance of bug categorization. For example, for autonomous vehicle software Garcia et al. (Garcia et al., 2020) stated that categorization of bugs can help to construct bug detection and testing tools. Linraes-Vásquez et al. (Linares-Vásquez et al., 2017) stated categorizing vulnerabilities can help Android practitioners “in focusing their verification and validation activities”. According to Catolino et al. (Catolino et al., 2019), “understanding the bug type represents the first and most time-consuming step to perform in the process of bug triage”. Categorization of bugs in COVID-19 software can help practitioners and researchers to (i) understand the nature of COVID-19 software bugs, (ii) construct bug detection and repair tools, and (iii) measure COVID-19 software quality by using reported frequency of bug categories as a benchmark.

Figure 1: An example of a bug report related to statistical modeling in a software project called ‘neherlab/covid19_scenarios’.

In prior work, researchers have categorized bugs for IaC (Rahman et al., 2020), autonomous vehicle (Garcia et al., 2020)

, machine learning 

(Thung et al., 2012; Islam et al., 2019), and blockchain (Wan et al., 2017) software. However, COVID-19 software is different from previously studied software in the following aspects: (i) development context: unlike previously studied software projects, COVID-19 software is developed in response to a pandemic that infected 6.1 million individuals in five months (John Hopkins University, 2020), and (ii) public health: unlike previously studied software projects, COVID-19 software has direct implications on public health and relevant policy making for inhabitants in 188 countries.

In response to the pandemic, researchers have conducted studies related to modeling (Dehning et al., 2020; Yang and Wang, 2020; Tamm, 2020), biological science (Jin et al., 2020; Wang et al., 2020; De Clercq, 2006; Helms et al., 2020), social science (Van Bavel et al., 2020; Pulido et al., 2020; Evans et al., 2020; Will, 2020; Jarynowski et al., 2020), and policy making (Corey et al., 2020; Mello and Wang, 2020; Rourke et al., 2020; Kraemer et al., 2020). However, characterization of bugs in COVID-19 software remains an unexplored area.

The goal of this paper is to help practitioners and researchers improve the quality of COVID-19 software through an empirical study of open source software projects related to COVID-19.

We answer the following research questions:

  • RQ1: What categories of open source COVID-19 software projects exist?

  • RQ2: What categories of bugs exist in open source COVID-19 software projects? How frequently do the identified bug categories appear? What is the resolution time for the identified bug categories?

We conduct an empirical study with 129 open source COVID-19 software projects hosted on GitHub. First, we apply qualitative analysis (Saldana, 2015) on the README files of the collected open source software (OSS) projects to identify what categories of OSS projects exist related to COVID-19. Next, we apply qualitative analysis on 550 bug reports from the collected OSS projects to identify bug categories. We also quantify the frequency and resolution time of each bug category across the identified project categories. An overview of our paper is available in Figure 2.

Contributions: We list our contributions as following:

  • A categorization of bugs that appear in COVID-19 software projects;

  • A categorization of OSS projects related to COVID-19;

  • An empirical study that identifies what category of bugs appear for what category of COVID-19 software projects; and

  • A curated dataset which maps each identified bug report to the identified bug categories 111https://figshare.com/s/7044678e1d7e7feb1efb .

We organize rest of the paper as following: we provide background and discuss related work in Section 2. We provide the methodology and results for RQ1 and RQ2 respectively, in Sections 3 and 4. We discuss our results with a summary of our findings in Section 5. We provide the limitations of our paper in Section 6. Finally, we conclude the paper in Section 7.

Figure 2: An overview of our empirical study.

2 Background and Related Work

In this section, we first provide background on COVID-19 in Section 2.1 and briefly describe related research in Section 2.2.

2.1 Background

COVID-19 stands for ‘Coronavirus disease 2019’ (John Hopkins University, 2020). COVID-19 is an infectious disease that causes severe respiratory problems for infected human beings. The first case of an infected COVID-19 patients was reported in December 2019 in Wuhan, China (John Hopkins University, 2020). Since then the disease has spread rapidly. To date 6,108,525 cases have been reported across 188 countries, which resulted in 370,247 deaths (John Hopkins University, 2020).

COVID-19 is highly contagious (Sanche et al., 2020) and is caused by a strain of Coronavirus called severe acute respiratory syndrome Coronavirus 2 (SARS-CoV-2) (Gorbalenya, 2020). Disagreements exist amongst researchers on how the SARS-CoV-2 was transmitted to human beings. By using genome sequence similarity, a group of researchers speculate that the virus was transmitted to human beings from pangolins (Wong et al., 2020). Researchers have found the SARS-CoV-2 virus to have 92% resemblance with the Coronavirus found in pangolins (Zhang et al., 2020). Pangolins are nocturnal mammals found in Asia and Sub-Saharan Africa. Other researchers speculate the SARS-CoV-2 to be transmitted from horseshoe bats and civets. Researchers from McMaster University found the SARS-CoV-2 virus to share 99.8% of it’s genome with a civet Coronavirus (Cyranoski, 2020). From another phylogenetic analysis, researchers observed that a virus from horseshoe bats have a 96% resemblance to the SARS-CoV-2 virus (Zhou et al., 2020).

COVID-19 is a human transmissible disease. Transmission occurs via respiratory droplets from coughs and sneezes within a range of 6 feet. The virus can also be transmitted to a human being via contaminated surfaces as well as through droplets generated from cough and sneezing. What makes COVID-19 more susceptible for transmission is that infected human beings do not exhibit symptoms up until 214 days from infection (John Hopkins University, 2020). Symptoms of COVID-19 include cough, shortness of breath, fever, sore throat, and loss of taste or smell.

As of May 31, 2020, no vaccines exist that can cure COVID-19. To prevent the spreading, health experts have strongly recommended for maintaining personal hygiene e.g., frequent hand washing, social distancing, and mandatory lockdown, when necessary (John Hopkins University, 2020).

Researchers (Kissler et al., 2020) (Chen et al., 2020) have provided evidence that support the recurring nature of COVID-19. About the recurrence of COVID-19 Kissler et al. (Kissler et al., 2020) stated “a resurgence in contagion could be possible as late as 2024.”.

2.2 Related Work

Our paper is related with prior research that has focused on computing research related COVID-19 and characterization of bugs in OSS projects. We briefly describe each of these areas in the following subsections:

2.2.1 Computing Research Related to COVID-19

Our paper is related to recent computing research on COVID-19. Since the outbreak of COVID-19 in December 2019, researchers have conducted extensive research on understanding the spread of the disease through lens of computer science. We describe related work as following:

  • [leftmargin=*]

  • Visualization techniques: Dey et al. (Dey et al., 2020) constructed a visualization tool called Visual Exploratory Data Analysis (V‐EDA) to understand the spread of COVID-19.

  • Machine learning and statistical modeling: Naude (Naudé, 2020) identified research areas where machine learning can be applied to combat COVID-19: (i) early notifications, (ii) tracking and prediction, (iii) data dashboards, (iv) diagnosis and prognosis, (v) treatments and cures, and (vi) social control. Researchers (Yang and Wang, 2020; Tamm, 2020) in separate studies have used statistical models to understand the COVID-19 outbreak. Yang and Wang (Yang and Wang, 2020) proposed a mathematical model to understand the COVID-19 outbreak in Wuhan, where the virus originated from. They observed the diseases to be endemic and advocated for long-term disease prevention and intervention public health programs. Tamm (Tamm, 2020) constructed a mathematical model to understand the outbreak in Moscow using five scenarios based on control measures. Tamm (Tamm, 2020) reported that fatality due to COVID-19 would remain extremely high and healthcare providing institutions would be overloaded. Randhawa et al. (Randhawa et al., 2020)

    applied machine learning to classify pathogens, and observed evidence that supports the hypothesis that COVID-19 originated from a bat, as their model classified the COVID-19 virus as ‘Sarbecovirus’, a sub-category within ‘Betacoronavirus’. Rao and Vasquez 

    (Rao and Vazquez, 2020) used machine learning algorithms to identify potential COVID-19 patients using mobile web-based survey data. Currie et al. (Currie et al., 2020) identified challenges in modeling the COVID-19 pandemic, which included quarantine strategies and case isolation, social distancing measures and applications, lock down management, and testing for COVID-19. Pandey et al. (Pandey et al., 2020) used machine learning to construct a contract tracing mobile application for COVID-19 that uses the smartphone’s senor data and self-assessment of the smartphone user. Santosh (Santosh, 2020)

    advocated for the usage of active learning and multi-modal data to identify COVID-19 outbreaks as the pandemic is world-wide and differences in location can impact model performance used in forecasting.

  • Robotics: Yang et al. (Yang et al., 2020) advocated for the usage of robotics to help combating COVID-19 as robots can be deployed to deliver food and medicine, as well as for disinfecting infrastructure, such as medical centers and schools.

The above-mentioned discussion shows a lack of research that have characterized bugs in COVID-19 software projects. We address this research gap in our paper.

2.2.2 Research Related to Bug Characterization

Our paper is also related with prior research that have characterized bugs in OSS. Mockus et al. (Mockus et al., 2002) studied the contribution nature in OSS Apache and Mozilla projects. They (Mockus et al., 2002) observed contributors who submit bug reports are approximately 8.2 times higher in number than contributors who address bugs in bug reports. Ma et al. (Ma et al., 2017) investigated Python GitHub projects that are used in the scientific domain, and observed developers to use stack traces, as well as communicate with upstream developers, to identify root causes of bugs. Zhang et al. (Zhang et al., 2019) examined bug reports for mobile and desktop software hosted on GitHub, and identified differences on how the reports are constructed. Ray et al. (Ray et al., 2014) studied the correlations between bugs and the language the software is being developed, and reported a modest correlation using an empirical study of 729 GitHub projects. Categorization of domain-specific OSS bugs has also been investigated: Thung et al. (Thung et al., 2012), Garcia et al. (Garcia et al., 2020), Wan et al. (Wan et al., 2017), Islam et al. (Islam et al., 2019), and Rahman et al. (Rahman et al., 2020)

in separate research papers used OSS projects to classify bug categories respectively, for machine learning, autonomous vehicle, blockchain, deep learning, and IaC.

We take motivation from above-mentioned research and study COVID-19 software bugs in the following manner:

  • categories of bugs;

  • frequency of identified bug categories;

  • resolution time of identified bug categories; and

  • categories of software projects.

3 RQ1: COVID-19 Software Project Categories

In this section, we answer “RQ1: What categories of open source COVID-19 software projects exist?”. We define COVID-19 software projects as software projects used for analysis and mitigation of COVID-19. We hypothesize multiple categories of COVID-19 software projects to exist in the OSS domain. We validate our hypothesis by systematically categorizing COVID-19 software projects. Our categorization will provide insights on how the software development community has responded to the COVID-19 pandemic.

3.1 Methodology for RQ1

We answer RQ1 by completing the following steps:

3.1.1 Dataset Collection

We conduct our empirical analysis by collecting COVID-19 software projects hosted on GitHub. To collect these projects we use GitHub’s search utility (GitHub, 2020c), where we first identified projects tagged as ‘covid-19’. We use the search string ‘covid-19’, as it is a topic designated for COVID-19 by GitHub (GitHub, 2020a). Upon collection of the projects we apply a set of filtering criteria so that we can identify projects that contain sufficient data for analysis. We describe the filtering criteria below:

  • [leftmargin=*]

  • Criterion-1: The project must have at least 2 developers. Our assumption is that this criterion will filter projects used for personal purposes.

  • Criterion-2: The project has at least 5 open issues. We use this filtering criterion to identify projects that are actively maintained.

  • Criterion-3: The project must have at least two commits per month. Munaiah et al. (Munaiah et al., 2017) used the threshold of at least two commits per month to determine which projects have enough development activity for software organizations. We use this threshold to filter projects with short development activity.

  • Criterion-4: The README of the project is written in English. README projects related to COVID-19 can be non-English. We do not include non-English projects as raters who will perform categorization are not familiar with non-English languages, such as Spanish and Cantonese.

  • Criterion-5: The project is actually related to COVID-19. Practitioners can mislabel projects using the ‘topic’ feature of GitHub. For example, from manual inspection we observe the ‘RehanSaeed/Schema.NET’ 222https://github.com/RehanSaeed/Schema.NET project to be tagged as ‘covid-19’, even though it is used to convert blob objects into C# classes.

3.1.2 Qualitative Analysis of README files

We apply a qualitative analysis called open coding (Saldana, 2015) on the content of README files for each of the downloaded projects from Section 3.1.1. README files describe the content of the project and give GitHub users an overview of the software project (Prana et al., 2019). We hypothesize that by systematically analyzing the content of the README files we can derive what types of software projects are developed that are related to COVID-19.

In open coding a rater identified and synthesizes patterns within unstructured text (Saldana, 2015). We select open coding because we can obtain detailed information on the software project categories. We use a hypothetical example to demonstrate our process of open coding in Figure 3. First, we collect text from the README files for each of the collected projects from Section 3.1.1. Next, we extract text snippets that describe the purpose of the software project. For example, from the raw text ‘The COVID-19 Vulnerability Index (CV19 Index) is a predictive model that identifies people who are likely to have a heightened vulnerability to severe complications from COVID-19’ we extract the text snippet ‘a predictive model’, as the extracted text snippet describes the purpose of the software project. Next, from the text snippets ‘a predictive model’ and ‘

modelling estimated deaths

’ we generate an initial category called ‘Models to predict’. Two initial categories ‘Models to predict’ and ‘Models to understand’ are combined into one category ‘Statistical modeling’, as they both indicate the descriptions of the software projects to be related with statistical modeling.

Figure 3: A hypothetical example to demonstrate our process of open coding to categorize COVID-19 software projects.

The first and second author conduct the open coding process separately. The first and second author respectively an experience of 10 and 6 years in software engineering and has experience in conducting open coding upon software project artifacts, such as commit messages (Rahman et al., 2020) and Stack Overflow posts (Farhana et al., 2019). Upon completion of open coding process the first and second author identify agreements and disagreements. Disagreements are resolved upon discussion, Agreement rate is calculated using Cohen’s Kappa (Cohen, 1960). During the discussion phase both authors agreed present their justification, and recheck the category derivation based on the discussion and revisiting content. The mapping determined upon discussion is considered final. One project can map to multiple categories.

3.1.3 Closed Coding

We apply closed coding (Crabtree and Miller, 1999) to identify which project maps to the identified categories from Section 3.1.2. Closed coding is the qualitative analysis techniques where a rater maps an artifact to a pre-defined category by inspecting the artifact (Crabtree and Miller, 1999). The first and second author separately conduct closed coding on the collected README files. After completing the closed coding process the first and second authors identify agreements and disagreements. Agreement rate is recorded using Cohen’s Kappa (Cohen, 1960). Disagreements are resolved using discussion. During the discussion phase both authors present their justification for disagreements. Next, based on the discussion the authors recheck the labeling based on the justification and content analysis. The categorization determined upon discussion is considered final.

3.1.4 Rater Verification

The derived categories are susceptible to the bias of the first and second author. We mitigate the limitation by allocating an additional rater who applied closed coding for a subset of the README files. The additional rater who is not an author of the paper, is a fourth year PhD student in the Department of Computer Science at Tennessee Technological University. The rater has a professional experience of 2 years in software engineering and has conduced qualitative analysis on software artifacts, such as bug reports. We randomly allocate a set of 100 README files mined from 100 projects to the rater. The rater applies closed coding on the content of the README files, to identify the mapping between each project and identified categories. Upon completion of closed coding we calculate Cohen’s Kappa (Cohen, 1960) between the rater and the first author, as well as with the second author, separately.

3.2 Answer to RQ1

We answer RQ1 by first providing summary statistics of our dataset in Section 3.2.1. Next, we report categories of the projects in Section 3.2.2.

3.2.1 Summary of Dataset

Altogether we download 129 projects for analysis. Using the search feature we identify 3,276 public projects upon which we apply our filtering criterion. A complete breakdown of our filtering criterion is available in Table 1. Summary statistics of the projects is available in Table 2. ‘Languages’ in Table 2 correspond to the count of main programming languages of the collected projects as determined by GitHub’s linguist tool (GitHub, 2020b). Example languages include JavaScript, Python and R.

A temporal evolution of the 129 COVID-19 software projects based on creation date is available in Figure 4. We observe sharp increase in project creation after Feb 29, 2020.

Criteria GitHub
Initial 3,276
Criterion-1 (Devs ) 1,287
Criterion-2 (Open issues ) 169
Criterion-3 (Commits/month ) 154
Criterion-4 (README is English) 131
Criterion-5 (Actually COVID-19) 129
Final 129
Table 1: Filtering of COVID-19 projects used in paper.
Attributes Total
Commits 38,152
Developers 2,243
Duration 12/2019-03/2020
Files 24,839
Issues 4,405
Languages 18
Releases 286
Projects 129
Table 2: Attributes of studies COVID-19 projects.
Figure 4: Temporal evolution of COVID-19 software projects based on their creation date. We observe sharp increase in project creation after Feb 29, 2020.

3.2.2 Categorization of COVID-19 Software Projects

We identify 7 categories of COVID-19 software projects. We describe each of the categories below in an alphabetic order:

I: Aggregation:: This category includes software projects that curate data related to COVID-19 and present collected COVID-19 data in an aggregated format using visualizations. The purpose of these projects is to help users understand the spread of the COVID-19 disease over time and location. Software projects that belong to this category can be country specific as done in ‘juanmnl/covid19-monitor’ (juanmnl, 2020) and ‘dsfsi/covid19za’ (dsfsi, 2020b) respectively, for Ecuador and South Africa. Aggregation of COVID-19 data can also be at a global level, for example, ‘boogheta/coronavirus-countries’ (boogheta, 2020) is a software that aggregates COVID-19 data across the world and allows software users to compare the reported cases on a country-by-country basis.

II: Education:: This category includes projects that provide utilities on educating people about COVID-19. Lack of knowledge related to infections and symptoms can contribute to rapid spreading of COVID-19. The purpose of these projects is to build software, where users can ask questions and obtain answers. We observe two categories of software: first, question and answer websites similar to Stack Overflow 333https://stackoverflow.com/, such as ‘nthopinion/covid19’ (nthopinion, 2020), where users can ask questions about COVID-19, and other users answer such questions. Second, we observe bot-specific software, such as ‘deepset-ai/COVID-QA’ (deepset ai, 2020) that provides answers for questions related to COVID-19 automatically.

III: Medical equipment:: This category includes projects to curate and maintain source code for the design and implementation of medical equipment used to treat COVID-19. The purpose of these projects is to create designs of COVID-19 related medical equipment, such as ventilators at scale, so that the growing need of medical equipment in hospitals is satisfied. One example of such repository is ‘makers-for-life/makair’ (makers-for life, 2020), which states the following in it’s README page: “Aims at helping hospitals cope with a possible shortage of professional ventilators during the outbreak. Worldwide. … We target a per-unit cost well under 500 EUR, which could easily be shrunk down to 200 EUR or even 100 EUR per ventilator given proper economies of scale, as well as choices of cheaper on-the-shelf components”. The project includes design of the proposed ventilators as CAD files, as well as relevant firmware available as C++ code files.

Another example is the ‘popsolutions/openventilator’ (popsolutions, 2020), which aims to provide cheap but reliable ventilators to treat COVID-19 in economically under-developed regions of the world. The software project initiated from a Facebook group called ‘Open Source COVID-19 Medical Supplies’ 444https://www.facebook.com/groups/opensourcecovid19medicalsupplies/, where members discussed the scarcity of ventilators and the importance of creating cheap ventilators through efficient design. In the project we notice developers to create, build, and share designs using OpenSCAD scripts. OpenSCAD is an open source tool to build computer-aided design (CAD) objects 555https://www.openscad.org/.

IV: Mining:: This category includes projects that provide APIs to mine COVID-19 data from data sources, such as the US Center for Disease Control and Prevention (CDC) (CDC, 2020), the World Health Organization (WHO) (WHO, 2020), and data reported from local institutions. The purpose of this category of software is to provide utilities for software developers so that they can get real-time access to COVID-19 data to build aggregation software, discussed above. Because of the nature of the pandemic, access to real-time data is pivotal for accurate aggregation and analysis. The mining tools help developers to get such support. Mining software can be location specific, for example ‘dsfsi/covid19africa’ (dsfsi, 2020a) is dedicated to curate and collate COVID-19 related data for African countries.

V: User tracking:: This category includes software projects that collects information from users regarding their COVID-19 infection status. Tracking of user information can happen voluntarily, where the user voluntarily self reports COVID-19 infection status. The ‘enigmampc/SafeTrace’ (enigmampc, 2020) software is an example where users self report their infection status as well as location history. Tracking of user information can also be done using inference, as done in ‘OpenMined/covid-alert’ (OpenMined, 2020), where the software collects user’s location information to predict if the user is in a location with high infection density. One utility of these projects is to identify high-risk locations so that users can have an understanding of which nearby location can be avoided. Self reporting software have yielded benefits for China and South Korea (Huang et al., 2020).

VI: Statistical modeling:: This category includes software that use statistical models to predict attributes related to COVID-19. The purpose of the projects is to make predictions for the future based on existing data. Example usage of statistical models include (i) predicting death rate as done in ‘ImperialCollegeLondon/covid19model’ (ImperialCollegeLondon, 2020), (ii) automating the process of lung segmentation with computerized tomography (CT) scan, as done in ‘JoHof/lungmask’ (JoHof, 2020), (iii) predicting the impact of the COVID-19 pandemic on hospital demands as done in ‘neherlab/covid19_scenarios’ (neherlab, 2020), and (iv) predicting presence of COVID-19 with X-ray images using deep learning as done in ‘elcronos/COVID-19’ (elcronos, 2020).

VII: Volunteer management:: This category includes software used to efficiently manage volunteering effort. The purpose of this software is to build software platforms so that users can volunteer and participate in activities to help distressed families and communities. One example is the ‘covid-volunteers’ (helpwithcovid, 2020) software, which provides a web portal where users can sign up for 650 projects that include donation of masks, personal protective equipment (PPEs), and testing of COVID-19 666https://helpwithcovid.com/medical. Platforms can be global, such as ‘covid-volunteers’, and also regional, for example ‘Applifting/pomuzeme.si’ (Applifting, 2020) creates a web portal so that people inside Czech Republic can volunteer.

3.2.3 Frequency of the Identified Categories

Based on project count aggregation is the most frequent category. Along with project count, we provide summary statistics of projects that belong to each category in Table 3. We also observe on average user tracking projects to be more frequently released compared to other project types.

We identify four software projects that belong to multiple categories. As an example, the ‘soroushchehresa/awesome-coronavirus’ (soroushchehresa, 2020) project belongs to the categories: aggregation, mining, and statistical modeling.

Proj. Categ. Projects Com. Devs Files Iss. Rele.
Aggregation 50 14,985 663 8,641 908 72
Mining 35 9,671 894 6,714 515 21
Stat. model. 22 7,214 429 3,464 491 38
Education 9 4,550 196 1,696 406 14
User track 9 2,020 152 2,291 119 286
Volunteer. 7 2,186 143 2,041 320 0
Med. equip. 3 859 38 790 14 63
Table 3: Summary statistics of projects that belong to each category. Based on project count ‘Aggregation’ is the most frequent category.

3.2.4 Rater Agreement

We report agreement rate for three steps: open coding, closed coding, and rater verification.

Open coding: After completing open coding, the first and second author respectively, identified 7 and 10 categories. The agreement rate is 70.0%, and the Cohen’s Kappa is 0.7, indicating ‘substantial’ agreement (Landis and Koch, 1977). The authors disagreed on ‘Volunteering software related to local communities’, ‘Education bots’, and ‘Aggregated visualizations’, additional categories identified the second author. Upon discussion both authors agree that the category ‘Education bots’ can be merged with ‘Education’ as it encompasses all categories of knowledge software, such as bots and web applications. The authors also agreed that ‘Volunteering software related to local communities’ can be merged with ‘Volunteer management’ and ‘Aggregated visualizations’ can be merged with ‘Aggregation’, as ‘Aggregation’ includes software that aggregates COVID-19 data and displays aggregated data with visualizations.

Closed coding: During closed coding the first and second author mapped each of the 129 project to an existing category. The agreement rate is 93.8%. The Cohen’s Kappa is 0.92. The authors disagreed on the labeling of 8 projects, which are resolved through discussion.

Rater verification: We also measured the agreement rate between an additional rater and the authors for categorizing README files of projects. Cohen’s Kappa between the additional rater and the first author for a randomly selected set of 50 README files is 0.73, indicating ‘substantial’ agreement (Landis and Koch, 1977). Cohen’s Kappa between the additional rater and the second author for a randomly selected set of 50 README files is 0.73, indicating ‘substantial’ agreement (Landis and Koch, 1977). The agreement rate between the additional rater and the first and second author is respectively, 78.0% and 76.0%.

4 RQ2: Bug Categories

In this section, we answer “RQ2: What categories of bugs appear in COVID-19 software projects? How frequently do the identified bug categories appear? What is the resolution time for each bug category?” A categorization of bugs for COVID-19 software projects can inform practitioners and researchers about how software related to COVID-19 is developed and in which areas they can help. Furthermore, educators can learn about the software bugs that occur in a software related to a pandemic and disseminate these findings in the classroom. Frequency of the identified bug categories can help us understand what categories of software tend to contain what types of software bugs and provide quality improvement suggestions accordingly. Quantifying the resolution time for bugs in software projects can help software engineering researchers provide actionable guidelines to practitioners. For example, Wan et al. (Wan et al., 2017) observed that for blockchain software projects security bugs can take longer to fix compared to other bug categories. Based on their findings Wan et al. (Wan et al., 2017) recommended that blockchain project maintainers can adopt security analysis and repair tools to fix security bugs quickly.

4.1 Methodology to Identify Bug Categories, Frequency, and Resolution Time

In this section we provide the methodology to identify bug categories, quantify bug category frequency, and bug resolution time.

Methodology to Identify Bug Categories: We identify bug categories using the following steps:

  • Step#1-Filtering: We collect the 4,405 issue reports from the 129 projects and manually inspect each issue report. We do not rely on automated approach, such as keyword search or using bug labels, as automated approaches tend to generate false positives, which may bias research results (Herzig et al., 2013). While inspecting each issue report we use the following IEEE definition for bugs: “an imperfection that needs to be replaced or repaired” (IEEE, 2010), similar to prior work (Rahman et al., 2020). By completing this step we will obtain a set of closed issues reports that correspond to bugs. We use closed reports because as open bug reports are often incomplete and may not help in identifying bugs (Wan et al., 2017).

    The first and second author manually inspect individually to identify what issue reports correspond to bugs. We record agreement rate and Cohen’s Kappa (Cohen, 1960) between the first and second author. Disagreements between the first and second author is resolved through discussions. The process is subjective and susceptible to the bias of the first and second author. We mitigate the bias by using an additional rater, who inspected randomly inspected 100 issue reports and classified them as bug reports and non-bug reports. The additional rater is the fourth year PhD student at Tennessee Technological University who is also involved in rater verification for RQ1.

  • Step#2-Open coding: We apply open coding (Saldana, 2015) on the content of the collected bug reports from Step#1. Our open coding process for deriving bug categories is similar to our process of deriving project categories illustrated in Figure 3. First, we extract raw text from bug report titles and description, from which we generate initial categories. Next, we merge initial categories based on the commonalities and generate categories.

    Similar to deriving project categories, the first and second author separately apply the process of open coding to generate bug categories. Upon completion of the process we quantify agreement rate and measure Cohen’s Kappa (Cohen, 1960). For disagreements we conduct discussion. Generated categories upon discussion is considered final.

Methodology to Quantify Bug Category Frequency: We apply the following steps to quantify the frequency of identified bug categories:

  • Step#1-Closed coding: We apply closed coding (Crabtree and Miller, 1999) to map each identified category to the bug reports that we study. The first and second author separately apply closed coding for the collected bugs from Step#1. Upon completion, we calculate the agreement rate and Cohen’s Kappa (Cohen, 1960). Disagreements are resolved using discussion.

  • Step#2-Metric calculation: We quantify the frequency of the identified bug categories using two metrics: ‘Proportion of Bugs Across All Projects (BugPropAll)’ and ‘Proportion of Bugs For a Certain Project Category (BugPropCateg)’. We use Equations 1 and 2 to respectively calculate ‘BugPropAll’ and ‘BugPropCateg’. The ‘BugPropAll’ metric using Equation 1 provides a holistic overview of the frequency of identified bug categories. The ‘BugPropCateg’ metric using Equation 2 provides a granular overview of bug category frequency for each software project types identified from Section 3.2.2.

  • Step#3-Rater verification: The use of first and second author as raters to conduct closed coding is susceptible to rater bias. We mitigate this limitation by allocating an additional rater. We assign randomly selected 250 bug reports to the additional rater who apply closed coding. We provide the additional rater with a document that provides definitions of each identified category with examples.

    Similar to our process of rater verification for project categorization, the additional rater is the fourth year PhD student in the Department of Computer Science in Tennessee Tech. University. The fourth year PhD student is involved in the rater verification process for identifying project categories and labeling issue reports as bug reports.

(1)
(2)

Methodology to Quantify Bug Resolution Time We use the open and closing timestamp for each closed bug report in our dataset to quantify the resolution time for each bug category, similar to Wan et al. (Wan et al., 2017). We calculate bug resolution time by computing the number of hours that have elapsed between when the bug report is opened and closed, and not re-opened again, as per our dataset , which was downloaded on April 04, 2020. We report bug resolution time for all bug categories, as well as for bug reports that belong to certain categories of software projects.

4.2 Answer to RQ2

We answer RQ2 by first providing a breakdown of how we obtained our bug reports in Table 4 and 5. As shown in Table 5, the categories with the most and least bug reports is respectively, aggregation and medical equipment. One project can belong to multiple categories, and that is why the total count of bug reports do not total to 550. On average, bug reports per project to vary from 1.36.4, as shown in the ‘BugReport/Project’ column.

Initial 4,095
Criterion-1 (Closed issues) 2,965
Criterion-2 (Valid bug reports) 550
Final 550
Table 4: Filtering of bug reports from COVID-19 software projects.
Project category Count (%) BugReport/Project
Aggregation 220 (40%) 4.4
Mining 150 (27.3%) 4.3
Stat. Model. 98 (17.8%) 4.4
Education 58 (10.5%) 6.4
Volunteer. 40 (7.3%) 4.4
User Track 31 (5.6%) 4.4
Med. Equip. 4 (0.7%) 1.3
Table 5: Count of bug reports for each category of COVID-19 software projects. Aggregation-related projects have the highest amount of bug reports.

Next, we describe the identified bug categories in Section 4.2.1 by applying open coding on the collected 550 bug reports. The frequency of the identified bug categories is provided in Section 4.2.2. We provide details of rater verification in Section 4.2.3. Finally, we provide the bug resolution time in Section 4.2.4.

4.2.1 Bug Categories of COVID-19 Projects

We identify 8 bug categories, which we describe below alphabetically:

I: Algorithm:: This category corresponds to bugs when implementation of an algorithm does not follow expected behavior. An algorithm is a sequence of computational steps that transform input into output (Cormen et al., 2009). We observe algorithm bugs to include two sub-categories: (i) bugs related to statistical modeling algorithms, where statistical modeling results are incorrect due to incorrect assumptions and/or implementations, and (ii) bugs related to incorrect logic implemented in the software. Algorithm bugs have been previously observed in autonomous vehicle software (Garcia et al., 2020) and machine learning software (Thung et al., 2012).

Example: We provide examples for the two sub-categories:

  • Statistical modeling: In a bug report titled “Death rates should increase when ICU’s are overwhelmed” (Begley, 2020a), a practitioner describes how incorrect assumption can result in incorrect modeling behavior. The practitioner discusses that bed space is correlated with estimation of fatality rate. When bed space of hospitals are exhausted hospitals will not be able to treat new COVID-19 new patients, which could potentially increase the fatality rate.

    The bug report provides evidence that if the context of COVID-19 is not correctly incorporated in statistical models, them models will provide incorrect results. Incorrect statistical models can be consequential, as countries are adopting public health policies specific to COVID-19. For example researchers have critiqued the statistical models derived by the Institute for Health Metrics and Evaluation at the University of Washington (IHME), and advised USA policy makers to use the modeling results with caution (Begley, 2020b).

  • Incorrect logic: In a bug report titled “Fix Prefecture Sorting” (reustle, 2020), a practitioner describes a sorting bug which occurs when trying to visualize COVID-19 cases based on prefectures in Japan. A prefecture is an administrative jurisdiction in a country similar to a state or province (Hu and Qian, 2017). The bug occurred due to an incorrect logic that did not perform sorting by prefectures.

II: Data:: This category corresponds to bugs that occur during mining and storage of COVID-19 data. As discussed in Section 3.2.2 we observed our dataset to include projects that mine and aggregate COVID-19 data. We observe four sub-categories of data bugs: (i) storage: bugs that occur while storing data in a database, (ii) mining: bugs that occur while retrieving data from data APIs, (iii) location: bugs where location information in stored data is incorrect, and (iv) time series: bugs that correspond to missing data for a certain time period. Data bugs have been previously reported for deep learning software (Islam et al., 2019).

Example: We provide examples for each of these sub-categories below:

  • Storage: In a bug report titled “Temperature data not saved in the backend” (pavel ilin, 2020), a practitioner describes a bug where patient temperature data is inserted in the front-end but not stored in the database.

  • Mining: When mining data from sources bugs occur. A practitioner describes a mining bug in a bug report titled “CDC Children scraper is outdated” (Timoeller, 2020). The mining tool mines data related to children affected by COVID-19.

  • Location: in a bug report titled “Rajasthan District names are wrong”, a practitioner describes that inserted location data for an Indian state called ‘Rajasthan’ is wrong (SinghRajenM, 2020).

  • Time series: missing data was reported for a project and reported in a bug report titled “Data has a gap between 2020-3-11 and 2020-3-24” (zbraniecki, 2020).

III: Dependency:: This category corresponds to bugs that occur when execution of the software is dependent on a software artifact that is either missing or incorrectly specified. For COVID-19 projects an artifact can be an API or a build artifact. Dependency bugs were previously reported in IaC (Rahman et al., 2020), machine learning (Thung et al., 2012), and audio processing software (Cotroneo et al., 2013).

Example: In a bug report titled “Missing PostGIS” (vaclavpavlicek, 2020), a practitioner describes that installation and execution of the software is prohibited due to a software package called ‘PostGIS’, which is used to store spatial and geographic measurements, such as area, distance, polygon, and perimeter in PostgreSQL databases.

IV: Documentation:: This category corresponds to bugs that occur when incorrect and/or incomplete information in specified in release notes, maintenance notes, and documentation files, such as README files. Documentation bugs have been reported in prior research related to autonomous vehicle (Garcia et al., 2020) and IaC (Rahman et al., 2020).

Example: In a bug report titled “Missing code of conduct”, a practitioner describes a ‘CODE_OF_CONDUCT.md’ file to be missing in a Markdown file that describes how practitioners can contribute to the project (mdeous, 2020).

V: Performance:: This category corresponds to bugs that cause performance discrepancies for the software. Performance bugs are manifested in slow response of the web or mobile app. Performance bugs have been previously reported in prior research related to blockchain software (Wan et al., 2017).

Example: In a bug report titled “Cluster animation slowing down the browser. It also takes much time”, a practitioner describes how a performance bug related to an animation feature is slowing down a Firefox browser on Windows 10 (Subratappt, 2020). The performance bug was reported for a website called ‘covid19india.org’ 777https://www.covid19india.org/, which aggregates COVID-19 data for India and displays them.

VI: Security:: This category corresponds to bugs that violate confidentiality, integrity, or availability for the software. Prior research on bug categorization have also observed security bugs to appear for blockchain software (Wan et al., 2017), video game software (Pascarella et al., 2018), and IaC (Rahman et al., 2020).

Example: In a bug report titled “Fix password reset procedure” (landovsky, 2020), a practitioner describes a password reset bug, where the password reset procedure ends arbitrarily after 500 login attempts.

VII: Syntax:: This category corresponds to bugs related with the syntax of the programming languages used to develop the software. Syntax-related bugs have been reported in prior research related to IaC (Rahman et al., 2020), deep learning (Islam et al., 2019), and OSS GitHub software (Ray et al., 2014).

Example: We notice bugs related to data types in ‘neherlab/covid19_scenarios’. In the bug report titled “Fix types and linting errors” (ivan aksamentov, 2020), a practitioner describes how linting and type checking was disabled for the project, which led to bugs related to linting and type checking.

VIII: UI:: This category corresponds to bugs that involve the user interface (UI) of the software. UI bugs include navigation-related bugs on web pages, bugs related to accessibility, displaying incorrect images, links, and color, and responsiveness. UI bugs have been previous reported for blockchain software (Wan et al., 2017).

Example: In a bug report titled “accessibility fixes” (abquirarte, 2020) describes a UI bug related to accessibility. According to the bug report, a screen reader incorrectly renders check marks and crosses in front of the “Do’s and Don’t as M’s and N’s”.

4.2.2 Frequency of Identified Bug Categories

Based on the ‘Proportion of Bugs Across All Projects (BugPropAll)’ metric we observe UI bugs to be the most frequent category, whereas documentation is the least frequent category. We provide a compete breakdown of the metric in Table 6. Data bugs have four sub-categories: storage, mining, location, and time series. The frequency for storage, mining, location, and time series is respectively, 4.7%, 5.8%, 87.2%, and 2.3%. Algorithm bugs have two sub-categories: statistical modeling and wrong logic. The frequency for statistical modeling and wrong logic is respectively, 42.3% and 57.7%.

Bug category BugPropAll (%)
UI 38.2
Data 30.9
Dependency 18.9
Algorithm 7.8
Syntax 6.7
Security 2.5
Performance 1.6
Documentation 1.4
Table 6: Frequency of identified bug categories. UI-related bugs are the most frequent.

We observe bug category frequency to vary for different categories of projects. We provide the ‘Proportion of Bugs For a Certain Project Category (BugPropCat)’ values for each project category in Table 7. ‘AGG’, ‘MINE’, ‘STA’, ‘EDU’, ‘TRAK’, ‘VOL’ and ‘EQU’ respectively, corresponds to the seven project categories: aggregation, mining, statistical modeling, education, user tracking, volunteer management system, and medical equipment.

According to Table 7, except for mining and medical equipment software, the dominant bug category is UI. One possible explanation can be the analyzed software projects have UIs, which may have contributed to the frequency of UI bugs. For mining software the dominant bug category is data bugs i.e., bugs that occur due to storing and processing of COVID-19 data. For medical equipment software the dominant bug category is dependency. We also notice algorithm bugs to be the second most frequent bug category for statistical modeling software. Similar to prior work on machine learning (Thung et al., 2012), we expected algorithm bugs to be the most dominant category for statistical modeling. Statistical modeling software also have UIs for user interaction, and the count of UI bugs may have foreshadowed the count of algorithm bugs.

AGG MINE STA EDU TRAK VOL EQU
Bug categ.
Algorithm 6.8% 6.7% 22.4% 3.4% 0.0% 2.5% 0.0%
Data 28.6% 60.6% 13.2% 15.5% 0.0% 12.5% 0.0%
Dependency 16.3% 18.0% 18.3% 24.1% 9.7% 27.5% 75.0%
Document 0.9% 1.3% 1.0% 0.0% 0.0% 10.0% 0.0%
Performance 2.7% 2.0% 0.0% 0.0% 3.2% 0.0% 0.0%
Security 1.8% 0.0% 3.0% 3.4% 6.4% 12.5% 0.0%
Syntax 5.9% 3.3% 14.3% 17.2% 3.2% 10.0% 0.0%
UI 50.0% 12.0% 34.7% 44.8% 77.4% 32.5% 25.0%
Table 7: Bug category frequency for each identified project type. All values are presented in (%).

4.2.3 Rater Agreement and Verification

We report agreement rate for four steps: issue labeling, open coding, closed coding, and rater verification.

Labeling issues as bugs: While labeling collected issue reports as bug reports and non-bug reports the agreement rate is 96.5% and the Cohen’s Kappa is 0.9.

Open coding to identify bug categories: The first and second author respectively, identified 9 and 10 categories. The agreement rate is 72.7%, and the Cohen’s Kappa is 0.70, indicating ‘substantial’ agreement (Landis and Koch, 1977). The first author identified ‘database’ as a category not identified by the second author. Upon discussion both authors agreed that ‘database’ is related to data storage and belongs to the data category. The second author identified two additional categories ‘Public health data’ and ‘Type errors’. After discussing the definitions of all categories both authors agreed that ‘Public health data’ and ‘Type errors’ can respectively, be merged with data and syntax.

Closed coding to quantify bug category frequency: During closed coding the first and second author mapped each project to an existing category. The agreement rate is 95.1% and the Cohen’s Kappa is 0.93. The authors disagreed on the labeling of 27 bug reports, which are resolved through discussion.

Rater verification: For the randomly selected 250 issue reports we allocate an additional rater who manually identified which of the issue reports are bug reports and non-bug reports. The Cohen’s Kappa between the additional rater and the first author is 0.80, indicating ‘substantial’ agreement (Landis and Koch, 1977). The Cohen’s Kappa between the additional rater and the second author is 0.84, indicating ‘perfect’ agreement (Landis and Koch, 1977). The agreement rate between the additional rater and the first and second author is respectively, 89.0% and 93.0%.

We have also measured the agreement rate between an additional rater and the authors for categorizing bug reports. Cohen’s Kappa between the additional rater and the first author for a randomly selected set of 250 bug reports is 0.65, indicating ‘substantial’ agreement (Landis and Koch, 1977). Cohen’s Kappa between the additional rater and the second author for a randomly selected set of 250 bug reports is 0.68, indicating ‘substantial’ agreement (Landis and Koch, 1977). The agreement rate between the additional rater and the first and second author is respectively, 78.0% and 81.6%.

4.2.4 Resolution Time of Identified Bug Categories

We provide bug resolution time as measured in hours for all bug categories in Table 8. From Table 8 we observe that based on min and median bug resolution times security bugs take the longest to resolve, followed algorithm bugs. We also observe data bugs to take as long as 548 hours to resolve.

A breakdown of bug resolution time across the six categories of projects is provided in Table 9. The ‘All’ row in Table 9 shows the minimum, median, and maximum bug resolution time for all bug categories measured in hours.

In Table 9 we observe four instances where the minimum bug resolution time is less than 6 minutes ( hours). One possible explanation can be practitioners’ habit of opening a bug report after they have developed the fix for a bug (Wan et al., 2017; Thung et al., 2012). In such cases, practitioners notice the bug early, construct the fix for the bug, and then submit the bug report by opening and closing the bug report promptly.

Bug category Min Median Max
Security 1.240 13.9 144.6
Algorithm 0.041 13.5 172.7
Syntax 0.004 12.1 174.2
UI 0.003 11.8 254.2
Data 0.003 8.4 548.0
Performance 0.961 7.1 104.4
Dependency 0.014 2.4 379.4
Documentation 0.013 1.4 76.8
Table 8: Resolution time of identified bug categories. Resolution times is measured in hours. Median resolution time is highest for security bugs.
Project category Min Median Max
Medical Equipment 5.0 29.4 46.4
Volunteer Management System 0.013 21.1 174.2
User Tracking 0.124 16.5 294.5
Education 0.121 11.2 294.5
Aggregation 0.003 8.7 379.4
Statistical Modeling 0.004 7.2 168.3
Mining 0.005 2.5 548.1
All 0.003 7.4 548.0
Table 9: Resolution time of bug categories grouped by project categories. We measure resolution time in hours. Median bug resolution time is highest for projects related to medical equipment software.

Median bug resolution duration for each project type and bug category is provided in Table 10. ‘AGG’, ‘MINE’, ‘STA’, ‘EDU’, ‘TRAK’, ‘VOL’ and ‘EQU’ respectively, corresponds to the seven project categories: aggregation, mining, statistical modeling, education, user tracking, volunteer management system, and medical equipment. We observe median bug resolution time to vary across bug categories as well as for project categories.

AGG MINE STA EDU TRAK VOL EQU
Bug cat.
Algorithm 9.8 10.8 13.9 10.1 13.5
Data 12.2 4.4 15.2 17.0 42.0
Dependency 5.6 0.1 0.3 4.5 5.3 2.9 22.4
Document 1.3 39.0 1.5 6.9
Performance 7.1 36.6 1.5
Security 8.1 3.1 84.1 13.9 20.4
Syntax 12.1 4.7 11.4 8.6 16.9 79.3
UI 8.3 2.7 13.1 16.8 18.7 21.9 46.4
Table 10: Median bug resolution time for each bug category and each project type measured in hours. ‘—’ indicates categories for which no bug reports exist.

5 Discussion

In this section, we first provide a summary of our findings in Section 5.1. Next, we provide a discussion on the implications of our findings in Section 5.2.

5.1 Summary

Project category: Aggregation Definition: Aggregate COVID-19 data and present using visualizations Count : 50 out of 129 (38.7%) Most frequent bug category: UI bugs Median bug resolution time: 8.7 hours Project category: Mining Definition: Mine COVID-19 data Count : 35 out of 129 (27.1%) Most frequent bug category: Data bugs Median bug resolution time: 2.5 hours Project category: Statistical modeling Definition: Use of statistical models to make COVID-19 predictions Count : 22 out of 129 (17.0%) Most frequent bug category: UI bugs Median bug resolution time: 7.2 hours Project category: Education Definition: Educate people about COVID-19 Count : 9 out of 129 (6.9%) Most frequent bug category: UI bugs Median bug resolution time: 11.2 hours Project category: User tracking Definition: Track user data related to COVID-19 Count : 9 out of 129 (6.9%) Most frequent bug category: UI bugs Median bug resolution time: 16.5 hours Project category: Volunteer management Definition: Efficiently manage volunteering effort related to COVID-19 Count : 7 out of 129 (5.4%) Most frequent bug category: UI bugs Median bug resolution time: 21.1 hours Project category: Medical equipment Definition: Source code for design and implementation of medical devices Count : 3 out of 129 (2.3%) Most frequent bug category: Dependency bugs Median bug resolution time: 29.4 hours

5.2 Implications

We discuss the implications of our findings below:

Security and privacy implications of user tracking software: From Table 3 we observe 9 projects to be related with user tracking. While the benefits of user tracking software has been documented for countries, such as Russia and South Korea (Morning, 2020), this category of software can have negative impacts on privacy of end-users. Data generated from user tracking software can be leveraged for marketing purposes. To preserve privacy of user data in user tracking software we make the following recommendations:

  • Policy makers should construct policies specific to COVID-19 software that collects user data.

  • Practitioners who develop user tracking software should leverage existing privacy policy frameworks, such as the ‘National Institute of Standards and Technology (NIST) Privacy Framework’ (National Institute of Standard and Technology, 2020).

  • Privacy researchers can build tools that will automatically detect and report privacy policy violations.

Evidence from Table 7 shows that security bugs to exist for user tracking software. We advocate security researchers to systematically investigate if user tracking software includes security bugs. Recent news articles suggest that user tracking software, such as contract tracing apps may become more and more prevalent as Apple and Google are already providing frameworks to build software that tracks user data. (Apple, 2020). Our hypothesis is that availability of these frameworks will facilitate rapid development and deployment of mobile apps that collect user data. Security weaknesses in these apps can provide malicious users opportunity to conduct large-scale data breaches. We notice anecdotal evidence in this regard: a researcher has identified vulnerabilities in a user tracking app that could leak user location data (Greenberg, 2020). Panelists at EuroCrypt 2020, a cryptography research conference, discussed limitations of user tracking mobile apps for COVID-19 with respect to API design, indoor location tracking, and informing users about privacy risks (EuroCrypt, 2020a) (EuroCrypt, 2020b).

Towards constructing correct statistical models: From Section 4.2.1 we have observed statistical modeling bugs to exist. Bugs related to statistical modeling can be consequential because based on the predictions generated by statistical models, policy makers enforce public health policies. One possible explanation for buggy statistical models can be attributed to the quality of datasets using which statistical models are build (Koerth et al., 2020). For example, fatality prediction models that are built using the ‘Diamond Princess Cruise Ship Dataset’ may not be applicable for a specific geographic region with low population density. Another possible explanation can be a lack of context and knowledge related to public health specific that hinders model builders to identify appropriate independent variables to construct the models. Incorrect estimation of hospital beds from our discussion in Section 4.2.1 is one example. Other examples of independent variables related to public health includes staff availability, count of known cases, hospitalization rate etc. (Attia, 2020). According to a health expert (Attia, 2020), statistical models that predicted 2.4 million US residents to die, assumed a hospitalization rate of 15-20%, which in reality was 5%.

Based on our findings and above-mentioned explanations we make two recommendations:

  • Automated testing for COVID-19 modeling: We hope to see novel research in the domain of COVID-19 that will test the correctness of constructed statistical models used in forecasting in an automated manner. In recent years, we have seen research efforts that test deep learning models (Tian et al., 2018; Pei et al., 2017; Ma et al., 2018). We expect similar research pursuits for COVID-19 statistical modeling.

  • Better synergies between data science and public health practitioners: Construction and verification of COVID-19 statistical modeling should involve practitioners from public health and data science. Public health practitioners within a specific locality can provide necessary context that data scientists can incorporate in their statistical models.

Implications for Educators: Our findings have implications for educators involved in teaching the following topics:

  • Data science: Educators who teach data science can use the examples of statistical modeling bugs to highlight the value of considering the full context and related limitations that accompany statistical modeling.

  • Information security and privacy: User tracking software can be discussed in information security and privacy courses to demonstrate the value of protecting user data. Such discussion can also include privacy policy frameworks that are already in place, such as the NIST Privacy framework (National Institute of Standard and Technology, 2020).

  • Software engineering: Our categorization of bugs related to COVID-19 software development can be discussed to demonstrate that understanding and repair of bugs requires contextualization.

Benchmark for practitioners and researchers: Tables 6— 10 can be used as a measuring stick by practitioners and researchers who are involved with COVID-19 software projects. Practitioners can estimate their bug resolution efforts by comparing median resolution times for bugs in their COVID-19 software projects to that of Tables 89, and 10.

Compared to prior work related to blockchain and machine learning (Thung et al., 2012; Wan et al., 2017), median bug resolution time is lower for COVID-19 software projects. We provide two possible explanations: one possible explanation can be related to the sense of urgency. Practitioners may have realized that bugs in COVID-19 software projects could hamper the analysis or mitigation of COVID-19, and therefore, needs immediate attention. Another possible explanation can be the limitations of our dataset. The age of our software projects do not exceed four months and that may have biased median bug resolution time. We advocate for future research that will confirm or refute our explanations.

Recurrence-related implications: In Section 2.1, we have discussed evidence related to recurrence of COVID-19. We hypothesize that COVID-19’s recurrence will lead to more COVID-19 software building. Whether or not our findings hold for these newly constructed COVID-19 software can be validated through a replication of our paper. We expect to observe more categories of COVID-19 software projects as well as more bug categories that could occur with varying distributions.

Mitigation strategies: We suggest the following mitigation strategies for each of the identified bug categories:

  • [leftmargin=*]

  • Algorithm: Algorithm bugs can be mitigated through peer reviews. One sub-category of algorithm bugs is bugs related to statistical models built for COVID-19. Mitigation of bugs in statistical models may require synergies between practitioners from data science and public health.

  • Data: Data bugs can be mitigated by adequate accumulation of data, a strategy which requires non-trivial amount of effort. We advocate for community effort so that COVID-19 data sources are curated.

  • Dependency: Practitioners can mitigate dependency bugs by using automated tools, such as Dependabot 888https://dependabot.com/.

  • Documentation: Practitioners can mitigate documentation bugs through peer review and documentation-specific linters that can alert practitioners when incorrect information is specified in README files.

  • Performance: Researchers can use research tools, such as Toddler (Nistor et al., 2013) and Clarity (Olivo et al., 2015) to detect performance bugs in software source code. Furthermore, performance bugs in UIs can be detected by record and replay tools, such as Pounder as suggested by researchers (Adamoli et al., 2011).

  • Security: We advocate practitioners to use automated security scanning tools applicable for the language used in the COVID-19 software repository. Prevalence of security bugs can be detected and mitigated by applying security scanning tools.

  • Syntax: We advocate practitioners to use existing static analysis tools to mitigate syntax bugs.

  • UI: Practitioners can mitigate UI bugs by using automated UI testing tools, such as Selenium 999https://www.selenium.dev/. We have noticed UI bugs related accessibility issues, which can be mitigated by automated tools, such as Applause 101010https://www.applause.com/.

6 Threats to Validity

We describe the limitations of our paper as following:

Conclusion validity: We have used raters who derived the software and bug categories. Both raters are authors of the paper. Our derived categoires are susceptible to the authors’ bias. We mitigate this limitation by allocating another rater who is not the author of the paper who verified our ratings.

Our categories might not be comprehensive because our categorization for projects and bugs is limited to the dataset that we collected. The bug resolution time could be limiting as our dataset includes projects that have a duration of four months.

We use the topic ‘covid-19’ to identify and filter COVID-19 software projects from GitHub. Any software project that is not labeled as ‘covid-19’ will not be included in our dataset.

Our datasets have limited lifetime as the COVID-19 was discovered in December 2019, and the lack of maturity in our datasets may influence our analysis. We mitigate this limitation by identifying projects using a filtering criteria so that we can identify projects with sufficient development activity.

Internal validity: For RQ1 and RQ2 we use ourselves, the authors of the paper, as raters who conduct open and closed coding on README files and bug reports. Our research is susceptible to mono-method bias, as our categorization and labeling may be influenced by the authors’ implicit expectations and hypotheses about the study.

External validity: Our findings are not comprehensive. We have not analyzed projects hosted outside GitHub and private projects hosted on GitHub. We mitigate this limitation by analyzing 129 software projects that belong to 7 categories.

7 Conclusion

The COVID-19 pandemic has impacted people all over the world causing thousands of deaths. Software practitioners have joined the fight in combating the spread and mitigating the dire consequences of COVID-19. An understanding of COVID-19 software categories and software bugs can give us clues on how the software engineering community can help even further in combating COVID-19.

We conduct an empirical study with 129 COVID-19 software projects hosted on GitHub. We identify 7 categories of software projects: aggregation, mining, statistical models, education, volunteer management, user tracking, and medical equipment. By applying open coding on 550 bug reports, we identify 8 categories of bugs: algorithm, data, dependency, documentation, performance, security, syntax, and UI. We observe bug category frequency to vary with project categories, e.g., for mining projects data-related bugs is the most frequently occurring category.

Our findings have implications for educators, practitioners, and researchers. Educators can use our categorization of COVID software projects and related bugs to educate students about the security and privacy implications of COVID-19 software. Privacy researchers can build tools that will check if user tracking software related to COVID-19 are not leaking user data. Practitioners in the data science domain can learn from our categorization of statistical modeling bugs to understand limitations of constructed statistical models and verify underlying assumptions that accompany constructed statistical models. Based on our findings we also advocate for better synergies between data scientists and public health experts so that statistical modeling bugs can be mitigated. We hope our paper will advance further research in the domain of COVID-19 software.

Acknowledgements.
We thank the PASER research group members at Tennessee Technological University for their useful feedback. We also thank Farzana Ahamed Bhuiyan of Tennessee Technological University for her help as an additional rater.

References

  • abquirarte (2020) abquirarte (2020) accessibility fixes. github.com/cagov/covid19/issues/137, [Online; accessed 10-May-2020]
  • Adamoli et al. (2011) Adamoli A, Zaparanuks D, Jovic M, Hauswirth M (2011) Automated gui performance testing. Software Quality Journal 19(4):801–839
  • deepset ai (2020) deepset ai (2020) deepset-ai/covid-qa. https://github.com/deepset-ai/COVID-QA, [Online; accessed 09-May-2020]
  • ivan aksamentov (2020) ivan aksamentov (2020) Fix types and linting errors. https://github.com/neherlab/covid19_scenarios/issues/101, [Online; accessed 10-May-2020]
  • Alasdair Sandford (2020) Alasdair Sandford (2020) Coronavirus: Half of humanity now on lockdown as 90 countries call for confinement. https://www.euronews.com/2020/04/02/coronavirus-in-europe-spain-s-death-toll-hits-10-000-after-record-950-new-deaths-in-24-hou, [Online; accessed 17-Apr-2020]
  • Apple (2020) Apple (2020) Privacy-preserving contact tracing. https://www.apple.com/covid19/contacttracing, [Online; accessed 25-May-2020]
  • Applifting (2020) Applifting (2020) pomuzeme.si. github.com/Applifting/pomuzeme.si, [Online; accessed 09-May-2020]
  • Attia (2020) Attia P (2020) Comparing covid-19 to past pandemics, preparing for the future, and reasons for optimism. https://peterattiamd.com/ameshadalja/, [Online; accessed 21-May-2020]
  • Begley (2020a) Begley S (2020a) Death rates should increase when icu’s are overwhelmed. https://github.com/neherlab/covid19_scenarios/issues/7, [Online; accessed 10-May-2020]
  • Begley (2020b) Begley S (2020b) Influential covid-19 model uses flawed methods and shouldn’t guide u.s. policies, critics say. https://www.statnews.com/2020/04/17/influential-covid-19-model-uses-flawed-methods-shouldnt-guide-policies-critics-say/comment-page-5/, [Online; accessed 10-May-2020]
  • boogheta (2020) boogheta (2020) boogheta/coronavirus-countries. https://github.com/boogheta/coronavirus-countries, [Online; accessed 09-May-2020]
  • Catolino et al. (2019) Catolino G, Palomba F, Zaidman A, Ferrucci F (2019) Not all bugs are the same: Understanding, characterizing, and classifying bug types. Journal of Systems and Software 152:165 – 181, DOI https://doi.org/10.1016/j.jss.2019.03.002, URL http://www.sciencedirect.com/science/article/pii/S0164121219300536
  • CDC (2020) CDC (2020) Cases, data, and surveillance. https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/index.html, [Online; accessed 09-May-2020]
  • Chen et al. (2020) Chen D, Xu W, Lei Z, Huang Z, Liu J, Gao Z, Peng L (2020) Recurrence of positive sars-cov-2 rna in covid-19: A case report. International Journal of Infectious Diseases 93:297 – 299, DOI https://doi.org/10.1016/j.ijid.2020.03.003, URL http://www.sciencedirect.com/science/article/pii/S1201971220301223
  • Cohen (1960) Cohen J (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1):37–46, DOI 10.1177/001316446002000104, URL http://dx.doi.org/10.1177/001316446002000104
  • Corey et al. (2020) Corey L, Mascola JR, Fauci AS, Collins FS (2020) A strategic approach to covid-19 vaccine r&d. Science
  • Cormen et al. (2009) Cormen TH, Leiserson CE, Rivest RL, Stein C (2009) Introduction to algorithms. MIT press
  • Cotroneo et al. (2013) Cotroneo D, Pietrantuono R, Russo S (2013) Testing techniques selection based on odc fault types and software metrics. J Syst Softw 86(6):1613–1637, DOI 10.1016/j.jss.2013.02.020, URL http://dx.doi.org/10.1016/j.jss.2013.02.020
  • Crabtree and Miller (1999) Crabtree BF, Miller WL (1999) Doing qualitative research. sage publications
  • Currie et al. (2020) Currie CS, Fowler JW, Kotiadis K, Monks T, Onggo BS, Robertson DA, Tako AA (2020) How simulation modelling can help reduce the impact of covid-19. Journal of Simulation 0(0):1–15, DOI 10.1080/17477778.2020.1751570, URL https://doi.org/10.1080/17477778.2020.1751570, https://doi.org/10.1080/17477778.2020.1751570
  • Cyranoski (2020) Cyranoski D (2020) Mystery deepens over animal source of coronavirus. Nature 579(7797):18–19
  • De Clercq (2006) De Clercq E (2006) Potential antivirals and antiviral strategies against sars coronavirus infections. Expert review of anti-infective therapy 4(2):291–302
  • Dehning et al. (2020) Dehning J, Zierenberg J, Spitzner FP, Wibral M, Neto JP, Wilczek M, Priesemann V (2020) Inferring change points in the spread of covid-19 reveals the effectiveness of interventions. Science
  • Dey et al. (2020) Dey SK, Rahman MM, Siddiqi UR, Howlader A (2020) Analyzing the epidemiological outbreak of covid-19: A visual exploratory data analysis approach. Journal of Medical Virology 92(6):632–638, DOI 10.1002/jmv.25743, URL https://onlinelibrary.wiley.com/doi/abs/10.1002/jmv.25743, https://onlinelibrary.wiley.com/doi/pdf/10.1002/jmv.25743
  • dsfsi (2020a) dsfsi (2020a) dsfsi/covid19africa. https://github.com/dsfsi/covid19africa, [Online; accessed 09-May-2020]
  • dsfsi (2020b) dsfsi (2020b) dsfsi/covid19za. https://github.com/dsfsi/covid19za, [Online; accessed 09-May-2020]
  • elcronos (2020) elcronos (2020) elcronos/covid-19. https://github.com/elcronos/COVID-19, [Online; accessed 09-May-2020]
  • enigmampc (2020) enigmampc (2020) Safetrace. github.com/enigmampc/SafeTrace, [Online; accessed 09-May-2020]
  • Erin Duffin (2020) Erin Duffin (2020) Impact of the coronavirus pandemic on the global economy - statistics & facts. https://www.statista.com/topics/6139/covid-19-impact-on-the-global-economy/, [Online; accessed 08-May-2020]
  • EuroCrypt (2020a) EuroCrypt (2020a) Eurocrypt 2020 program. https://eurocrypt.iacr.org/2020/program.php, [Online; accessed 16-May-2020]
  • EuroCrypt (2020b) EuroCrypt (2020b) s-212 panel discussion on contact tracing. https://youtu.be/Xt4P8E_Y-xc, [Online; accessed 16-May-2020]
  • Evans et al. (2020) Evans AB, Blackwell J, Dolan P, Fahlén J, Hoekman R, Lenneis V, McNarry G, Smith M, Wilcock L (2020) Sport in the face of the covid-19 pandemic: towards an agenda for research in the sociology of sport
  • Farhana et al. (2019) Farhana E, Imtiaz N, Rahman A (2019) Synthesizing program execution time discrepancies in julia used for scientific software. In: 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp 496–500
  • Garcia et al. (2020) Garcia J, Feng Y, Shen J, Almanee Y Sumaya Xia, Chen QA (2020) A comprehensive study of autonomous vehicle bugs. In: Proceedings of the 42nd International Conference on Software Engineering, ICSE ’20, to appear
  • GitHub (2020a) GitHub (2020a) Covid-19 : Github topics. https://github.com/topics/covid-19, [Online; accessed 07-May-2020]
  • GitHub (2020b) GitHub (2020b) Language savant. https://github.com/github/linguist, [Online; accessed 07-May-2020]
  • GitHub (2020c) GitHub (2020c) Search : Covid-19. https://github.com/search?q=covid-19, [Online; accessed 07-May-2020]
  • Gorbalenya (2020) Gorbalenya BSBRea AE (2020) The species severe acute respiratory syndrome-related coronavirus: classifying 2019-ncov and naming it sars-cov-2. Nature Microbiology p 1
  • Greenberg (2020) Greenberg A (2020) India’s covid-19 contact tracing app could leak patient locations. https://www.wired.com/story/india-covid-19-contract-tracing-app-patient-location-privacy/, [Online; accessed 23-May-2020]
  • Helms et al. (2020) Helms J, Kremer S, Merdji H, Clere-Jehl R, Schenck M, Kummerlen C, Collange O, Boulay C, Fafi-Kremer S, Ohana M, et al. (2020) Neurologic features in severe sars-cov-2 infection. New England Journal of Medicine
  • helpwithcovid (2020) helpwithcovid (2020) helpwithcovid/covid-volunteers. https://github.com/helpwithcovid/covid-volunteers, [Online; accessed 09-May-2020]
  • Herzig et al. (2013) Herzig K, Just S, Zeller A (2013) It’s not a bug, it’s a feature: How misclassification impacts bug prediction. In: Proceedings of the 2013 International Conference on Software Engineering, IEEE Press, ICSE ’13, p 392–401
  • Hu and Qian (2017) Hu FZ, Qian J (2017) Land-based finance, fiscal autonomy and land supply for affordable housing in urban china: A prefecture-level analysis. Land Use Policy 69:454 – 460, DOI https://doi.org/10.1016/j.landusepol.2017.09.050, URL http://www.sciencedirect.com/science/article/pii/S0264837717309535
  • Huang et al. (2020) Huang Y, Sun M, Sui Y (2020) How digital contact tracing slowed covid-19 in east asia. https://hbr.org/2020/04/how-digital-contact-tracing-slowed-covid-19-in-east-asia, [Online; accessed 09-May-2020]
  • IEEE (2010) IEEE (2010) Ieee standard classification for software anomalies. IEEE Std 1044-2009 (Revision of IEEE Std 1044-1993) pp 1–23, DOI 10.1109/IEEESTD.2010.5399061
  • pavel ilin (2020) pavel ilin (2020) Temperature data not saved in the backend. https://github.com/COVID-19-electronic-health-system/Corona-tracker/issues/351, [Online; accessed 10-May-2020]
  • ImperialCollegeLondon (2020) ImperialCollegeLondon (2020) Imperialcollegelondon/covid19model. https://github.com/ImperialCollegeLondon/covid19model, [Online; accessed 09-May-2020]
  • Islam et al. (2019) Islam MJ, Nguyen G, Pan R, Rajan H (2019) A comprehensive study on deep learning bug characteristics. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Association for Computing Machinery, New York, NY, USA, ESEC/FSE 2019, p 510–520, DOI 10.1145/3338906.3338955, URL https://doi.org/10.1145/3338906.3338955
  • Jarynowski et al. (2020) Jarynowski A, Wójta-Kempa M, Płatek D, Czopek K (2020) Attempt to understand public health relevant social dimensions of covid-19 outbreak in poland. Available at SSRN 3570609
  • Jin et al. (2020) Jin Z, Zhao Y, Sun Y, Zhang B, Wang H, Wu Y, Zhu Y, Zhu C, Hu T, Du X, et al. (2020) Structural basis for the inhibition of sars-cov-2 main protease by antineoplastic drug carmofur. Nature Structural & Molecular Biology pp 1–4
  • John Hopkins University (2020) John Hopkins University (2020) Corona Virus Resource Center. https://coronavirus.jhu.edu/, [Online; accessed 31-May-2020]
  • JoHof (2020) JoHof (2020) Johof/lungmask. https://github.com/JoHof/lungmask, [Online; accessed 09-May-2020]
  • juanmnl (2020) juanmnl (2020) covid19-monitor. github.com/juanmnl/covid19-monitor, [Online; accessed 09-May-2020]
  • Kissler et al. (2020) Kissler SM, Tedijanto C, Goldstein E, Grad YH, Lipsitch M (2020) Projecting the transmission dynamics of sars-cov-2 through the postpandemic period. Science
  • Koerth et al. (2020) Koerth M, Bronner L, Mithani J (2020) Why it’s so freaking hard to make a good covid-19 model. https://fivethirtyeight.com/features/why-its-so-freaking-hard-to-make-a-good-covid-19-model/, [Online; accessed 22-May-2020]
  • Kraemer et al. (2020) Kraemer MU, Yang CH, Gutierrez B, Wu CH, Klein B, Pigott DM, du Plessis L, Faria NR, Li R, Hanage WP, et al. (2020) The effect of human mobility and control measures on the covid-19 epidemic in china. Science 368(6490):493–497
  • Landis and Koch (1977) Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174, URL http://www.jstor.org/stable/2529310
  • landovsky (2020) landovsky (2020) Fix password reset procedure. https://github.com/Applifting/pomuzeme.si/issues/99, [Online; accessed 10-May-2020]
  • makers-for life (2020) makers-for life (2020) makers-for-life/makair. https://github.com/makers-for-life/makair, [Online; accessed 09-May-2020]
  • Linares-Vásquez et al. (2017) Linares-Vásquez M, Bavota G, Escobar-Velasquez C (2017) An empirical study on android-related vulnerabilities. In: Proceedings of the 14th International Conference on Mining Software Repositories, IEEE Press, Piscataway, NJ, USA, MSR ’17, pp 2–13, DOI 10.1109/MSR.2017.60, URL https://doi.org/10.1109/MSR.2017.60
  • Ma et al. (2018) Ma L, Zhang F, Sun J, Xue M, Li B, Juefei-Xu F, Xie C, Li L, Liu Y, Zhao J, Wang Y (2018) Deepmutation: Mutation testing of deep learning systems. In: 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE), pp 100–111
  • Ma et al. (2017) Ma W, Chen L, Zhang X, Zhou Y, Xu B (2017) How do developers fix cross-project correlated bugs? a case study on the github scientific python ecosystem. In: Proceedings of the 39th International Conference on Software Engineering, IEEE Press, ICSE ’17, p 381–392, DOI 10.1109/ICSE.2017.42, URL https://doi.org/10.1109/ICSE.2017.42
  • mdeous (2020) mdeous (2020) Missing code of conduct. https://github.com/reach4help/reach4help/issues/135, [Online; accessed 10-May-2020]
  • Mello and Wang (2020) Mello MM, Wang CJ (2020) Ethics and governance for digital disease surveillance. Science
  • Mitchell Hartman (2020) Mitchell Hartman (2020) Covid-19 jobless claims are now over 40 million. many are still waiting for unemployment benefits. https://www.marketplace.org/2020/05/28/covid-19-jobless-claims-unemployment-benefits-waiting/, [Online; accessed 31-May-2020]
  • Mockus et al. (2002) Mockus A, Fielding RT, Herbsleb JD (2002) Two case studies of open source software development: Apache and mozilla. ACM Trans Softw Eng Methodol 11(3):309–346, DOI 10.1145/567793.567795, URL https://doi.org/10.1145/567793.567795
  • Morning (2020) Morning C (2020) Mobile applications for covid tracking & tracing – balancing the need for personal information and privacy rights in the time of coronavirus. https://www.crowell.com/NewsEvents/AlertsNewsletters/all/Mobile-Applications-For-COVID-Tracking-Tracing-Balancing-the-Need-for-Personal-Information-and-Privacy-Rights-in-the-Time-of-Coronavirus, [Online; accessed 20-May-2020]
  • Munaiah et al. (2017) Munaiah N, Kroh S, Cabrey C, Nagappan M (2017) Curating github for engineered software projects. Empirical Software Engineering pp 1–35, DOI 10.1007/s10664-017-9512-6, URL http://dx.doi.org/10.1007/s10664-017-9512-6
  • National Institute of Standard and Technology (2020) National Institute of Standard and Technology (2020) Nist privacy framework. https://www.nist.gov/privacy-framework, [Online; accessed 24-May-2020]
  • Naudé (2020)

    Naudé W (2020) Artificial intelligence against covid-19: An early review

  • neherlab (2020) neherlab (2020) covid19_scenarios. github.com/neherlab/covid19_scenarios, [Online; accessed 09-May-2020]
  • Nistor et al. (2013) Nistor A, Song L, Marinov D, Lu S (2013) Toddler: Detecting performance problems via similar memory-access patterns. In: Proceedings of the 2013 International Conference on Software Engineering, IEEE Press, ICSE ’13, p 562–571
  • nthopinion (2020) nthopinion (2020) nthopinion/covid19. https://github.com/nthopinion/covid19, [Online; accessed 09-May-2020]
  • Olivo et al. (2015) Olivo O, Dillig I, Lin C (2015) Static detection of asymptotic performance bugs in collection traversals. In: Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, Association for Computing Machinery, New York, NY, USA, PLDI ’15, p 369–378, DOI 10.1145/2737924.2737966, URL https://doi.org/10.1145/2737924.2737966
  • OpenMined (2020) OpenMined (2020) covid-alert. github.com/OpenMined/covid-alert, [Online; accessed 09-May-2020]
  • Pandey et al. (2020) Pandey R, Gautam V, Bhagat K, Sethi T (2020) A machine learning application for raising wash awareness in the times of covid-19 pandemic. ArXiv abs/2003.07074
  • Pascarella et al. (2018) Pascarella L, Palomba F, Di Penta M, Bacchelli A (2018) How is video game development different from software development in open source? In: Proceedings of the 15th International Conference on Mining Software Repositories, ACM, New York, NY, USA, MSR ’18, pp 392–402, DOI 10.1145/3196398.3196418, URL http://doi.acm.org/10.1145/3196398.3196418
  • Pei et al. (2017) Pei K, Cao Y, Yang J, Jana S (2017) Deepxplore: Automated whitebox testing of deep learning systems. In: Proceedings of the 26th Symposium on Operating Systems Principles, Association for Computing Machinery, New York, NY, USA, SOSP ’17, p 1–18, DOI 10.1145/3132747.3132785, URL https://doi.org/10.1145/3132747.3132785
  • popsolutions (2020) popsolutions (2020) popsolutions/openventilator. https://github.com/popsolutions/openventilator, [Online; accessed 09-May-2020]
  • Prana et al. (2019) Prana GA, Treude C, Thung F, Atapattu T, Lo D (2019) Categorizing the content of github readme files. Empirical Softw Engg 24(3):1296–1327, DOI 10.1007/s10664-018-9660-3, URL https://doi.org/10.1007/s10664-018-9660-3
  • Pulido et al. (2020) Pulido CM, Villarejo-Carballido B, Redondo-Sama G, Gómez A (2020) Covid-19 infodemic: More retweets for science-based information on coronavirus than for false information. International Sociology p 0268580920914755
  • Rahman et al. (2020) Rahman A, Farhana E, Parnin C, Williams L (2020) Gang of eight: A defect taxonomy for infrastructure as code scripts. In: Proceedings of the 42nd International Conference on Software Engineering, ICSE ’20, to appear
  • Randhawa et al. (2020) Randhawa GS, Soltysiak MPM, El Roz H, de Souza CPE, Hill KA, Kari L (2020) Machine learning using intrinsic genomic signatures for rapid classification of novel pathogens: Covid-19 case study. PLOS ONE 15(4):1–24, DOI 10.1371/journal.pone.0232391, URL https://doi.org/10.1371/journal.pone.0232391
  • Rao and Vazquez (2020) Rao ASS, Vazquez JA (2020) Identification of covid-19 can be quicker through artificial intelligence framework using a mobile phone-based survey in the populations when cities/towns are under quarantine. Infection Control and Hospital Epidemiology p 1–18, DOI 10.1017/ice.2020.61
  • Ray et al. (2014) Ray B, Posnett D, Filkov V, Devanbu P (2014) A large scale study of programming languages and code quality in github. In: Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, ACM, New York, NY, USA, FSE 2014, pp 155–165, DOI 10.1145/2635868.2635922, URL http://doi.acm.org/10.1145/2635868.2635922
  • reustle (2020) reustle (2020) Fix prefecture sorting. reustle/covid19japan/issues/15, [Online; accessed 10-May-2020]
  • Rourke et al. (2020) Rourke M, Eccleston-Turner M, Phelan A, Gostin L (2020) Policy opportunities to enhance sharing for pandemic research. Science 368(6492):716–718
  • Saldana (2015) Saldana J (2015) The coding manual for qualitative researchers. Sage
  • Sanche et al. (2020) Sanche S, Lin YT, Xu C, Romero-Severson E, Hengartner N, Ke R (2020) High contagiousness and rapid spread of severe acute respiratory syndrome coronavirus 2. Journal of Emerging Infectious Disease 26, DOI https://doi.org/10.3201/eid2607.200282, URL https://doi.org/10.3201/eid2607.200282
  • Santosh (2020) Santosh K (2020) Ai-driven tools for coronavirus outbreak: Need of active learning and cross-population train/test models on multitudinal/multimodal data. Journal of Medical Systems 44, DOI 10.1007/s10916-020-01562-1
  • SinghRajenM (2020) SinghRajenM (2020) Rajasthan district names are wrong. https://github.com/covid19india/covid19india-react/issues/321, [Online; accessed 10-May-2020]
  • soroushchehresa (2020) soroushchehresa (2020) soroushchehresa/awesome-coronavirus. github.com/soroushchehresa/awesome-coronavirus, [Online; accessed 16-May-2020]
  • Subratappt (2020) Subratappt (2020) Cluster animation slowing down the browser. it also takes much time. https://github.com/covid19india/covid19india-react/issues/497, [Online; accessed 10-May-2020]
  • Tamm (2020) Tamm MV (2020) Covid-19 in moscow: prognoses and scenarios. FARMAKOEKONOMIKA Modern Pharmacoeconomic and Pharmacoepidemiology 13(1):43–51, DOI 10.17749/2070-4909.2020.13.1.43-51
  • Thung et al. (2012) Thung F, Wang S, Lo D, Jiang L (2012) An empirical study of bugs in machine learning systems. In: 2012 IEEE 23rd International Symposium on Software Reliability Engineering, pp 271–280, DOI 10.1109/ISSRE.2012.22
  • Tian et al. (2018)

    Tian Y, Pei K, Jana S, Ray B (2018) Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th International Conference on Software Engineering, Association for Computing Machinery, New York, NY, USA, ICSE ’18, p 303–314, DOI 

    10.1145/3180155.3180220, URL https://doi.org/10.1145/3180155.3180220
  • Timoeller (2020) Timoeller (2020) Cdc children scraper is outdated. https://github.com/deepset-ai/COVID-QA/issues/43, [Online; accessed 10-May-2020]
  • Tom Simonite (2020) Tom Simonite (2020) Software that reads ct lung scans had been used primarily to detect cancer. now it’s retooled to look for signs of pneumonia caused by coronavirus. https://www.wired.com/story/chinese-hospitals-deploy-ai-help-diagnose-covid-19/, [Online; accessed 08-May-2020]
  • vaclavpavlicek (2020) vaclavpavlicek (2020) Missing postgis. Applifting/pomuzeme.si/issues/164, [Online; accessed 10-May-2020]
  • Van Bavel et al. (2020) Van Bavel JJ, Baicker K, Boggio PS, Capraro V, Cichocka A, Cikara M, Crockett MJ, Crum AJ, Douglas KM, Druckman JN, et al. (2020) Using social and behavioural science to support covid-19 pandemic response. Nature Human Behaviour pp 1–12
  • Wan et al. (2017) Wan Z, Lo D, Xia X, Cai L (2017) Bug characteristics in blockchain systems: A large-scale empirical study. In: 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), pp 413–424, DOI 10.1109/MSR.2017.59
  • Wang et al. (2020) Wang C, Li W, Drabek D, Okba NM, van Haperen R, Osterhaus AD, van Kuppeveld FJ, Haagmans BL, Grosveld F, Bosch BJ (2020) A human monoclonal antibody blocking sars-cov-2 infection. Nature Communications 11(1):1–6
  • WHO (2020) WHO (2020) Global research on coronavirus disease (covid-19). https://www.who.int/emergencies/diseases/novel-coronavirus-2019/global-research-on-novel-coronavirus-2019-ncov, [Online; accessed 09-May-2020]
  • Why Hunger (2020) Why Hunger (2020) Why hunger. https://whyhunger.org/map.php, [Online; accessed 08-May-2020]
  • Will (2020) Will CM (2020) ‘and breathe…’? the sociology of health and illness in covid-19 time. Sociology of Health & Illness
  • Wong et al. (2020) Wong MC, Cregeen SJJ, Ajami NJ, Petrosino JF (2020) Evidence of recombination in coronaviruses implicating pangolin origins of ncov-2019. Biorxiv
  • Yang and Wang (2020) Yang CY, Wang J (2020) A mathematical model for the novel coronavirus epidemic in wuhan, china. Mathematical Biosciences and Engineering 17(3):2708–2724, DOI 10.3934/mbe.2020148
  • Yang et al. (2020) Yang GZ, J Nelson B, Murphy RR, Choset H, Christensen H, H Collins S, Dario P, Goldberg K, Ikuta K, Jacobstein N, Kragic D, Taylor RH, McNutt M (2020) Combating covid-19—the role of robotics in managing public health and infectious diseases. Science Robotics 5(40), DOI 10.1126/scirobotics.abb5589, URL https://robotics.sciencemag.org/content/5/40/eabb5589, https://robotics.sciencemag.org/content/5/40/eabb5589.full.pdf
  • zbraniecki (2020) zbraniecki (2020) Data has a gap between 2020-3-11 and 2020-3-24. https://github.com/covidatlas/coronadatascraper/issues/375, [Online; accessed 10-May-2020]
  • Zhang et al. (2019) Zhang T, Chen J, Luo X, Li T (2019) Bug reports for desktop software and mobile apps in github: What’s the difference? IEEE Software 36(1):63–71
  • Zhang et al. (2020)

    Zhang T, Wu Q, Zhang Z (2020) Probable pangolin origin of sars-cov-2 associated with the covid-19 outbreak. Current Biology

  • Zhou et al. (2020) Zhou P, Yang XL, Wang XG, Hu B, Zhang L, Zhang W, Si HR, Zhu Y, Li B, Huang CL, et al. (2020) A pneumonia outbreak associated with a new coronavirus of probable bat origin. nature 579(7798):270–273