The Impact of a Major Security Event on an Open Source Project: The Case of OpenSSL

05/28/2020
by   James Walden, et al.
0

Context: The Heartbleed vulnerability brought OpenSSL to international attention in 2014. The almost moribund project was a key security component in public web servers and over a billion mobile devices. This vulnerability led to new investments in OpenSSL. Objective: The goal of this study is to determine how the Heartbleed vulnerability changed the software evolution of OpenSSL. We study changes in vulnerabilities, code quality, project activity, and software engineering practices. Method: We use a mixed methods approach, collecting multiple types of quantitative data and qualitative data from web sites and an interview with a developer who worked on post-Heartbleed changes. We use regression discontinuity analysis to determine changes in levels and slopes of code and project activity metrics resulting from Heartbleed. Results: The OpenSSL project made tremendous improvements to code quality and security after Heartbleed. By the end of 2016, the number of commits per month had tripled, 91 vulnerabilities were found and fixed, code complexity decreased significantly, and OpenSSL obtained a CII best practices badge, certifying its use of good open source development practices. Conclusions: The OpenSSL project provides a model of how an open source project can adapt and improve after a security event. The evolution of OpenSSL shows that the number of known vulnerabilities is not a useful indicator of project security. A small number of vulnerabilities may simply indicate that a project does not expend much effort to finding vulnerabilities. This study suggests that project activity and CII badge best practices may be better indicators of code quality and security than vulnerability counts.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

06/24/2020

Exploring the Security Awareness of the Python and JavaScript Open Source Communities

Software security is undoubtedly a major concern in today's software eng...
03/19/2021

An Exploratory Study of Project Activity Changepoints in Open Source Software Evolution

To explore the prevalence of abrupt changes (changepoints) in open sourc...
02/07/2019

A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software

Advancing our understanding of software vulnerabilities, automating thei...
12/13/2021

A Novel Model for Vulnerability Analysis through Enhanced Directed Graphs and Quantitative Metrics

Industrial components are of high importance because they control critic...
03/14/2021

On the combination of static analysis for software security assessment – a case study of an open-source e-government project

Static Application Security Testing (SAST) is a popular quality assuranc...
07/24/2020

A Case Study on Software Vulnerability Coordination

Context: Coordination is a fundamental tenet of software engineering. Co...
09/03/2020

The Sound of Silence: Mining Security Vulnerabilities from Secret Integration Channels in Open-Source Projects

Public development processes are a key characteristic of open source pro...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

The OpenSSL project came to the attention of the world with the Heartbleed vulnerability on April 7, 2014. The open source cryptographic library was widely used to secure communications using the Transport Layer Security (TLS) protocol. An estimated 24-55% of popular web sites using TLS were exposed to Heartbleed attacks 

(Durumeric et al., 2014), which allowed attackers to remotely access private keys and passwords. The number of client devices impacted was much larger, as OpenSSL was the default cryptographic library for Android devices, and more than one billion Android devices shipped in 2014 (Reisinger, 2015).

At the time of Heartbleed, the OpenSSL project was largely inactive, except for an ever-growing number of unaddressed issues. The project had no full time developers, and the OpenSSL Software Foundation received about $2,000 per year in donations (Marquess, 2014). There were no policies for handling issues or vulnerabilities. OpenSSL had no release plan, and the project was still supporting version 0.9.8, which had been released in 2005. Project source code was complex and difficult to understand and maintain, while the project team was small with static membership (Hudson and Salz, 2016).

The goal of this case study is to understand how the OpenSSL project responded to the Heartbleed vulnerability. We build this understanding with a mixed methods approach. We collect qualitative data from web sites and a developer interview, and we use quantitative data in the form of software metrics, including both project activity and code metrics. Our choice of metrics was guided in part by Lehman’s laws of software evolution (Lehman, 1996). In particular, we analyze the first (continuing change) law using project activity metrics, the second law (increasing complexity), using code complexity metrics, and the sixth law (continuing growth) using code size metrics. We use regression discontinuity analysis to determine changes in levels and slopes of software metrics resulting from Heartbleed.

We believe this is the first study of the impact of a major security incident on software evolution. Our primary contribution is an understanding of how a project can recover from a major security incident. We learn that the number of reported vulnerabilities is a poor indicator of security. Project activity and software engineering practices required by the CII best practices badge 111https://bestpractices.coreinfrastructure.org/ may be better indicators of project security. Finally, we provide a replication package that includes both the data used in this paper and the code used to collect and analyze the data (Walden, 2020). Code to build the tables and figures in this paper is also included.

Version Release Date Last Release Last Release Date Release Count Lifespan End of Life Code Size Vulnerabilities
0.9.1 1998-12-21 0.9.1c 1998-12-23 1 2 days 39,049
0.9.2 1999-03-22 0.9.2b 1999-04-06 1 15 days 40,112
0.9.3 1999-05-25 0.9.3a 1999-05-27 2 2 days 41,222
0.9.4 1999-08-09 0.9.4 1999-08-09 1 0 days 41,973
0.9.5 2000-02-28 0.9.5a 2000-04-01 2 33 days 44,594
0.9.6 2000-09-24 0.9.6m 2004-03-17 14 1270 days 47,025 10
0.9.7 2002-12-31 0.9.7m 2007-02-23 14 1515 days 60,442 9
0.9.8 2005-07-05 0.9.8zh 2015-12-03 35 3803 days 73,837 103
1.0.0 2010-03-29 1.0.0t 2015-12-03 21 2075 days 91,765 92
1.0.1 2012-03-14 1.0.1u 2016-09-22 22 1653 days 2016-12-31 100,415 127
1.0.2 2015-01-22 1.0.2u 2019-12-20 22 1793 days 2019-12-31 107,862 94
1.1.0 2016-08-25 1.1.0l 2019-09-10 13 1111 days 2019-09-11 100,957 30
1.1.1 2018-09-11 1.1.1d 2019-09-10 5 364 days 2023-09-11 116,486 8
Table 1. OpenSSL Versions

2. Context

The first version of OpenSSL was released on December 23, 1998. It was based on a fork of the SSLeay project. The project name comes from the Secure Sockets Layer (SSL) protocol, which has been deprecated in favor of Transport Layer Security. OpenSSL consists of two libraries, which provide support for TLS and general cryptographic algorithms respectively, and a command line tool, openssl, which can be used for encryption, decryption, and certificate generation.

Table 1 summarizes the major versions of OpenSSL, including lifespan, minor releases, code size (number of statements), and number of vulnerabilities reported per release. The version number format is three digits followed by an optional letter, indicating minor releases. Prior to Heartbleed, OpenSSL had no release policy. Therefore, earlier versions have no end of life date. Lifespan is the period between the first and last release of a numerical version.

Vulnerability reporting started in 2002, so earlier versions have no vulnerabilities associated with them. Vulnerabilities often affect multiple versions of OpenSSL, so the total number of vulnerabilities is smaller than the sum of the vulnerabilities in the table.

Heartbleed focused intense scrutiny on OpenSSL. Two approaches to OpenSSL were advocated: replacement or repair. Two forks were created shortly after Heartbleed as potential replacements. Google’s fork, BoringSSL (Langley, 2015), focused on supporting Android and Chrome. The OpenBSD project’s fork was called LibreSSL (Beck, 2014) and focused on improving security and code quality.

The repair approach was supported by the Core Infrastructure Initiative (CII), which was started by the Linux Foundation as a response to Heartbleed. The purpose of the CII was to fund and support open-source projects that are critical to the functioning of the Internet (Foundation, 2014). OpenSSL was among the first projects funded. CII funded two full-time developers and a code audit. An additional two developers were funded by donations (Hudson and Salz, 2016). CII supported the project’s first face-to-face meeting in late 2014, during which project members drafted major policies, including a release strategy, coding style guide, and security policy.

3. Related Work

Lehman’s laws (Lehman, 1996) provide a framework for understanding software evolution that has been widely studied. Previous studies have operationalized these laws terms of software metrics in multiple ways (Neamtiu et al., 2013; Gonzalez-Barahona et al., 2014; Li et al., 2017). We examine the first, second, and sixth laws in this work. Multiple studies using a variety of metrics have found that these three laws hold for most but not all open source projects studied (Godfrey and Tu, 2001; Paulson et al., 2004; Wu and Holt, 2004; Izurieta and Bieman, 2006; Neamtiu et al., 2013).

To study Lehman’s second law, we use code complexity metrics. McCabe developed his cyclomatic complexity metric as a quantitative measure of which software modules are difficult to maintain or test (McCabe, 1976). Halstead developed complexity metrics for similar purposes (Halstead et al., 1977). Midha et al. (Midha et al., 2010) found that the number of bugs in software increased with increasing cyclomatic complexity, while Gill and Kemerer (Gill and Kemerer, 1991) found that complexity density (the ratio of cyclomatic complexity to lines of code) was a useful predictor of maintenance productivity. Halstead’s and McCabe’s complexity metrics have been used as predictors in defect prediction (Menzies et al., 2002; Menzies et al., 2006) and vulnerability prediction (Shin et al., 2010) models.

We use code size metrics to study Lehman’s sixth law. Code size metrics have been used in multiple studies of Lehman’s laws (Gonzalez-Barahona et al., 2014; Li et al., 2017). Jimenez et al. describe characteristics of vulnerable files in OpenSSL using code size and code complexity metrics (Jimenez et al., 2016), but do not use them to study software evolution. We compute both code size and code complexity metrics using the cqmetrics tool that Spinellis used to study the evolution of the Unix operating system (Spinellis et al., 2016).

In addition to code size and complexity, we examine code style and language feature use. Following a consistent coding style may be an important aspect of readability and maintainability of software (Oracle, 1999; Smit et al., 2011). Programming languages like Java (Oracle, 1999) and Python (van Rossum et al., 2001) have style guides, while organizations like Google (Google, [n.d.]) publish style guides for a variety of languages. Smit et al. (Smit et al., 2011) found that in the absence of automated style checkers, the number of style violations grows in a linear relationship with code size.

While the idea that use of goto is harmful (Dijkstra, 1968) has been widely heard by programmers, an empirical study of GitHub projects suggested that use of goto for specific purposes like error handling was considered good practice by open source developers (Nagappan et al., 2015). On the other hand, use of C preprocessor conditionals for portability across different architectures has continued to receive blame for making code difficult to read and maintain (Spencer and Collyer, 1992; Miller, 2005).

While there are many studies of open source evolution, few studies examine the impact of an external change on software evolution. We use a regression discontinuity design (Reichardt, 2019; Jacob et al., 2012) (RDD) to assess the impact of Heartbleed on the evolution of OpenSSL. This methodology has recently begun to be used in empirical software engineering analyses of time series data. Zhao et al. used RDD to evaluate the impact of adopting continuous integration on other software development practices (Zhao et al., 2017), while Trockman et al. used RDD to test whether repository badges were reliable signals of software quality (Trockman et al., 2018). Zimmermann and Artís evaluated the impact of switching bug trackers on a single project with a RDD model (Zimmermann and Artís, 2019).

Durumeric et al. measured the reaction to Heartbleed, finding that Alexa Top 100 sites patched within 48 hours, while less popular sites took longer to deploy patches (Durumeric et al., 2014). Kupsch and Miller discuss the difficulty software security tools have in finding vulnerabilities like Heartbleed (Kupsch and Miller, 2014). Wheeler describes software engineering practices and technologies that could find vulnerabilities like Heartbleed, including simplifying the code, fuzzing with address checking, and thorough security-focused testing (Wheeler, 2014). We will examine how OpenSSL adopted some of those approaches below.

4. Research Questions

We begin our study by examining the security of OpenSSL. We examine the number and severity of vulnerability reports over time to understand how the Heartbleed vulnerability affected vulnerability reporting. Our first research question is:

[boxsep=0mm] Research Question 1: How did the number and severity of reported vulnerabilities change after Heartbleed?

We expect to see an increased number of vulnerability reports after Heartbleed, as more effort was devoted to finding security issues. Such an increase is an indicator of improving project security as vulnerabilities are found and remediated.

OpenSSL has been under development for over two decades, a substantial span of time in which open source development has greatly changed. Lehman’s sixth law of software evolution states that software continually grows in functionality to maintain user satisfaction (Lehman, 1996). Adding new functionality typically requires adding more code to a project. Therefore, we assess the size of OpenSSL to see if this law holds after a major security incident. It is worth noting that both of the post-Heartbleed forks of OpenSSL began development by removing older cryptographic algorithms and support for a variety of computing environments (Langley, 2015; Beck, 2014). Our second research question is:

[boxsep=0mm] Research Question 2: How did OpenSSL change in size after Heartbleed?

The quality of OpenSSL code before Heartbleed was perceived as poor (Beck, 2014). It was difficult to read and maintain, discouraging new contributors from working on the project (Hudson and Salz, 2016). Therefore, we want to measure code quality using metrics that impact the readability and maintainability of code, such as code complexity metrics.

Lehman’s second law of software evolution states that program complexity increases over time unless work is done to prevent that (Lehman, 1996). With two decades of history, there has been ample time for OpenSSL’s code to increase in complexity. While some complexity is necessary for cryptographic code, unnecessary complexity can accumulate over time. The complexity of OpenSSL code was implicated as one of the causes of the Debian project accidentally breaking the OpenSSL pseudo-random number generator in 2006 (Cox, 2008).

[boxsep=0mm] Research Question 3: How did the complexity of OpenSSL source code change after Heartbleed?

As code style and the use of certain programming language features can affect the readability and maintainability of software, we also study these characteristics of the OpenSSL code base. We examine the consistency of stylistic choices like bracket placement and indentation, and we study the use of certain language features in our study of coding style, such as the C preprocessor and the goto statement.

[boxsep=0mm] Research Question 4: How did the coding style of OpenSSL source code change after Heartbleed?

Lehman’s first law focuses on continuing change of a software project. One of the major problems with OpenSSL at the time of Heartbleed was insufficient developer activity to address technical debt. The project team was also small and included no full time developers, while team membership was static (Hudson and Salz, 2016). The number of contributions from outside developers was small. There were no guidelines for contributing to the project, while the source code was difficult to understand and maintain, discouraging new contributors.

The time for the developers to respond to issues was high, and most issues were not addressed. The number of open, unaddressed issues had grown steadily, reaching almost 1500 at the time of Heartbleed (Salz, 2016).

To measure the change in project activity, we count the number of commits before and after Heartbleed and measure the number of commits per month. We also want to determine whether the OpenSSL project was able to grow its development team and accept a substantial number of outside contributions in the wake of Heartbleed, so we measure the number of authors.

[boxsep=0mm] Research Question 5: How did the number of commits and the number of authors change after Heartbleed?

We also study the software engineering practices that led to changes in project activity, code quality, and security. As the only requirements and design documents available for OpenSSL are for the forthcoming 3.0.0 release, we focus on implementation and testing activities that have been identified as best practices by the CII Best Practices Badge project (Wheeler, [n.d.]).

[boxsep=0mm] Research Question 6: How did software engineering practices change after Heartbleed?

Finally, we want to determine if the changes in OpenSSL were sustained well after the discovery of the Heartbleed vulnerability. To address this question, we will not only examine trends in our code and activity metrics through the end of 2019 but we will also examine external assessments of the OpenSSL project.

[boxsep=0mm] Research Question 7: Are the changes in OpenSSL’s security, source code, and software engineering practices sustained five years after Heartbleed?

5. Data

We collected data on the OpenSSL project from a variety of sources, including the project web site, vulnerabilities list, and GitHub repository. We also interviewed one OpenSSL developer.

Vulnerabilities: Vulnerability data was collected from the OpenSSL vulnerabilities list 222https://www.openssl.org/news/vulnerabilities.html. We count vulnerabilities using unique Common Vulnerabilities and Exposures (CVE) identifiers.

Code Metrics: The source code of OpenSSL was obtained from the project’s GitHub repository 333https://github.com/openssl/openssl. We used the cqmetrics package 444https://github.com/dspinellis/cqmetrics to compute code metrics, including size, complexity, language feature use, and style metrics. This open source tool was chosen in part because of its prior use in computing metrics on many versions of Unix released over several decades (Spinellis et al., 2016). For our monthly time series data, we compute code metrics on the first commit made during a month.

Project Activity: Metrics on project activity, such as the number of authors and commits were obtained from the project’s GitHub repository using PyDriller (Spadini et al., 2018). We compute authors per month as the number of unique author names in commits made during a month.

Software Engineering Practices: We reviewed the OpenSSL web site, mailing lists, and GitHub repository for information on changes in software engineering practices. We collected data on unit testing from the project GitHub repository and from test coverage data reported via coveralls.io.

Interviews: Our interview requests received a single response from an OpenSSL team member who worked on the project immediately after Heartbleed. We interviewed Rich Salz of Akamai, who joined the project a couple of months after Heartbleed. He helped change software engineering processes, including moving to a GitHub pull request work flow, transitioning issue tracking to GitHub, and adopting continuous integration.

6. Methods

We use data visualization and statistical modeling to discover changes in code metrics and project activity. In particular, we use a regression discontinuity design 

(Reichardt, 2019; Jacob et al., 2012) approach to analyze time series of code and project activity metrics. RDD allows us to determine whether changes in metrics occurred and to measure the effect size of those changes.

Regression discontinuity design is a rigorous quasi-experimental approach for analyzing the casual effect of an intervention. It is based on the idea that in the absence of the intervention, the observed trend before the intervention would continue afterwards. In such a design, observations are assigned to an intervention condition based on a cutoff score. In time series analysis, the cutoff is a date. The cutoff date can be determined by visually inspecting the time series for discontinuities or by the date of a specific event. The effect of the intervention is estimated as a discontinuity between the intervention groups before and after the cutoff date. When using a linear regression model, the intervention effect can include both a change in level and a change in slope at the cutoff.

Regression discontinuity can be performed with a global or local regression approach. In global regression approaches, the entire data set is used to fit the model. In local regression models, the model is fitted using a subset of the time series, with some data on each side on the cutoff. This amount is called the bandwidth. RDD works better with an equal amount of data before and after the cutoff. While the global approach offers greater precision, it includes data far from the cutoff, which may be influenced by trends other than the intervention being studied. Given the extensive history of OpenSSL before Heartbleed, we choose a local regression approach.

We use the following regression discontinuity model equation to estimate changes in level and trend in code metrics after the beginning of major post-Heartbleed work:

where D is a function that represents the discontinuity

In the equation above, is the cutoff date, when code metrics showed substantial changes. The coefficient represents the level of the regression line before the cutoff, while represents the change in level after the cutoff. The sum is the level of the regression line after the discontinuity.

The response variable

is the value of a particular code metric at time . The variable represents time in months. The coefficient is the slope of the regression line before the cutoff, while is the change in slope after the cutoff. The sum is the slope of the regression line after the cutoff. The variable represents the error at time .

In our analysis of code metrics, we build RDD models with a cutoff of February 2015. This cutoff was initially identified visually. It is clear to see in the plots of nesting depth and style inconsistency in Figure 2, which approach a step function that transitions from one interval to the other in this month. While Heartbleed was the initial impetus for changing OpenSSL, large changes to the code base began in February 2015. Investigation of the project web site and mailing lists reveal that the time between Heartbleed and our cutoff date was spent building the team, developing policies, and planning how to change the code base.

Confirming our choice of cutoff, the OpenSSL project blog published an article about reformatting the entire code base to meet the project’s new coding style guidelines in February 2015 (Caswell, 2015). Additional code cleanup was performed in the following months, described in another blog entry published in July 2015 (Salz, 2015). We choose a bandwidth of 25 months on each side of the cutoff date, so that we have a sufficient number of data points for our model without including data points that are so far from the cutoff that they are influenced by factors other than Heartbleed.

7. Results

7.1. Vulnerabilities

Our first research question focuses on security vulnerabilities. There have been 177 vulnerabilities reported in OpenSSL through the end of 2019. Figure 1 shows the trend of the annual number of vulnerabilities. The date of Heartbleed is indicated by a dashed line. While 66 (37.3%) vulnerabilities were reported in the approximately sixteen years before Heartbleed, 110 (62.1%) vulnerabilities were reported in the five years after Heartbleed.

We can see three eras of vulnerability reporting for the OpenSSL project in Figure 1: the pre-Heartbleed era, the high vulnerability reporting era from 2014 to 2016, and the modern era from 2017 to the present. Vulnerability statistics for the three eras are detailed in Table 2.

Figure 1. OpenSSL Vulnerabilities Reported by Year

Time series graph of OpenSSL vulnerabilities.

The pre-Heartbleed era has the lowest number of vulnerabilities per year but with slightly higher severity scores (CVSSv2) than the other two eras. There is a dramatic increase in the number of vulnerabilities reported per year, from 4 to 30.33, in the high vulnerability era, with severities remaining on par with the pre-Heartbleed era. A majority (51.4%) of all vulnerabilities were reported in the three year period between 2014 and 2016. Only five of the 91 vulnerabilities found in the 2014-2016 time period were found in code written after Heartbleed, so this era produced a substantial improvement in the security of OpenSSL.

In the modern era starting in 2017, the number of vulnerabilities reported per-year is almost twice as high as the annual count before Heartbleed, but the mean severity of vulnerabilities has declined, as has variability around that mean. However, the pre-Heartbleed mean number of vulnerabilities per year is computed across many more years than the mean for the modern era. If we restrict our view to the four years before Heartbleed, the mean number of vulnerabilities reported per-year is 7.0, which is quite close the modern era mean of 7.33. The mean CVSSv2 score for the four pre-Heartbleed years is 5.45 and the standard deviation is 0.84, so we still observe a decline in vulnerability severity between the late pre-Heartbleed era and the modern era.

Pre-Heartbleed 2014-2016 2017-2019
Vulnerability Count 64 91 22
Vulnerability Percent 36.2% 51.4% 12.4%
Vulnerabilities per Year 4.00 30.33 7.33
Average CVSS2 5.63 5.25 4.18
StdDev CVSS2 1.11 1.11 0.39
Table 2. OpenSSL Vulnerabilities

7.2. Code Size

(a) Code Size Metrics
(b) Code Complexity Metrics
(c) Code Style Metrics
(d) Project Activity Metrics
Figure 2. Regression Discontinuity Models

In this and the following three sections, we analyze monthly time series of the software metrics shown in Figure 2. We compute monthly code metrics by checking out the version of the code available from the first commit made during the month and running cqmetrics on that version. Project activity metrics are computed using all commits during the month.

To study RQ2, we analyze multiple code size metrics. Code size, both in terms of the number of files and number of statements, starts dropping shortly after Heartbleed, as can be seen in Figure 1(a). However, much larger changes begin during the code cleanup in February 2015, which we chose as the cutoff date for our RDD models. Table 3 shows a significant decrease of 12,260 statements (10.8% of pre-cutoff size) after code cleanup began in February 2015. We see a similar decrease in the number of files. Both models were good fits as measured using the adjusted metric, which is given in parentheses after the model name in Table 3.

To better understand these size changes, we examined 126 commits made in the 25 months after Heartbleed, in which the only file changes were deletions. A total of 315 C source and header files were deleted in those commits. We found multiple mentions of obsolete or old files, including support for obsolete platforms, old insecure protocols like SSLv2, and unmaintained demo code in developer commit messages of these commits. These messages point to the removal of files being an improvement of code quality, especially since support for old protocols exposed OpenSSL to protocol downgrade attacks like DROWN (Aviram et al., 2016).

In Figure 1(a), we can see code size as measured in terms of the number of files and statements decreasing from Heartbleed until about the release of version 1.1.0 in August 2016. Version 1.1.0 was the only major release of OpenSSL to have a smaller size at the time of release than its predecessors. Code sizes measured by number of statements for each version are provided in Table 1.

Statements (0.88) Files (0.92) Functions (0.79) Mean Function Length (0.99)
(l3ptr3pt)2-3 (l3ptr3pt)4-5 (l3ptr3pt)6-7 (l3ptr3pt)8-9 Estimate Std Error Estimate Std Error Estimate Std Error Estimate Std Error
Pre-Intervention Level 113340.58*** 856.579 940.79*** 6.893 7876.18*** 50.579 14.389*** 0.034
Change in Level -12259.623*** 1166.553 -91.328*** 9.387 -737.394*** 68.882 -0.28*** 0.046
Pre-Intervention Slope 77.288 57.620 -1.293** 0.464 4.697 3.402 0.001 0.002
Change in Slope -72.429 79.191 1.093 0.637 37.12*** 4.676 -0.072*** 0.003
RDD models for code size metrics, showing changes in each metric’s mean and slope at February 2015. Each model’s
metric is shown in parentheses after metric name. Statistical significance is indicated by *** p ¡ 0.001, ** p ¡ 0.01, * p ¡ 0.05
Table 3. Code Size Metrics Regression Discontinuity Models

After February 2015, the number of functions had decreased significantly by 737 (9.3%), but the number of functions being added per month was almost eight times the pre-cutoff number. To understand how the number of functions was increasing rapidly while the numbers of statements and files were declining, we examine mean function length. Function length transitions from an essentially flat pre-Heartbleed trend to a decreasing trend (-0.072 lines/month) after Heartbleed. This evidence suggests that the OpenSSL project began to factor their code into smaller functions after Heartbleed. Many programmers find that smaller functions are easier to understand (Martin, 2009).

7.3. Code Complexity

Complex code is difficult to understand and maintain, so unnecessary complexity should be avoided. RQ3 focuses on changes to OpenSSL’s code complexity. We examine three types of code complexity metrics: nesting complexity, cyclomatic complexity, and Halstead complexity. Nesting complexity is the number of levels of nesting in source code.

Max Nesting (1.00) Mean Nesting (1.00) Cyclomatic (0.85) Halstead (0.87)
(l3ptr3pt)2-3 (l3ptr3pt)4-5 (l3ptr3pt)6-7 (l3ptr3pt)8-9 Estimate Std Error Estimate Std Error Estimate Std Error Estimate Std Error
Pre-Intervention Level 13*** 0 1.294*** 0.002 8.486*** 0.096 904.286*** 10.425
Change in Level -6*** 0 -0.508*** 0.002 -0.62*** 0.131 -60.071*** 14.197
Pre-Intervention Slope 0* 0 0*** 0.000 0.005 0.006 0.717 0.701
Change in Slope 0 0 -0.002*** 0.000 -0.044*** 0.009 -6.032*** 0.964
RDD models for complexity metrics, showing changes in each metric’s mean and slope at February 2015. Each model’s
metric is shown in parentheses after metric name. Statistical significance is indicated by *** p ¡ 0.001, ** p ¡ 0.01, * p ¡ 0.05
Table 4. Code Complexity Metrics Regression Discontinuity Models

We study mean and maximum values of nesting complexity and mean values of cyclomatic and Halstead complexity. Mean complexity metrics are computed as a mean of the per-file mean values of these metrics for each month in the dataset. We construct four RDD models, which are summarized in Table 4. Figure 1(b) shows both data and model fit for these four metrics.

The maximum depth of nesting in OpenSSL source files dropped suddenly from 13 to 7 in February 2015. This change affected 94 files, when the OpenSSL project reformatted its code base (Caswell, 2015). The RDD model of mean nesting depth shows a drop of approximately half a level from 1.29 to 0.79 in February 2015.

Mean cyclomatic complexity dropped by 7.3% after February 2015. The trend of cyclomatic complexity evolution changes from a small increasing slope before intervention to a larger decreasing slope after the cutoff. The evolution of Halstead complexity follows the same pattern, with a drop of 6.6% and a similar change in slope. These changes suggest that the post-Heartbleed code cleanup was effective at reducing the complexity of OpenSSL code and thus improving readability and maintainability.

7.4. Coding Style

RQ4 focuses on coding style. Poor coding style can make code difficult to understand and maintain, increasing the likelihood of programmer errors and decreasing the likelihood of attracting contributors to the project. Prior to the Heartbleed vulnerability, the OpenSSL project had no coding style guide, and different sections of code used a range of different styles (Caswell, 2015). In January 2015, the project published a coding style guide (Team, 2015). In early February 2015, project members reformatted the source code of the versions supported at the time (0.9.8 through 1.0.2) to ensure style consistency throughout the project.

Style Inconsistency (1.00) Goto Density (0.86) Preprocessor (0.98) Conditional (0.98)
(l3ptr3pt)2-3 (l3ptr3pt)4-5 (l3ptr3pt)6-7 (l3ptr3pt)8-9 Estimate Std Error Estimate Std Error Estimate Std Error Estimate Std Error
Pre-Intervention Level 57.045*** 0.276 0.07*** 0 0.168*** 0.001 0.014*** 0
Change in Level -52.957*** 0.377 0.002*** 0 -0.016*** 0.001 -0.003*** 0
Pre-Intervention Slope -0.09*** 0.019 0 0 0*** 0.000 0 0
Change in Slope 0.103*** 0.026 0 0 0*** 0.000 0 0
RDD models show changes in style and language feature metric means and slopes at February 2015. Each model’s metric
is shown in parentheses after metric name. Statistical significance is indicated by *** p ¡ 0.001, ** p ¡ 0.01, * p ¡ 0.05
Table 5. Code Style and Language Feature Regression Discontinuity Models

The cqmetrics tool computes a coding style inconsistency metric based on style rules selected from sources such as the Google, FreeBSD, and GNU coding style documents (Spinellis et al., 2016). For each way to format a particular C construct, such as placing a space after the while keyword, cqmetrics computes the sum of how many times the formatting rule is followed, , and the sum of how many times the rule is not followed, . The style inconsistency is the ratio of the smaller of two sums with the total times the rule could be applied in either way.

This style inconsistency metric shows a dramatic drop immediately after the reformatting, from 57.045 to 4.088, as can be seen in Figure 1(c). The slope prior to reformatting was slightly downwards, starting in mid-2014 after Heartbleed, while the slope after reformatting is slowly upwards, showing a gradual increase in deviations from the coding style as new contributions are made.

As part of coding style, we examine the use of both goto and the C preprocessor. The density of goto statements does not change substantially after the code reformatting and cleanup in 2015. However, the density of preprocessor statements decreases by about 10% from 0.168 to 0.152 at the same point, with potentially problematic conditional preprocessor statements decreasing 23% from 0.014 to 0.011.

The substantial improvements in coding style consistency and reduction in C preprocessor use described above suggest that the OpenSSL project focused on improving the readability and maintainability of their code after Heartbleed.

7.5. Project Activity

RQ5 addresses project activity. The OpenSSL project has become much more active since the discovery of Heartbleed. Contributors made 13,238 git commits (52.7%) in the 68 months after Heartbleed. Only 11,905 commits (47.3%) were made in the 197 months before Heartbleed. The difference in the number of commit authors is even larger, with only 51 commit authors (8.8%) before Heartbleed and 554 authors (96.0%) after Heartbleed. Note that the percentages do not sum to 100%, as a small subset of authors were active both before and after Heartbleed.

The tenfold growth in the number of unique authors does not equate to a tenfold growth in the number of commits after Heartbleed. This can be explained in part by the substantial number of authors who contributed only a single commit. Only ten (19.6%) of the 51 pre-Heartbleed authors contributed just one commit. After Heartbleed, 331 (59.7%) of the 554 authors contributed only a single commit. The large growth in both total and single commit authors suggests that the OpenSSL project has become much more attractive to outside contributors after Heartbleed.

In addition to looking at total numbers of commits and authors, we perform a time series analysis of the number of commits and the number of authors per month. We compute the number of authors per month by counting the number of unique authors who appear in git commit author fields during a month. Authors are identified by name, not by e-mail address. We fit RDD models to both time series.

Project activity begins changing immediately after Heartbleed, so we use the month of Heartbleed (April 2014) as the cutoff date. Due to the high variance in both of these metrics after Heartbleed, we use a bandwidth of 50 months on each side of the Heartbleed month rather than 25 months used above. This high variance is reflected in the lower adjusted

values for these models compared to most of our previous models, as shown in Table 6. Figure 1(d) shows the model fit and data points. While project activity occurs in spurts, there are clear and substantial changes in activity after Heartbleed. The change in number of commits per month (122.44) is much larger than the number of monthly commits before Heartbleed (46.22). Similarly, the change in number of unique authors per month (13.65) is much larger than the number of monthly authors before Heartbleed (7.06).

Commits per Month (0.51) Unique Authors per Month (0.80)
(l3ptr3pt)2-3 (l3ptr3pt)4-5 Estimate Std Error Estimate Std Error
Pre-Intervention Level 46.22* 21.801 7.057*** 1.416
Change in Level 122.438*** 31.461 13.653*** 2.043
Pre-Intervention Slope -0.23 0.751 0.108* 0.049
Change in Slope 1.904 1.079 0.039 0.070
Models show changes in metric means and slopes at April 2014. Each model’s is shown in
parentheses after metric name. Significance indicated by *** p ¡ 0.001, ** p ¡ 0.01, * p ¡ 0.05
Table 6. Project Activity Regression Discontinuity Models

7.6. Software Engineering Practices

RQ6 focuses on OpenSSL’s software engineering practices. In June 2014, shortly after Heartbleed, the OpenSSL project published a project roadmap (Team, 2016), identifying current issues, objectives, and forthcoming features. Project issues included a backlog of bug reports, some of which had been open for years, incomplete and incorrect documentation, code complexity, inconsistent coding style, a lack of code review, and the absence of a release strategy and a security policy. The developer we interviewed indicated that the OpenSSL team did not use code metrics to direct their software engineering efforts.

The CII began funding OpenSSL in 2014, enabling the developer team to grow rapidly. By December 2014, team size had increased from two main developers to fifteen project members and four full time funded developers (Hudson and Salz, 2016). The project began formalizing decision making and published a vulnerability handling policy. Project development moved to GitHub. According to the developer we interviewed, the motivation for using GitHub was to increase transparency and attract more developers.

OpenSSL published its first release strategy in December 2014, establishing end of life dates and planning for future versions. Version 1.0.2 was created as a long term support release with a backwards compatible API, while version 1.1.0 was planned to improve design and code while breaking compatibility with earlier versions.

The project published bylaws in February 2017, describing project roles, including the OpenSSL Management Committee (OMC) and Committers (Team, 2017a). Starting in July 2014, the OpenSSL project required that code submissions be reviewed and approved by a core team member. The updated 2017 committer policy required that all code submissions to be reviewed and approved by at least two committers, one of whom must be an OMC member (Team, 2017b).

The CII’s Best Practices Badge project provides a guide to good open source development practices. The badge project was established soon after Heartbleed to provide a method for open source developers to certify that their projects follow best practices. Attainment of a badge requires meeting 66 criteria in six categories: basics, change control, reporting, quality, security, and analysis (Wheeler, [n.d.]).

Prior to the changes made in response to Heartbleed, OpenSSL had completed 62% of badge requirements. The OpenSSL project attained its CII badge in February 2016, by enabling TLS for its web site, protecting downloads of OpenSSL with TLS, publishing processes for reporting vulnerabilities and contributing code, using static and dynamic analysis before public releases, and using continuous integration.

The OpenSSL project began using Travis for continuous integration in August 2015. In additional to building the software and performing unit tests, OpenSSL’s Travis configuration reports test coverage to coveralls.io and runs the flake8 static analysis tool on the library’s python scripts. Continuous integration has been shown to change other software development practices, such as code contribution processes, issue handling, and testing (Zhao et al., 2017). After adopting continuous integration, OpenSSL changed in all of those areas, adopting a new code contribution policy, migrating issue tracking from Request Tracker to GitHub (Salz, 2016), and increasing unit testing.

Figure 3. OpenSSL Unit Tests by Version

Plot of OpenSSL unit tests by version

OpenSSL testing practices changed considerably after Heartbleed. The project adopted a new framework to make it easier to write unit tests. The number of unit tests increased from 42 in version 1.0.1 (the most recent version as of Heartbleed) to 152 in 1.1.1 (the current version). Figure 3 displays the growth in unit tests by major version, including the trend for the number of tests at the time of initial release and the number of tests for the most recent release. It is important to note that the time between initial and final release of a version is much larger for older versions. Version 0.9.8 adds three tests from first to final release, which happened over the course of ten years and 35 minor releases. Version 1.1.1 also adds three tests, but over a much shorter time period: three minor releases and a single year. Modern versions (1.1.0 and 1.1.1) added approximately one test per minor release, while older versions rarely added tests for minor releases.

The OpenSSL project began measuring test coverage in 2016. Test coverage grew from 54.6% in 2016 to 64.2% in 2019 as measured by coveralls.io. There is no coverage data prior to 2016, but the increase in the number of unit tests from the last pre-Heartbleed version (1.0.1), which had 47 unit tests, to the version released in 2016 (1.1.0), which had 85 unit tests on initial release, suggests that code coverage was lower before test coverage statistics were collected.

OpenSSL has incorporated fuzz testing in its unit tests, and Google’s OSS Fuzz project 555https://google.github.io/oss-fuzz/ also tests OpenSSL. While only sixteen OpenSSL vulnerabilities identify the technique or tool used to find the vulnerability, all sixteen vulnerabilities identify fuzz testing as the technique. Nine of the sixteen identify OSS Fuzz as the tool used, while five identify libFuzzer and two identify TLS-Attacker.

The OpenSSL project’s code is regularly scanned by a security-oriented static analysis tool as part of the Coverity Scan project 666https://scan.coverity.com/. When OpenSSL was initially scanned in 2006, commit messages reported fixing several bugs reported by Coverity. Coverity related commit messages continue through January 2, 2009. The next mention of Coverity in a commit message occurs on May 5, 2014, a month after Heartbleed. Similar commit messages continue through the end of 2019, so static analysis has continued to be used since 2014. Our interviewee indicated that Coverity scan results are primarily used when preparing a new release of OpenSSL.

7.7. Sustainability of Changes

Our final research question, RQ7, focuses on the durability of the changes made in response to Heartbleed. Improvements in the OpenSSL project during the immediate post-Heartbleed era can be seen in the project’s attainment of a CII Best Practices badge. The badge score improved from 62% to 105%. While the project has not updated its badge application since 2016, we were able to use the application to manually verify that OpenSSL has continued to follow the best practices identified in its application, including continuous integration, static analysis, and fuzz testing.

When we examine the evolution of code metrics, we find that most improvements have been sustained. Code complexity, as measured by nesting depth, cyclomatic complexity, and Halstead complexity, has continued to decline since 2016. The maximum nesting depth remains at the level it dropped to after the code cleanup. Style inconsistency has begun to grow at a slow rate, but the December 2019 value of 7.09 is still far below the April 2014 value of 58.9.

Growth of the code base resumed in 2016, as features like TLS 1.3 were added, showing that code shrinkage inspired by Heartbleed was a temporary change in software evolution. Figure 4 shows the post-Heartbleed evolution of a sample of metrics, including code size in statements, mean nesting complexity, mean cyclomatic complexity, and style inconsistency.

Figure 4. Four Code Metrics after Heartbleed

Time series graph of code metrics 2014-2019

The security audit commissioned by the CII was performed on version 1.1.0 of OpenSSL in 2016 by the Open Crypto Audit Project (OCAP) (White, 2016). The audit included both code review and fuzz testing. The audit reported several potential bugs in OpenSSL, including two possible code execution vulnerabilities and two possible denial of service (DoS) vulnerabilities. Other issues were reported by the audit team as low severity or difficult to exploit. Only one OpenSSL vulnerability report identifies the OCAP audit team as the reporter.

The Open Source Technology Improvement Fund (OSTIF) performed an audit of OpenSSL 1.1.1, with a focus on the new TLS 1.3 protocol and changes made to the Pseudo Random Number Generator (PRNG) (Quarkslab, 2019). Their report was published in 2019. The OSTIF audit combined manual code review with fuzz testing. OSTIF found two DoS vulnerabilities prior to the release date, enabling the OpenSSL team to fix them before release. The report identified some areas where code quality could be improved by checking return values and implementing global checks for NULL values.

8. Threats to Validity

Construct Validity: We use multiple widely used metrics for measuring code size and complexity to avoid bias in measuring those code characteristics. The style inconsistency metric we use has been used in other studies too (Spinellis et al., 2016).

Internal Validity: One of the most important validity threats to RDD time series analysis is the presence of an event near the cutoff date that influences the observed changes. We mitigated this threat by thoroughly examining OpenSSL blog entries, commit messages, and email archives in the months before and after our cutoff dates for such events. Another threat to internal validity is that only one OpenSSL developer consented to be interviewed, which may bias our qualitative data.

External Validity: As this work is a case study of a single project, we cannot generalize our conclusions to other open source projects. We therefore leave the question of how open source projects can successfully react to security events to future work.

9. Conclusions

The Heartbleed vulnerability brought dramatic changes to OpenSSL, transforming an almost moribund project to an active project with substantial improvements in code quality and security. OpenSSL remains the most commonly used TLS library on public web servers five years after Heartbleed, according to IPv4 scan data collected by censys.io. These improvements provide a model for how open source projects can adapt and improve after a major security event.

We found substantial and sustained improvements in code quality. Code complexity declined sharply during the major code cleanup activity in 2015, and both cyclomatic and Halstead complexity have continued to decline. The code cleanup made coding style much more consistent. While style inconsistency has slowly increased since the cleanup, it remains much lower than before Heartbleed.

The number of vulnerability reports dramatically increased for three years after Heartbleed before returning to previous levels. Only five of the 91 vulnerabilities found in those three years were in post-Heartbleed code, so this represents a substantial improvement in security. Positive results from two external code audits also suggest that the security of OpenSSL has greatly improved. This means that vulnerability count is not a useful indicator of project security. Low vulnerability counts may just indicate that a project is devoting little effort to finding vulnerabilities.

Our results suggest that project activity and practices may be better predictors of project security. The CII badge may be a good indicator of project security, since it requires good open source development practices. To understand how generalizable these recommendations are, we plan to compare the software evolution of OpenSSL with that of related projects, such as BoringSSL and GnuTLS. We also plan to examine the impact of the CII badge and project activity on the development practices and security across multiple open source projects to validate these ideas.

We have provided a replication package for this paper (Walden, 2020), which includes the software metrics and project activity data used in this paper. The package includes data collection scripts and the code used to create the models and generate the figures and tables for this paper. Documentation on how to use the scripts is also provided.

References

  • (1)
  • Aviram et al. (2016) Nimrod Aviram, Sebastian Schinzel, Juraj Somorovsky, Nadia Heninger, Maik Dankel, Jens Steube, Luke Valenta, David Adrian, J Alex Halderman, Viktor Dukhovni, et al. 2016. DROWN: Breaking TLS Using SSLv2. In 25th USENIX Security Symposium (USENIX Security 16). 689–706.
  • Beck (2014) Bob Beck. 2014. LibreSSL: The first 30 Days, and where we go from here. Presentation at the BSDCan, Ottowa, Canada.
  • Caswell (2015) Matt Caswell. 2015. Code Reformat Finished. OpenSSL Blog (2015). https://www.openssl.org/blog/blog/2015/02/11/code-reformat-finished/
  • Cox (2008) Russ Cox. 2008. Lessons from the Debian/OpenSSL Fiasco. Online.
  • Dijkstra (1968) Edsger W Dijkstra. 1968. Go to statement considered harmful. Commun. ACM 11, 3 (1968), 147–148.
  • Durumeric et al. (2014) Zakir Durumeric, Frank Li, James Kasten, Johanna Amann, Jethro Beekman, Mathias Payer, Nicolas Weaver, David Adrian, Vern Paxson, Michael Bailey, et al. 2014. The Matter of Heartbleed. In Proceedings of the 2014 Conference on Internet Measurement. ACM, 475–488.
  • Foundation (2014) The Linux Foundation. 2014. Amazon Web Services, Cisco, Dell, Facebook, Fujitsu, Google, IBM, Intel, Microsoft, NetApp, Rackspace, VMware and The Linux Foundation Form New Initiative to Support Critical Open Source Projects. The Linux Foundation. https://www.linuxfoundation.org/press-release/2014/04/amazon-web-services-cisco-dell-facebook-fujitsu-google-ibm-intel-microsoft-netapp-rackspace-vmware-and-the-linux-foundation-form-new-initiative-to-support-critical-open-source-projects/
  • Gill and Kemerer (1991) Geoffrey K Gill and Chris F Kemerer. 1991. Cyclomatic complexity density and software maintenance productivity. IEEE transactions on software engineering 17, 12 (1991), 1284.
  • Godfrey and Tu (2001) Michael Godfrey and Qiang Tu. 2001. Growth, evolution, and structural change in open source software. In Proceedings of the 4th international workshop on principles of software evolution. ACM, 103–106.
  • Gonzalez-Barahona et al. (2014) Jesus M Gonzalez-Barahona, Gregorio Robles, Israel Herraiz, and Felipe Ortega. 2014. Studying the laws of software evolution in a long-lived FLOSS project. Journal of Software: Evolution and Process 26, 7 (2014), 589–612.
  • Google ([n.d.]) Google. [n.d.]. Google Style Guides. https://google.github.io/styleguide/
  • Halstead et al. (1977) Maurice Howard Halstead et al. 1977. Elements of software science. Vol. 7. Elsevier New York.
  • Hudson and Salz (2016) Tim Hudson and Rich Salz. 2016. OpenSSL after Heartbleed. Presentation at the RSA Conference, San Francisco, CA.
  • Izurieta and Bieman (2006) Clemente Izurieta and James Bieman. 2006. The evolution of FreeBSD and Linux. In Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering. ACM, 204–211.
  • Jacob et al. (2012) Robin Jacob, Pei Zhu, Marie-Andrée Somers, and Howard Bloom. 2012. A Practical Guide to Regression Discontinuity. MDRC (2012).
  • Jimenez et al. (2016) Matthieu Jimenez, Mike Papadakis, and Yves Le Traon. 2016. An empirical analysis of vulnerabilities in openssl and the linux kernel. In 2016 23rd Asia-Pacific Software Engineering Conference (APSEC). IEEE, 105–112.
  • Kupsch and Miller (2014) James A Kupsch and Barton P Miller. 2014. Why do software assurance tools have problems finding bugs like heartbleed. Continuous Software Assurance Marketplace 22 (2014).
  • Langley (2015) Adam Langley. 2015. BoringSSL. Imperial Violet (2015). https://www.imperialviolet.org/2015/10/17/boringssl.html
  • Lehman (1996) Manny M Lehman. 1996. Laws of software evolution revisited. In European Workshop on Software Process Technology. Springer, 108–124.
  • Li et al. (2017) Deguang Li, Bing Guo, Yan Shen, Junke Li, and Yanhui Huang. 2017. The evolution of open-source mobile applications: An empirical study. Journal of software: Evolution and process 29, 7 (2017), e1855.
  • Marquess (2014) Steve Marquess. 2014. Of Money, Responsibility, and Pride. http://veridicalsystems.com/blog/of-money-responsibility-and-pride/
  • Martin (2009) Robert C Martin. 2009. Clean code: a handbook of agile software craftsmanship. Pearson Education.
  • McCabe (1976) Thomas J McCabe. 1976. A complexity measure. IEEE Transactions on software Engineering 4 (1976), 308–320.
  • Menzies et al. (2002) Tim Menzies, Justin S Di Stefano, Mike Chapman, and Ken McGill. 2002. Metrics that matter. In 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings. IEEE, 51–57.
  • Menzies et al. (2006) Tim Menzies, Jeremy Greenwald, and Art Frank. 2006. Data mining static code attributes to learn defect predictors. IEEE transactions on software engineering 33, 1 (2006), 2–13.
  • Midha et al. (2010) Vishal Midha, Rahul Singh, Prashant Palvia, and Nir Kshetri. 2010. Improving open source software maintenance. Journal of Computer Information Systems 50, 3 (2010), 81–90.
  • Miller (2005) Damien Miller. 2005. Secure Portability. https://www.openbsd.org/papers/portability.pdf
  • Nagappan et al. (2015) Meiyappan Nagappan, Romain Robbes, Yasutaka Kamei, Éric Tanter, Shane McIntosh, Audris Mockus, and Ahmed E Hassan. 2015. An empirical study of goto in C code from GitHub repositories. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 404–414.
  • Neamtiu et al. (2013) Iulian Neamtiu, Guowu Xie, and Jianbo Chen. 2013. Towards a better understanding of software evolution: an empirical study on open-source software. Journal of Software: Evolution and Process 25, 3 (2013), 193–218.
  • Oracle (1999) Oracle. 1999. Code Conventions for the Java Programming Languages. https://www.oracle.com/technetwork/java/javase/documentation/codeconventions-139411.html
  • Paulson et al. (2004) James W Paulson, Giancarlo Succi, and Armin Eberlein. 2004. An empirical study of open-source and closed-source software products. IEEE Transactions on Software Engineering 30, 4 (2004), 246–256.
  • Quarkslab (2019) Quarkslab. 2019. OpenSSL Security Assessment. https://ostif.org/wp-content/uploads/2019/01/18-04-720-REP_v1.2.pdf
  • Reichardt (2019) Charles S Reichardt. 2019. Quasi-Experimentation: A Guide to Design and Analysis. Guilford Publications.
  • Reisinger (2015) Don Reisinger. 2015. Android shipments in 2014 exceed 1 billion for first time. CNET (2015). https://www.cnet.com/news/android-shipments-exceed-1-billion-for-first-time-in-2014/
  • Salz (2015) Rich Salz. 2015. Beyond Reformatting: More Code Cleanup. OpenSSL Blog (2015). https://www.openssl.org/blog/blog/2015/07/28/code-cleanup/
  • Salz (2016) Rich Salz. 2016. Face to Face: Goodbye RT, Hello GitHub. OpenSSL Blog (2016). https://www.openssl.org/blog/blog/2016/10/12/f2f-rt-github/
  • Shin et al. (2010) Yonghee Shin, Andrew Meneely, Laurie Williams, and Jason A Osborne. 2010. Evaluating complexity, code churn, and developer activity metrics as indicators of software vulnerabilities. IEEE Transactions on Software Engineering 37, 6 (2010), 772–787.
  • Smit et al. (2011) Michael Smit, Barry Gergel, H James Hoover, and Eleni Stroulia. 2011. Maintainability and source code conventions: An analysis of open source projects. University of Alberta, Department of Computing Science, Tech. Rep. TR11 6 (2011).
  • Spadini et al. (2018) Davide Spadini, Maurício Aniche, and Alberto Bacchelli. 2018. PyDriller: Python framework for mining software repositories. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering - ESEC/FSE 2018. ACM Press, New York, New York, USA, 908–911. https://doi.org/10.1145/3236024.3264598
  • Spencer and Collyer (1992) Henry Spencer and Geoff Collyer. 1992. # ifdef confirmed harmful or portability experience with C news. In Proceedings of the Summer 1992 USENIX Conference. USENIX, 185–198.
  • Spinellis et al. (2016) Diomidis Spinellis, Panos Louridas, and Maria Kechagia. 2016. The Evolution of C Programming Practices: A Study of the Unix Operating System 1973-2015. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, 748–759.
  • Team (2015) OpenSSL Project Team. 2015. OpenSSL Coding Style. (2015). https://www.openssl.org/policies/codingstyle.html
  • Team (2016) OpenSSL Project Team. 2016. Project Roadmap. (2016). https://web.archive.org/web/20150905203707/https://www.openssl.org/policies/roadmap.html
  • Team (2017a) OpenSSL Project Team. 2017a. OpenSSL Bylaws. (2017). https://www.openssl.org/policies/omc-bylaws.html
  • Team (2017b) OpenSSL Project Team. 2017b. Policy for OpenSSL Committers. (2017). https://www.openssl.org/policies/committers.html
  • Trockman et al. (2018) Asher Trockman, Shurui Zhou, Christian Kästner, and Bogdan Vasilescu. 2018. Adding sparkle to social coding: an empirical study of repository badges in the npm ecosystem. In Proceedings of the 40th International Conference on Software Engineering. ACM, 511–522.
  • van Rossum et al. (2001) Guido van Rossum, Barry Warsaw, and Nick Coghlan. 2001. PEP 8 – Style Guide for Python Code. https://www.python.org/dev/peps/pep-0008/
  • Walden (2020) James Walden. 2020. OpenSSL Case Study Replication Package (MSR 2020). https://doi.org/10.6084/M9.FIGSHARE.11991348
  • Wheeler ([n.d.]) David Wheeler. [n.d.]. Core Infrastructure Initiative (CII) Best-Practices Badge Criteria. https://www.ida.org/-/media/feature/publications/c/co/core-infrastructure-initiative-cii-bestpractices-badge-criteria/d-8054.ashx
  • Wheeler (2014) David A Wheeler. 2014. How to Prevent the next Heartbleed. Online (2014). http://www.dwheeler.com/essays/heartbleed.html
  • White (2016) Kenneth White. 2016. The OpenSSL 1.1 Audit. International Cryptographic Module Conference, Ottowa, Ontario. https://icmconference.org/wp-content/uploads/G12b-White.pdf
  • Wu and Holt (2004) Jingwei Wu and Richard C Holt. 2004. Linker-based program extraction and its uses in studying software evolution. In Proceedings of the International Workshop on Foundations of Unanticipated Software Evolution. 1–15.
  • Zhao et al. (2017) Yangyang Zhao, Alexander Serebrenik, Yuming Zhou, Vladimir Filkov, and Bogdan Vasilescu. 2017. The impact of continuous integration on other software development practices: a large-scale empirical study. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 60–71.
  • Zimmermann and Artís (2019) Théo Zimmermann and Annalí Casanueva Artís. 2019. Impact of switching bug trackers: a case study on a medium-sized open source project. In Proceedings of the 36th International Conference on Software Maintenance and Evolution. IEEE Press.