Empirical Study of Restarted and Flaky Builds on Travis CI

03/26/2020
by   Thomas Durieux, et al.
0

Continuous Integration (CI) is a development practice where developers frequently integrate code into a common codebase. After the code is integrated, the CI server runs a test suite and other tools to produce a set of reports (e.g., output of linters and tests). If the result of a CI test run is unexpected, developers have the option to manually restart the build, re-running the same test suite on the same code; this can reveal build flakiness, if the restarted build outcome differs from the original build. In this study, we analyze restarted builds, flaky builds, and their impact on the development workflow. We observe that developers restart at least 1.72 builds, amounting to 56,522 restarted builds in our Travis CI dataset. We observe that more mature and more complex projects are more likely to include restarted builds. The restarted builds are mostly builds that are initially failing due to a test, network problem, or a Travis CI limitations such as execution timeout. Finally, we observe that restarted builds have a major impact on development workflow. Indeed, in 54.42 developers analyze and restart a build within an hour of the initial failure. This suggests that developers wait for CI results, interrupting their workflow to address the issue. Restarted builds also slow down the merging of pull requests by a factor of three, bringing median merging time from 16h to 48h.

READ FULL TEXT
research
08/19/2023

Repeated Builds During Code Review: An Empirical Study of the OpenStack Community

Code review is a popular practice where developers critique each others'...
research
08/27/2021

LaForge: Always-Correct and Fast Incremental Builds from Simple Specifications

Developers rely on build systems to generate software from code. At a mi...
research
12/19/2017

Built to Last or Built Too Fast? Evaluating Prediction Models for Build Times

Automated builds are integral to the Continuous Integration (CI) softwar...
research
02/18/2021

Smart Build Targets Batching Service at Google

Google has a monolithic codebase with tens of millions build targets. Ea...
research
08/19/2022

Fixing Dockerfile Smells: An Empirical Study

Background. Containerization technologies are widely adopted in the DevO...
research
05/15/2018

The remote_build Tool

This is an introduction to the remote_build tool for transparent remote ...
research
01/22/2019

Towards Predicting the Impact of Software Changes on Building Activities

The pervasive adoption of Continuous Integration practices -- both in in...

Please sign up or login with your details

Forgot password? Click here to reset