Understanding Flaky Tests: The Developer's Perspective

07/02/2019
by   Moritz Eck, et al.
0

Flaky tests are software tests that exhibit a seemingly random outcome (pass or fail) when run against the same, identical code. Previous work has examined fixes to flaky tests and has proposed automated solutions to locate as well as fix flaky tests--we complement it by examining the perceptions of software developers about the nature, relevance, and challenges of this phenomenon. We asked 21 professional developers to classify 200 flaky tests they previously fixed, in terms of the nature of the flakiness, the origin of the flakiness, and the fixing effort. We complement this analysis with information about the fixing strategy. Subsequently, we conducted an online survey with 121 developers with a median industrial programming experience of five years. Our research shows that: The flakiness is due to several different causes, four of which have never been reported before, despite being the most costly to fix; flakiness is perceived as significant by the vast majority of developers, regardless of their team's size and project's domain, and it can have effects on resource allocation, scheduling, and the perceived reliability of the test suite; and the challenges developers report to face regard mostly the reproduction of the flaky behavior and the identification of the cause for the flakiness. Data and materials [https://doi.org/10.5281/zenodo.3265785].

READ FULL TEXT
research
03/01/2022

A Survey on How Test Flakiness Affects Developers and What Support They Need To Address It

Non-deterministically passing and failing test cases, so-called flaky te...
research
02/14/2022

Gamekins: Gamifying Software Testing in Jenkins

Developers have to write thorough tests for their software in order to f...
research
02/10/2021

CyberSecurity Challenges for Software Developer Awareness Training in Industrial Environments

Awareness of cybersecurity topics facilitates software developers to pro...
research
08/05/2019

Testability First!

The pivotal role of testing in high-quality software production has driv...
research
05/07/2023

On the perceived relevance of critical internal quality attributes when evolving software features

Several refactorings performed while evolving software features aim to i...
research
02/11/2021

Why Don't Developers Detect Improper Input Validation?'; DROP TABLE Papers; –

Improper Input Validation (IIV) is a software vulnerability that occurs ...

Please sign up or login with your details

Forgot password? Click here to reset