An Empirical Study of Flaky Tests in JavaScript

07/03/2022
by   Negar Hashemi, et al.
0

Flaky tests (tests with non-deterministic outcomes) can be problematic for testing efficiency and software reliability. Flaky tests in test suites can also significantly delay software releases. There have been several studies that attempt to quantify the impact of test flakiness in different programming languages (e.g., Java and Python) and application domains (e.g., mobile and GUI-based). In this paper, we conduct an empirical study of the state of flaky tests in JavaScript. We investigate two aspects of flaky tests in JavaScript projects: the main causes of flaky tests in these projects and common fixing strategies. By analysing 452 commits from large, top-scoring JavaScript projects from GitHub, we found that flakiness caused by concurrency-related issues (e.g., async wait, race conditions or deadlocks) is the most dominant reason for test flakiness. The other top causes of flaky tests are operating system-specific (e.g., features that work on specific OS or OS versions) and network stability (e.g., internet availability or bad socket management). In terms of how flaky tests are dealt with, the majority of those flaky tests (>80 quarantine or remove flaky tests.

READ FULL TEXT
research
01/22/2021

An Empirical Study of Flaky Tests in Python

Tests that cause spurious failures without any code changes, i.e., flaky...
research
03/03/2021

An Empirical Analysis of UI-based Flaky Tests

Flaky tests have gained attention from the research community in recent ...
research
05/08/2023

FlaPy: Mining Flaky Python Tests at Scale

Flaky tests obstruct software development, and studying and proposing mi...
research
11/03/2021

Smells in System User Interactive Tests

Test smells are known as bad development practices that reflect poor des...
research
03/17/2023

On the Effect of Instrumentation on Test Flakiness

Test flakiness is a problem that affects testing and processes that rely...
research
12/01/2022

Test Flakiness' Causes, Detection, Impact and Responses: A Multivocal Review

Flaky tests (tests with non-deterministic outcomes) pose a major challen...
research
03/23/2021

What is the Vocabulary of Flaky Tests? An Extended Replication

Software systems have been continuously evolved and delivered with high ...

Please sign up or login with your details

Forgot password? Click here to reset