What is the Vocabulary of Flaky Tests? An Extended Replication

03/23/2021
by   B. H. P. Camara, et al.
0

Software systems have been continuously evolved and delivered with high quality due to the widespread adoption of automated tests. A recurring issue hurting this scenario is the presence of flaky tests, a test case that may pass or fail non-deterministically. A promising, but yet lacking more empirical evidence, approach is to collect static data of automated tests and use them to predict their flakiness. In this paper, we conducted an empirical study to assess the use of code identifiers to predict test flakiness. To do so, we first replicate most parts of the previous study of Pinto et al. (MSR 2020). This replication was extended by using a different ML Python platform (Scikit-learn) and adding different learning algorithms in the analyses. Then, we validated the performance of trained models using datasets with other flaky tests and from different projects. We successfully replicated the results of Pinto et al. (2020), with minor differences using Scikit-learn; different algorithms had performance similar to the ones used previously. Concerning the validation, we noticed that the recall of the trained models was smaller, and classifiers presented a varying range of decreases. This was observed in both intra-project and inter-projects test flakiness prediction.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/26/2021

On the use of test smells for prediction of flaky tests

Regression testing is an important phase to deliver software with qualit...
research
04/13/2022

Lessons learned from replicating a study on information-retrieval based test case prioritization

Objective: In this study, we aim to replicate an artefact-based study on...
research
07/03/2022

An Empirical Study of Flaky Tests in JavaScript

Flaky tests (tests with non-deterministic outcomes) can be problematic f...
research
01/30/2023

Faster or Slower? Performance Mystery of Python Idioms Unveiled with Empirical Evidence

The usage of Python idioms is popular among Python developers in a forma...
research
05/31/2022

A Replication Study on Predicting Metamorphic Relations at Unit Testing Level

Metamorphic Testing (MT) addresses the test oracle problem by examining ...
research
09/11/2019

Iterative versus Exhaustive Data Selection for Cross Project Defect Prediction: An Extended Replication Study

Context: The effectiveness of data selection approaches in improving the...
research
05/08/2023

FlaPy: Mining Flaky Python Tests at Scale

Flaky tests obstruct software development, and studying and proposing mi...

Please sign up or login with your details

Forgot password? Click here to reset