Syntactic Vs. Semantic similarity of Artificial and Real Faults in Mutation Testing Studies

12/29/2021
by   Milos Ojdanic, et al.
0

Fault seeding is typically used in controlled studies to evaluate and compare test techniques. Central to these techniques lies the hypothesis that artificially seeded faults involve some form of realistic properties and thus provide realistic experimental results. In an attempt to strengthen realism, a recent line of research uses advanced machine learning techniques, such as deep learning and Natural Language Processing (NLP), to seed faults that look like (syntactically) real ones, implying that fault realism is related to syntactic similarity. This raises the question of whether seeding syntactically similar faults indeed results in semantically similar faults and more generally whether syntactically dissimilar faults are far away (semantically) from the real ones. We answer this question by employing 4 fault-seeding techniques (PiTest - a popular mutation testing tool, IBIR - a tool with manually crafted fault patterns, DeepMutation - a learning-based fault seeded framework and CodeBERT - a novel mutation testing tool that use code embeddings) and demonstrate that syntactic similarity does not reflect semantic similarity. We also show that 60 resembled by CodeBERT, PiTest, IBIR, and DeepMutation faults. We then perform an objective comparison between the techniques and find that CodeBERT and PiTest have similar fault detection capabilities that subsume IBIR and DeepMutation, and that IBIR is the most cost-effective technique. Moreover, the overall fault detection of PiTest, CodeBERT, IBIR, and DeepMutation was, on average, 54

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/11/2020

IBIR: Bug Report driven Fault Injection

Much research on software engineering and software testing relies on exp...
research
02/12/2020

DeepMutation: A Neural Mutation Tool

Mutation testing can be used to assess the fault-detection capabilities ...
research
03/21/2018

Selecting Fault Revealing Mutants

Mutant selection refers to the problem of choosing, among a large number...
research
03/07/2022

μBERT: Mutation Testing using Pre-Trained Language Models

We introduce μBERT, a mutation testing tool that uses a pre-trained lang...
research
08/24/2022

Automatic detection of faults in race walking from a smartphone camera: a comparison of an Olympic medalist and university athletes

Automatic fault detection is a major challenge in many sports. In race w...
research
08/19/2022

A Two-phase Metamorphic Approach for Testing Industrial Control Systems

We elaborate on a metamorphic approach for testing industrial control sy...
research
02/26/2019

Amortising the Cost of Mutation Based Fault Localisation using Statistical Inference

Mutation analysis can effectively capture the dependency between source ...

Please sign up or login with your details

Forgot password? Click here to reset