Adaptive Metamorphic Testing with Contextual Bandits

10/01/2019
by   Helge Spieker, et al.
0

Metamorphic Testing is a software testing paradigm which aims at using necessary properties of a system-under-test, called metamorphic relations, to either check its expected outputs, or to generate new test cases. Metamorphic Testing has been successful to test programs for which a full oracle is not available or to test programs for which there are uncertainties on expected outputs such as learning systems. In this article, we propose Adaptive Metamorphic Testing as a generalization of a simple yet powerful reinforcement learning technique, namely contextual bandits, to select one of the multiple metamorphic relations available for a program. By using contextual bandits, Adaptive Metamorphic Testing learns which metamorphic relations are likely to transform a source test case, such that it has higher chance to discover faults. We present experimental results over two major case studies in machine learning, namely image classification and object detection, and identify weaknesses and robustness boundaries. Adaptive Metamorphic Testing efficiently identifies weaknesses of the tested systems in context of the source test case.

READ FULL TEXT

page 8

page 10

09/01/2022

Prioritization of Metamorphic Relations to reduce the cost of testing

An oracle is a mechanism to decide whether the outputs of the program fo...
04/26/2022

Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective

Metamorphic testing has recently been used to check the safety of neural...
02/20/2018

Using Semi-Supervised Learning for Predicting Metamorphic Relations

Software testing is difficult to automate, especially in programs which ...
02/19/2018

Predicting Metamorphic Relation for Matrix Calculation Programs

Matrices often represent important information in scientific application...
06/11/2018

Greybox fuzzing as a contextual bandits problem

Greybox fuzzing is one of the most useful and effective techniques for t...
08/23/2018

Enhancing POI testing approach through the use of additional information

Recently, a new approach to perform regression testing has been defined:...
08/19/2019

An Autonomous Performance Testing Framework using Self-Adaptive Fuzzy Reinforcement Learning

Test automation can result in reduction in cost and human effort. If the...

Code Repositories

tetraband

Implementation of "Adaptive Metamorphic Testing with Contextual Bandits"


view repo