-
DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems
Deep learning defines a new data-driven programming paradigm that constr...
read it
-
What Are We Really Testing in Mutation Testing for Machine Learning? A Critical Reflection
Mutation testing is a well-established technique for assessing a test su...
read it
-
The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models
In this paper, we introduce the TrojAI software framework, an open sourc...
read it
-
DeepCruiser: Automated Guided Testing for Stateful Deep Learning Systems
Deep learning (DL) defines a data-driven programming paradigm that autom...
read it
-
Measuring Coverage of Prolog Programs Using Mutation Testing
Testing is an important aspect in professional software development, bot...
read it
-
Towards Characterizing Adversarial Defects of Deep Learning Software from the Lens of Uncertainty
Over the past decade, deep learning (DL) has been successfully applied t...
read it
-
Towards Testing of Deep Learning Systems with Training Set Reduction
Testing the implementation of deep learning systems and their training r...
read it
DeepMutation: Mutation Testing of Deep Learning Systems
Deep learning (DL) defines a new data-driven programming paradigm where the internal system logic is largely shaped by the training data. The standard way of evaluating DL models is to examine their performance on a test dataset. The quality of the test dataset is of great importance to gain confidence of the trained models. Using inadequate test dataset, DL models that have achieved high test accuracy may still suffer from vulnerability against (adversarial) attacks. In software testing, mutation testing is a well-established technique to evaluate the quality of test suites. However, due to the fundamental difference of traditional software and deep learning-based software, traditional mutation testing techniques cannot be directly applied to DL systems. In this paper, we propose the mutation testing framework specialized for DL systems. We first propose a source-level mutation testing technique to slightly modify source (i.e., training data and training programs) of DL software, which shares the same spirit of traditional mutation testing. Then we design a set of model-level mutation testing operators that directly mutate on DL models without a training process. The effectiveness of the proposed mutation techniques is demonstrated on two public datasets MNIST and CIFAR-10 with three DL models.
READ FULL TEXT
Comments
There are no comments yet.