DeepMutation: Mutation Testing of Deep Learning Systems

by   Lei Ma, et al.

Deep learning (DL) defines a new data-driven programming paradigm where the internal system logic is largely shaped by the training data. The standard way of evaluating DL models is to examine their performance on a test dataset. The quality of the test dataset is of great importance to gain confidence of the trained models. Using inadequate test dataset, DL models that have achieved high test accuracy may still suffer from vulnerability against (adversarial) attacks. In software testing, mutation testing is a well-established technique to evaluate the quality of test suites. However, due to the fundamental difference of traditional software and deep learning-based software, traditional mutation testing techniques cannot be directly applied to DL systems. In this paper, we propose the mutation testing framework specialized for DL systems. We first propose a source-level mutation testing technique to slightly modify source (i.e., training data and training programs) of DL software, which shares the same spirit of traditional mutation testing. Then we design a set of model-level mutation testing operators that directly mutate on DL models without a training process. The effectiveness of the proposed mutation techniques is demonstrated on two public datasets MNIST and CIFAR-10 with three DL models.


DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems

Deep learning defines a new data-driven programming paradigm that constr...

What Are We Really Testing in Mutation Testing for Machine Learning? A Critical Reflection

Mutation testing is a well-established technique for assessing a test su...

DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score

Deep Learning (DL) components are routinely integrated into software sys...

Copy, Right? A Testing Framework for Copyright Protection of Deep Learning Models

Deep learning (DL) models, especially those large-scale and high-perform...

The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models

In this paper, we introduce the TrojAI software framework, an open sourc...

DeepCruiser: Automated Guided Testing for Stateful Deep Learning Systems

Deep learning (DL) defines a data-driven programming paradigm that autom...

Testing Deep Learning Models: A First Comparative Study of Multiple Testing Techniques

Deep Learning (DL) has revolutionized the capabilities of vision-based s...