DeepMutation: Mutation Testing of Deep Learning Systems

05/14/2018 ∙ by Lei Ma, et al. ∙ 0 ∙ share

Deep learning (DL) defines a new data-driven programming paradigm where the internal system logic is largely shaped by the training data. The standard way of evaluating DL models is to examine their performance on a test dataset. The quality of the test dataset is of great importance to gain confidence of the trained models. Using inadequate test dataset, DL models that have achieved high test accuracy may still suffer from vulnerability against (adversarial) attacks. In software testing, mutation testing is a well-established technique to evaluate the quality of test suites. However, due to the fundamental difference of traditional software and deep learning-based software, traditional mutation testing techniques cannot be directly applied to DL systems. In this paper, we propose the mutation testing framework specialized for DL systems. We first propose a source-level mutation testing technique to slightly modify source (i.e., training data and training programs) of DL software, which shares the same spirit of traditional mutation testing. Then we design a set of model-level mutation testing operators that directly mutate on DL models without a training process. The effectiveness of the proposed mutation techniques is demonstrated on two public datasets MNIST and CIFAR-10 with three DL models.

READ FULL TEXT