Testing Deep Neural Networks

03/10/2018
by   Youcheng Sun, et al.
0

Deep neural networks (DNNs) have a wide range of applications, and software employing them must be thoroughly tested, especially in safety critical domains. However, traditional software testing methodology, including test coverage criteria and test case generation algorithms, cannot be applied directly to DNNs. This paper bridges this gap. First, inspired by the traditional MC/DC coverage criterion, we propose a set of four test criteria that are tailored to the distinct features of DNNs. Our novel criteria are incomparable and complement each other. Second, for each criterion, we give an algorithm for generating test cases based on linear programming (LP). The algorithms produce a new test case (i.e., an input to the DNN) by perturbing a given one. They encode the test requirement and a fragment of the DNN by fixing the activation pattern obtained from the given input example, and then minimize the difference between the new and the current inputs. Finally, we validate our method on a set of networks trained on the MNIST dataset. The utility of our method is shown experimentally with four objectives: (1) bug finding; (2) DNN safety statistics; (3) testing efficiency and (4) DNN internal structure analysis.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/24/2019

DeepSmartFuzzer: Reward Guided Test Generation For Deep Learning

Testing Deep Neural Network (DNN) models has become more important than ...
research
02/17/2019

Towards Improved Testing For Deep Learning

The growing use of deep neural networks in safety-critical applications ...
research
05/19/2020

SINVAD: Search-based Image Space Navigation for DNN Image Classifier Test Input Generation

The testing of Deep Neural Networks (DNNs) has become increasingly impor...
research
02/15/2019

DeepFault: Fault Localization for Deep Neural Networks

Deep Neural Networks (DNNs) are increasingly deployed in safety-critical...
research
11/17/2019

Coverage Testing of Deep Learning Models using Dataset Characterization

Deep Neural Networks (DNNs), with its promising performance, are being i...
research
07/10/2021

HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks

Deep Neural Networks (DNN) applications are increasingly becoming a part...
research
11/23/2019

On Functional Test Generation for Deep Neural Network IPs

Machine learning systems based on deep neural networks (DNNs) produce st...

Please sign up or login with your details

Forgot password? Click here to reset