DeepAI AI Chat
Log In Sign Up

DeepSmartFuzzer: Reward Guided Test Generation For Deep Learning

by   Samet Demir, et al.
Max Planck Institute for Software Systems
Boğaziçi University

Testing Deep Neural Network (DNN) models has become more important than ever with the increasing usage of DNN models in safety-critical domains such as autonomous cars. The traditional approach of testing DNNs is to create a test set, which is a random subset of the dataset about the problem of interest. This kind of approach is not enough for testing most of the real-world scenarios since these traditional test sets do not include corner cases, while a corner case input is generally considered to introduce erroneous behaviors. Recent works on adversarial input generation, data augmentation, and coverage-guided fuzzing (CGF) have provided new ways to extend traditional test sets. Among those, CGF aims to produce new test inputs by fuzzing existing ones to achieve high coverage on a test adequacy criterion (i.e. coverage criterion). Given that the subject test adequacy criterion is a well-established one, CGF can potentially find error inducing inputs for different underlying reasons. In this paper, we propose a novel CGF solution for structural testing of DNNs. The proposed fuzzer employs Monte Carlo Tree Search to drive the coverage-guided search in the pursuit of achieving high coverage. Our evaluation shows that the inputs generated by our method result in higher coverage than the inputs produced by the previously introduced coverage-guided fuzzing techniques.


page 3

page 6


Testing Deep Neural Networks

Deep neural networks (DNNs) have a wide range of applications, and softw...

Distribution-Aware Testing of Neural Networks Using Generative Models

The reliability of software that has a Deep Neural Network (DNN) as a co...

Graph-Based Fuzz Testing for Deep Learning Inference Engine

Testing deep learning (DL) systems are increasingly crucial as the incre...

Deep Neural Network Test Coverage: How Far Are We?

DNN testing is one of the most effective methods to guarantee the qualit...

DeepHunter: Hunting Deep Neural Network Defects via Coverage-Guided Fuzzing

In company with the data explosion over the past decade, deep neural net...

You Can't See the Forest for Its Trees: Assessing Deep Neural Network Testing via NeuraL Coverage

This paper summarizes eight design requirements for DNN testing criteria...

SINVAD: Search-based Image Space Navigation for DNN Image Classifier Test Input Generation

The testing of Deep Neural Networks (DNNs) has become increasingly impor...