Distribution-Aware Testing of Neural Networks Using Generative Models

by   Swaroopa Dola, et al.

The reliability of software that has a Deep Neural Network (DNN) as a component is urgently important today given the increasing number of critical applications being deployed with DNNs. The need for reliability raises a need for rigorous testing of the safety and trustworthiness of these systems. In the last few years, there have been a number of research efforts focused on testing DNNs. However the test generation techniques proposed so far lack a check to determine whether the test inputs they are generating are valid, and thus invalid inputs are produced. To illustrate this situation, we explored three recent DNN testing techniques. Using deep generative model based input validation, we show that all the three techniques generate significant number of invalid test inputs. We further analyzed the test coverage achieved by the test inputs generated by the DNN testing techniques and showed how invalid test inputs can falsely inflate test coverage metrics. To overcome the inclusion of invalid inputs in testing, we propose a technique to incorporate the valid input space of the DNN model under test in the test generation process. Our technique uses a deep generative model-based algorithm to generate only valid inputs. Results of our empirical studies show that our technique is effective in eliminating invalid tests and boosting the number of valid test inputs generated.


page 2

page 8


DeepSmartFuzzer: Reward Guided Test Generation For Deep Learning

Testing Deep Neural Network (DNN) models has become more important than ...

A Forgotten Danger in DNN Supervision Testing: Generating and Detecting True Ambiguity

Deep Neural Networks (DNNs) are becoming a crucial component of modern s...

To test, or not to test: A proactive approach for deciding complete performance test initiation

Software performance testing requires a set of inputs that exercise diff...

SINVAD: Search-based Image Space Navigation for DNN Image Classifier Test Input Generation

The testing of Deep Neural Networks (DNNs) has become increasingly impor...

Boosting Operational DNN Testing Efficiency through Conditioning

With the increasing adoption of Deep Neural Network (DNN) models as inte...

Unveiling Hidden DNN Defects with Decision-Based Metamorphic Testing

Contemporary DNN testing works are frequently conducted using metamorphi...

Controlled time series generation for automotive software-in-the-loop testing using GANs

Testing automotive mechatronic systems partly uses the software-in-the-l...

Please sign up or login with your details

Forgot password? Click here to reset