Log In Sign Up

Deep Neural Network Test Coverage: How Far Are We?

by   Junjie Chen, et al.

DNN testing is one of the most effective methods to guarantee the quality of DNN. In DNN testing, many test coverage metrics have been proposed to measure test effectiveness, including structural coverage and non-structural coverage (which are classified according to whether considering which structural elements are covered during testing). Those test coverage metrics are proposed based on the assumption: they are correlated with test effectiveness (i.e., the generation of adversarial test inputs or the error-revealing capability of test inputs in DNN testing studies). However, it is still unknown whether the assumption is tenable. In this work, we conducted the first extensive study to systematically validate the assumption by controlling for the size of test sets. In the study, we studied seven typical test coverage metrics based on 9 pairs of datasets and models with great diversity (including four pairs that have never been used to evaluate these test coverage metrics before). The results demonstrate that the assumption fails for structural coverage in general but holds for non-structural coverage on more than half of subjects, indicating that measuring the difference of DNN behaviors between test inputs and training data is more promising than measuring which structural elements are covered by test inputs for measuring test effectiveness. Even so, the current non-structural coverage metrics still can be improved from several aspects such as unfriendly parameters and unstable performance. That indicates that although a lot of test coverage metrics have been proposed before, there is still a lot of room for improvement of measuring test effectiveness in DNN testing, and our study has pointed out some promising directions.


An Overview of Structural Coverage Metrics for Testing Neural Networks

Deep neural network (DNN) models, including those used in safety-critica...

Black-Box Testing of Deep Neural Networks through Test Case Diversity

Deep Neural Networks (DNNs) have been extensively used in many areas inc...

DeepSmartFuzzer: Reward Guided Test Generation For Deep Learning

Testing Deep Neural Network (DNN) models has become more important than ...

You Can't See the Forest for Its Trees: Assessing Deep Neural Network Testing via NeuraL Coverage

This paper summarizes eight design requirements for DNN testing criteria...

Situation Coverage Testing for a Simulated Autonomous Car – an Initial Case Study

It is hard to test autonomous robot (AR) software because of the range a...

A Comprehensive Study of Pseudo-tested Methods

Pseudo-tested methods are defined as follows: they are covered by the te...

Revisiting Neuron Coverage Metrics and Quality of Deep Neural Networks

Deep neural networks (DNN) have been widely applied in modern life, incl...