DeepAI AI Chat
Log In Sign Up

An Overview of Structural Coverage Metrics for Testing Neural Networks

by   Muhammad Usman, et al.
Carnegie Mellon University

Deep neural network (DNN) models, including those used in safety-critical domains, need to be thoroughly tested to ensure that they can reliably perform well in different scenarios. In this article, we provide an overview of structural coverage metrics for testing DNN models, including neuron coverage (NC), k-multisection neuron coverage (kMNC), top-k neuron coverage (TKNC), neuron boundary coverage (NBC), strong neuron activation coverage (SNAC) and modified condition/decision coverage (MC/DC). We evaluate the metrics on realistic DNN models used for perception tasks (including LeNet-1, LeNet-4, LeNet-5, and ResNet20) as well as on networks used in autonomy (TaxiNet). We also provide a tool, DNNCov, which can measure the testing coverage for all these metrics. DNNCov outputs an informative coverage report to enable researchers and practitioners to assess the adequacy of DNN testing, compare different coverage measures, and to more conveniently inspect the model's internals during testing.


There is Limited Correlation between Coverage and Robustness for Deep Neural Networks

Deep neural networks (DNN) are increasingly applied in safety-critical s...

Deep Neural Network Test Coverage: How Far Are We?

DNN testing is one of the most effective methods to guarantee the qualit...

DeepSensor: Deep Learning Testing Framework Based on Neuron Sensitivity

Despite impressive capabilities and outstanding performance, deep neural...

Revisiting Neuron Coverage Metrics and Quality of Deep Neural Networks

Deep neural networks (DNN) have been widely applied in modern life, incl...

You Can't See the Forest for Its Trees: Assessing Deep Neural Network Testing via NeuraL Coverage

This paper summarizes eight design requirements for DNN testing criteria...

Is Neuron Coverage Needed to Make Person Detection More Robust?

The growing use of deep neural networks (DNNs) in safety- and security-c...

White-box Testing of NLP models with Mask Neuron Coverage

Recent literature has seen growing interest in using black-box strategie...