Input Prioritization for Testing Neural Networks

01/11/2019
by   Taejoon Byun, et al.
0

Deep neural networks (DNNs) are increasingly being adopted for sensing and control functions in a variety of safety and mission-critical systems such as self-driving cars, autonomous air vehicles, medical diagnostics, and industrial robotics. Failures of such systems can lead to loss of life or property, which necessitates stringent verification and validation for providing high assurance. Though formal verification approaches are being investigated, testing remains the primary technique for assessing the dependability of such systems. Due to the nature of the tasks handled by DNNs, the cost of obtaining test oracle data---the expected output, a.k.a. label, for a given input---is high, which significantly impacts the amount and quality of testing that can be performed. Thus, prioritizing input data for testing DNNs in meaningful ways to reduce the cost of labeling can go a long way in increasing testing efficacy. This paper proposes using gauges of the DNN's sentiment derived from the computation performed by the model, as a means to identify inputs that are likely to reveal weaknesses. We empirically assessed the efficacy of three such sentiment measures for prioritization---confidence, uncertainty, and surprise---and compare their effectiveness in terms of their fault-revealing capability and retraining effectiveness. The results indicate that sentiment measures can effectively flag inputs that expose unacceptable DNN behavior. For MNIST models, the average percentage of inputs correctly flagged ranged from 88

READ FULL TEXT
research
02/15/2019

DeepFault: Fault Localization for Deep Neural Networks

Deep Neural Networks (DNNs) are increasingly deployed in safety-critical...
research
05/19/2020

SINVAD: Search-based Image Space Navigation for DNN Image Classifier Test Input Generation

The testing of Deep Neural Networks (DNNs) has become increasingly impor...
research
07/18/2023

CertPri: Certifiable Prioritization for Deep Neural Networks via Movement Cost in Feature Space

Deep neural networks (DNNs) have demonstrated their outperformance in va...
research
03/08/2023

DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks

Deep neural networks (DNNs) are widely used in various application domai...
research
03/02/2019

DeepGini: Prioritizing Massive Tests to Reduce Labeling Cost

Deep neural network (DNN) based systems have been deployed to assist var...
research
01/26/2021

Can Offline Testing of Deep Neural Networks Replace Their Online Testing?

We distinguish two general modes of testing for Deep Neural Networks (DN...
research
07/10/2021

HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks

Deep Neural Networks (DNN) applications are increasingly becoming a part...

Please sign up or login with your details

Forgot password? Click here to reset