Can Offline Testing of Deep Neural Networks Replace Their Online Testing?

01/26/2021
by   Fitash Ul Haq, et al.
0

We distinguish two general modes of testing for Deep Neural Networks (DNNs): Offline testing where DNNs are tested as individual units based on test datasets obtained independently from the DNNs under test, and online testing where DNNs are embedded into a specific application environment and tested in a closed-loop mode in interaction with the application environment. Typically, DNNs are subjected to both types of testing during their development life cycle where offline testing is applied immediately after DNN training and online testing follows after offline testing and once a DNN is deployed within a specific application environment. In this paper, we study the relationship between offline and online testing. Our goal is to determine how offline testing and online testing differ or complement one another and if we can use offline testing results to run fewer tests during online testing to reduce the testing cost. Though these questions are generally relevant to all autonomous systems, we study them in the context of automated driving systems where, as study subjects, we use DNNs automating end-to-end controls of steering functions of self-driving vehicles. Our results show that offline testing is more optimistic than online testing as many safety violations identified by online testing could not be identified by offline testing, while large prediction errors generated by offline testing always led to severe safety violations detectable by online testing. Further, we cannot use offline testing results to run fewer tests during online testing in practice since we are not able to identify specific situations where offline testing could be as accurate as online testing in identifying safety requirement violations.

READ FULL TEXT

page 16

page 28

research
11/28/2019

Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study

There is a growing body of research on developing testing techniques for...
research
12/03/2021

Enhancing Deep Neural Networks Testing by Traversing Data Manifold

We develop DEEPTRAVERSAL, a feedback-driven framework to test DNNs. DEEP...
research
10/09/2022

Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gap

There is an ever-growing zoo of modern neural network models that can ef...
research
03/02/2023

Iterative Assessment and Improvement of DNN Operational Accuracy

Deep Neural Networks (DNN) are nowadays largely adopted in many applicat...
research
10/27/2022

Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems

Deep Neural Networks (DNNs) have been widely used to perform real-world ...
research
07/09/2019

Click-Through Rate Prediction with the User Memory Network

Click-through rate (CTR) prediction is a critical task in online adverti...
research
01/11/2019

Input Prioritization for Testing Neural Networks

Deep neural networks (DNNs) are increasingly being adopted for sensing a...

Please sign up or login with your details

Forgot password? Click here to reset