Runtime Monitoring Neuron Activation Patterns

09/18/2018
by   Chih-Hong Cheng, et al.
0

For using neural networks in safety critical domains, it is important to know if a decision made by a neural network is supported by prior similarities in training. We propose runtime neuron activation pattern monitoring - after the standard training process, one creates a monitor by feeding the training data to the network again in order to store the neuron activation patterns in abstract form. In operation, a classification decision over an input is further supplemented by examining if a pattern similar (measured by Hamming distance) to the generated pattern is contained in the monitor. If the monitor does not contain any pattern similar to the generated pattern, it raises a warning that the decision is not based on the training data. Our experiments show that, by adjusting the similarity-threshold for activation patterns, the monitors can report a significant portion of misclassfications to be not supported by training with a small false-positive rate, when evaluated on a test set.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/18/2018

Runtime Monitoring Neural Activation Patterns

For using neural networks in safety critical domains, it is important to...
research
11/24/2020

Provably-Robust Runtime Monitoring of Neuron Activation Patterns

For deep neural networks (DNNs) to be used in safety-critical autonomous...
research
11/16/2018

nn-dependability-kit: Engineering Neural Networks for Safety-Critical Systems

nn-dependability-kit is an open-source toolbox to support safety enginee...
research
06/05/2023

Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization

The out-of-distribution (OOD) problem generally arises when neural netwo...
research
07/24/2023

Safety Performance of Neural Networks in the Presence of Covariate Shift

Covariate shift may impact the operational safety performance of neural ...
research
04/18/2016

A Repeated Signal Difference for Recognising Patterns

This paper describes a new mechanism that might help with defining patte...

Please sign up or login with your details

Forgot password? Click here to reset