Prior Activation Distribution (PAD): A Versatile Representation to Utilize DNN Hidden Units

07/05/2019

∙

In this paper, we introduce the concept of Prior Activation Distribution (PAD) as a versatile and general technique to capture the typical activation patterns of hidden layer units of a Deep Neural Network used for classification tasks. We show that the combined neural activations of such a hidden layer have class-specific distributional properties, and then define multiple statistical measures to compute how far a test sample's activations deviate from such distributions. Using a variety of benchmark datasets (including MNIST, CIFAR10, Fashion-MNIST & notMNIST), we show how such PAD-based measures can be used, independent of any training technique, to (a) derive fine-grained uncertainty estimates for inferences; (b) provide inferencing accuracy competitive with alternatives that require execution of the full pipeline, and (c) reliably isolate out-of-distribution test samples.

READ FULL TEXT

Prior Activation Distribution (PAD): A Versatile Representation to Utilize DNN Hidden Units

Multi-Activation Hidden Units for Neural Networks with Random Weights

Improving Classification Neural Networks by using Absolute activation function (MNIST/LeNET-5 example)

Coloring the Black Box: Visualizing neural network behavior with a self-introspective model

Learning ReLU Networks via Alternating Minimization

Training for temporal sparsity in deep neural networks, application in video processing

Examining the Proximity of Adversarial Examples to Class Manifolds in Deep Networks

KCNet: An Insect-Inspired Single-Hidden-Layer Neural Network with Randomized Binary Weights for Prediction and Classification Tasks

Prior Activation Distribution (PAD): A Versatile Representation to Utilize DNN Hidden Units

Related Research

Multi-Activation Hidden Units for Neural Networks with Random Weights

Improving Classification Neural Networks by using Absolute activation function (MNIST/LeNET-5 example)

Coloring the Black Box: Visualizing neural network behavior with a self-introspective model

Learning ReLU Networks via Alternating Minimization

Training for temporal sparsity in deep neural networks, application in video processing

Examining the Proximity of Adversarial Examples to Class Manifolds in Deep Networks

KCNet: An Insect-Inspired Single-Hidden-Layer Neural Network with Randomized Binary Weights for Prediction and Classification Tasks