An Information Theoretic Interpretation to Deep Neural Networks

05/16/2019
by   Shao-Lun Huang, et al.
0

It is commonly believed that the hidden layers of deep neural networks (DNNs) attempt to extract informative features for learning tasks. In this paper, we formalize this intuition by showing that the features extracted by DNN coincide with the result of an optimization problem, which we call the `universal feature selection' problem, in a local analysis regime. We interpret the weights training in DNN as the projection of feature functions between feature spaces, specified by the network structure. Our formulation has direct operational meaning in terms of the performance for inference tasks, and gives interpretations to the internal computation results of DNNs. Results of numerical experiments are provided to support the analysis.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/21/2019

Semantics for Global and Local Interpretation of Deep Neural Networks

Deep neural networks (DNNs) with high expressiveness have achieved state...
research
05/31/2021

Dominant Patterns: Critical Features Hidden in Deep Neural Networks

In this paper, we find the existence of critical features hidden in Deep...
research
07/26/2020

Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projection

Deep neural networks (DNNs) have achieved state-of-the-art performance a...
research
06/01/2018

Interpreting Deep Learning: The Machine Learning Rorschach Test?

Theoretical understanding of deep learning is one of the most important ...
research
10/13/2020

Neural Gaussian Mirror for Controlled Feature Selection in Neural Networks

Deep neural networks (DNNs) have become increasingly popular and achieve...
research
07/09/2021

Understanding the Distributions of Aggregation Layers in Deep Neural Networks

The process of aggregation is ubiquitous in almost all deep nets models....
research
04/14/2021

Do Neural Network Weights account for Classes Centers?

The exploitation of Deep Neural Networks (DNNs) as descriptors in featur...

Please sign up or login with your details

Forgot password? Click here to reset