Log In Sign Up

Mutual Information of Neural Network Initialisations: Mean Field Approximations

by   Jared Tanner, et al.

The ability to train randomly initialised deep neural networks is known to depend strongly on the variance of the weight matrices and biases as well as the choice of nonlinear activation. Here we complement the existing geometric analysis of this phenomenon with an information theoretic alternative. Lower bounds are derived for the mutual information between an input and hidden layer outputs. Using a mean field analysis we are able to provide analytic lower bounds as functions of network weight and bias variances as well as the choice of nonlinear activation. These results show that initialisations known to be optimal from a training point of view are also superior from a mutual information perspective.


Variational Information Maximization for Feature Selection

Feature selection is one of the most fundamental problems in machine lea...

Information-Theoretic Bayes Risk Lower Bounds for Realizable Models

We derive information-theoretic lower bounds on the Bayes risk and gener...

Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channels

In deep learning, neural networks serve as noisy channels between input ...

Softmax Is Not an Artificial Trick: An Information-Theoretic View of Softmax in Neural Networks

Despite great popularity of applying softmax to map the non-normalised o...

Entropy and mutual information in models of deep neural networks

We examine a class of deep learning models with a tractable method to co...

Communication in Plants: Comparison of Multiple Action Potential and Mechanosensitive Signals with Experiments

Both action potentials and mechanosensitive signalling are an important ...

Doing the impossible: Why neural networks can be trained at all

As deep neural networks grow in size, from thousands to millions to bill...