DeepAI
Log In Sign Up

Mutual Information of Neural Network Initialisations: Mean Field Approximations

02/08/2021
by   Jared Tanner, et al.
0

The ability to train randomly initialised deep neural networks is known to depend strongly on the variance of the weight matrices and biases as well as the choice of nonlinear activation. Here we complement the existing geometric analysis of this phenomenon with an information theoretic alternative. Lower bounds are derived for the mutual information between an input and hidden layer outputs. Using a mean field analysis we are able to provide analytic lower bounds as functions of network weight and bias variances as well as the choice of nonlinear activation. These results show that initialisations known to be optimal from a training point of view are also superior from a mutual information perspective.

READ FULL TEXT
06/09/2016

Variational Information Maximization for Feature Selection

Feature selection is one of the most fundamental problems in machine lea...
11/08/2021

Information-Theoretic Bayes Risk Lower Bounds for Realizable Models

We derive information-theoretic lower bounds on the Bayes risk and gener...
12/04/2022

Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channels

In deep learning, neural networks serve as noisy channels between input ...
10/07/2019

Softmax Is Not an Artificial Trick: An Information-Theoretic View of Softmax in Neural Networks

Despite great popularity of applying softmax to map the non-normalised o...
05/24/2018

Entropy and mutual information in models of deep neural networks

We examine a class of deep learning models with a tractable method to co...
11/12/2019

Communication in Plants: Comparison of Multiple Action Potential and Mechanosensitive Signals with Experiments

Both action potentials and mechanosensitive signalling are an important ...
05/13/2018

Doing the impossible: Why neural networks can be trained at all

As deep neural networks grow in size, from thousands to millions to bill...