On the Maximum Mutual Information Capacity of Neural Architectures

06/10/2020
by   Brandon Foggo, et al.
0

We derive the closed-form expression of the maximum mutual information - the maximum value of I(X;Z) obtainable via training - for a broad family of neural network architectures. The quantity is essential to several branches of machine learning theory and practice. Quantitatively, we show that the maximum mutual information for these families all stem from generalizations of a single catch-all formula. Qualitatively, we show that the maximum mutual information of an architecture is most strongly influenced by the width of the smallest layer of the network - the "information bottleneck" in a different sense of the phrase, and by any statistical invariances captured by the architecture.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/11/2019

Generalized Mutual Information

Mutual information is one of the essential building blocks of informatio...
research
02/01/2021

On conditional Sibson's α-Mutual Information

In this work, we analyse how to define a conditional version of Sibson's...
research
04/22/2019

Learning gradient-based ICA by neurally estimating mutual information

Several methods of estimating the mutual information of random variables...
research
12/08/2020

Mutual Information Decay Curves and Hyper-Parameter Grid Search Design for Recurrent Neural Architectures

We present an approach to design the grid searches for hyper-parameter o...
research
04/24/2022

On Finite-Time Mutual Information

Shannon-Hartley theorem can accurately calculate the channel capacity wh...
research
03/31/2022

Mutual information estimation for graph convolutional neural networks

Measuring model performance is a key issue for deep learning practitione...
research
09/29/2015

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

The mutual information is a core statistical quantity that has applicati...

Please sign up or login with your details

Forgot password? Click here to reset