On the Information Plane of Autoencoders

by   Nicolás I. Tapia, et al.

The training dynamics of hidden layers in deep learning are poorly understood in theory. Recently, the Information Plane (IP) was proposed to analyze them, which is based on the information-theoretic concept of mutual information (MI). The Information Bottleneck (IB) theory predicts that layers maximize relevant information and compress irrelevant information. Due to the limitations in MI estimation from samples, there is an ongoing debate about the properties of the IP for the supervised learning case. In this work, we derive a theoretical convergence for the IP of autoencoders. The theory predicts that ideal autoencoders with a large bottleneck layer size do not compress input information, whereas a small size causes compression only in the encoder layers. For the experiments, we use a Gram-matrix based MI estimator recently proposed in the literature. We propose a new rule to adjust its parameters that compensates scale and dimensionality effects. Using our proposed rule, we obtain experimental IPs closer to the theory. Our theoretical IP for autoencoders could be used as a benchmark to validate new methods to estimate MI in neural networks. In this way, experimental limitations could be recognized and corrected, helping with the ongoing debate on the supervised learning case.


page 1

page 2

page 3

page 4


Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernels

Analyzing deep neural networks (DNNs) via information plane (IP) theory ...

Understanding Autoencoders with Information Theoretic Concepts

Despite their great success in practical applications, there is still a ...

Compression phase is not necessary for generalization in representation learning

The outstanding performance of deep learning in various fields has been ...

A Critical Review of Information Bottleneck Theory and its Applications to Deep Learning

In the past decade, deep neural networks have seen unparalleled improvem...

Pathologies in information bottleneck for deterministic supervised learning

Information bottleneck (IB) is a method for extracting information from ...

Justices for Information Bottleneck Theory

This study comes as a timely response to mounting criticism of the infor...

Variational Encoders and Autoencoders : Information-theoretic Inference and Closed-form Solutions

This work develops problem statements related to encoders and autoencode...

Please sign up or login with your details

Forgot password? Click here to reset