Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning

03/27/2020
by   Andreas Kirsch, et al.
18

The information bottleneck (IB) principle offers both a mechanism to explain how deep neural networks train and generalize, as well as a regularized objective with which to train models. However, multiple competing objectives have been proposed based on this principle. Moreover, the information-theoretic quantities in the objective are difficult to compute for large deep neural networks, and this limits its use as a training objective. In this work, we review these quantities, compare and unify previously proposed objectives and relate them to surrogate objectives more friendly to optimization. We find that these surrogate objectives allow us to apply the information bottleneck to modern neural network architectures. We demonstrate our insights on Permutation-MNIST, MNIST and CIFAR10.

READ FULL TEXT

page 3

page 10

page 18

research
04/19/2023

To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review

Deep neural networks have demonstrated remarkable performance in supervi...
research
02/10/2022

Information Flow in Deep Neural Networks

Although deep neural networks have been immensely successful, there is n...
research
03/21/2020

On Information Plane Analyses of Neural Network Classifiers – A Review

We review the current literature concerned with information plane analys...
research
10/20/2019

Towards Further Understanding of Sparse Filtering via Information Bottleneck

In this paper we examine a formalization of feature distribution learnin...
research
05/07/2021

A Critical Review of Information Bottleneck Theory and its Applications to Deep Learning

In the past decade, deep neural networks have seen unparalleled improvem...
research
07/11/2022

Bottlenecks CLUB: Unifying Information-Theoretic Trade-offs Among Complexity, Leakage, and Utility

Bottleneck problems are an important class of optimization problems that...
research
03/13/2020

What Information Does a ResNet Compress?

The information bottleneck principle (Shwartz-Ziv Tishby, 2017) sugg...

Please sign up or login with your details

Forgot password? Click here to reset