MINE: Mutual Information Neural Estimation

01/12/2018
by   Ishmael Belghazi, et al.
0

We argue that the estimation of the mutual information between high dimensional continuous random variables is achievable by gradient descent over neural networks. This paper presents a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size. MINE is back-propable and we prove that it is strongly consistent. We illustrate a handful of applications in which MINE is succesfully applied to enhance the property of generative models in both unsupervised and supervised settings. We apply our framework to estimate the information bottleneck, and apply it in tasks related to supervised classification problems. Our results demonstrate substantial added flexibility and improvement in these settings.

READ FULL TEXT
research
05/04/2020

Renormalized Mutual Information for Extraction of Continuous Features

We derive a well-defined renormalized version of mutual information that...
research
01/17/2020

Exact Information Bottleneck with Invertible Neural Networks: Getting the Best of Discriminative and Generative Modeling

The Information Bottleneck (IB) principle offers a unified approach to m...
research
05/08/2019

Data-Efficient Mutual Information Neural Estimator

Measuring Mutual Information (MI) between high-dimensional, continuous, ...
research
05/03/2020

Mutual Information Gradient Estimation for Representation Learning

Mutual Information (MI) plays an important role in representation learni...
research
11/16/2020

Regularized Mutual Information Neural Estimation

With the variational lower bound of mutual information (MI), the estimat...
research
06/16/2016

Estimating mutual information in high dimensions via classification error

Multivariate pattern analyses approaches in neuroimaging are fundamental...
research
05/08/2023

High-Dimensional Smoothed Entropy Estimation via Dimensionality Reduction

We study the problem of overcoming exponential sample complexity in diff...

Please sign up or login with your details

Forgot password? Click here to reset