On Variational Bounds of Mutual Information

05/16/2019
by   Ben Poole, et al.
8

Estimating and optimizing Mutual Information (MI) is core to many problems in machine learning; however, bounding MI in high dimensions is challenging. To establish tractable and scalable objectives, recent work has turned to variational bounds parameterized by neural networks, but the relationships and tradeoffs between these bounds remains unclear. In this work, we unify these recent developments in a single framework. We find that the existing variational lower bounds degrade when the MI is large, exhibiting either high bias or high variance. To address this problem, we introduce a continuum of lower bounds that encompasses previous bounds and flexibly trades off bias and variance. On high-dimensional, controlled problems, we empirically characterize the bias and variance of the bounds and their gradients and demonstrate the effectiveness of our new bounds for estimation and representation learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/05/2020

DEMI: Discriminative Estimator of Mutual Information

Estimating mutual information between continuous random variables is oft...
research
06/09/2020

Neural Methods for Point-wise Dependency Estimation

Since its inception, the neural estimation of mutual information (MI) ha...
research
05/31/2023

Variational f-Divergence and Derangements for Discriminative Mutual Information Estimation

The accurate estimation of the mutual information is a crucial task in v...
research
11/17/2020

Reducing the Variance of Variational Estimates of Mutual Information by Limiting the Critic's Hypothesis Space to RKHS

Mutual information (MI) is an information-theoretic measure of dependenc...
research
05/30/2020

On lower bounds for the bias-variance trade-off

It is a common phenomenon that for high-dimensional and nonparametric st...
research
10/14/2019

Understanding the Limitations of Variational Mutual Information Estimators

Variational approaches based on neural networks are showing promise for ...
research
04/26/2023

Understanding the limitation of Total Correlation Estimation Based on Mutual Information Bounds

The total correlation(TC) is a crucial index to measure the correlation ...

Please sign up or login with your details

Forgot password? Click here to reset