DeepAI
Log In Sign Up

On Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling

06/12/2020
by   Sina Molavipour, et al.
0

The estimation of mutual information (MI) or conditional mutual information (CMI) from a set of samples is a long-standing problem. A recent line of work in this area has leveraged the approximation power of artificial neural networks and has shown improvements over conventional methods. One important challenge in this new approach is the need to obtain, given the original dataset, a different set where the samples are distributed according to a specific product density function. This is particularly challenging when estimating CMI. In this paper, we introduce a new technique, based on k nearest neighbors (k-NN), to perform the resampling and derive high-confidence concentration bounds for the sample average. Then the technique is employed to train a neural network classifier and the CMI is estimated accordingly. We propose three estimators using this technique and prove their consistency, make a comparison between them and similar approaches in the literature, and experimentally show improvements in estimating the CMI in terms of accuracy and variance of the estimators.

READ FULL TEXT

page 1

page 2

page 3

page 4

10/05/2020

DEMI: Discriminative Estimator of Mutual Information

Estimating mutual information between continuous random variables is oft...
10/13/2017

Potential Conditional Mutual Information: Estimators, Properties and Applications

The conditional mutual information I(X;Y|Z) measures the average informa...
04/22/2019

Learning gradient-based ICA by neurally estimating mutual information

Several methods of estimating the mutual information of random variables...
09/07/2016

Breaking the Bandwidth Barrier: Geometrical Adaptive Entropy Estimation

Estimators of information theoretic measures such as entropy and mutual ...
10/28/2019

Locality-Sensitive Hashing for f-Divergences: Mutual Information Loss and Beyond

Computing approximate nearest neighbors in high dimensional spaces is a ...