Efficient Estimation of Mutual Information for Strongly Dependent Variables

11/07/2014
by   Shuyang Gao, et al.
0

We demonstrate that a popular class of nonparametric mutual information (MI) estimators based on k-nearest-neighbor graphs requires number of samples that scales exponentially with the true MI. Consequently, accurate estimation of MI between two strongly dependent variables is possible only for prohibitively large sample size. This important yet overlooked shortcoming of the existing estimators is due to their implicit reliance on local uniformity of the underlying joint distribution. We introduce a new estimator that is robust to local non-uniformity, works well with limited data, and is able to capture relationship strengths over many orders of magnitude. We demonstrate the superior performance of the proposed estimator on both synthetic and real-world data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/02/2017

Geometric k-nearest neighbor estimation of entropy and mutual information

Like most nonparametric estimators of information functionals involving ...
research
04/11/2016

Demystifying Fixed k-Nearest Neighbor Information Estimators

Estimating mutual information from i.i.d. samples drawn from an unknown ...
research
03/09/2010

Estimation of Rényi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs

We present simple and computationally efficient nonparametric estimators...
research
09/20/2016

Robust Estimation of Multiple Inlier Structures

The robust estimator presented in this paper processes each structure in...
research
10/31/2022

A robust estimator of mutual information for deep learning interpretability

We develop the use of mutual information (MI), a well-established metric...
research
09/07/2016

Breaking the Bandwidth Barrier: Geometrical Adaptive Entropy Estimation

Estimators of information theoretic measures such as entropy and mutual ...
research
04/30/2018

On the Effect of Suboptimal Estimation of Mutual Information in Feature Selection and Classification

This paper introduces a new property of estimators of the strength of st...

Please sign up or login with your details

Forgot password? Click here to reset