DeepAI AI Chat
Log In Sign Up

Efficient Estimation of Mutual Information for Strongly Dependent Variables

by   Shuyang Gao, et al.
USC Information Sciences Institute

We demonstrate that a popular class of nonparametric mutual information (MI) estimators based on k-nearest-neighbor graphs requires number of samples that scales exponentially with the true MI. Consequently, accurate estimation of MI between two strongly dependent variables is possible only for prohibitively large sample size. This important yet overlooked shortcoming of the existing estimators is due to their implicit reliance on local uniformity of the underlying joint distribution. We introduce a new estimator that is robust to local non-uniformity, works well with limited data, and is able to capture relationship strengths over many orders of magnitude. We demonstrate the superior performance of the proposed estimator on both synthetic and real-world data.


page 1

page 2

page 3

page 4


Geometric k-nearest neighbor estimation of entropy and mutual information

Like most nonparametric estimators of information functionals involving ...

Demystifying Fixed k-Nearest Neighbor Information Estimators

Estimating mutual information from i.i.d. samples drawn from an unknown ...

Estimation of Rényi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs

We present simple and computationally efficient nonparametric estimators...

A robust estimator of mutual information for deep learning interpretability

We develop the use of mutual information (MI), a well-established metric...

Robust Estimation of Multiple Inlier Structures

The robust estimator presented in this paper processes each structure in...

Breaking the Bandwidth Barrier: Geometrical Adaptive Entropy Estimation

Estimators of information theoretic measures such as entropy and mutual ...

Nonparanormal Information Estimation

We study the problem of using i.i.d. samples from an unknown multivariat...

Code Repositories


local non-uniformity correction for mutual information estimation

view repo