DeepAI AI Chat
Log In Sign Up

Estimating Information-Theoretic Quantities with Random Forests

by   Richard Guo, et al.

Information-theoretic quantities, such as mutual information and conditional entropy, are useful statistics for measuring the dependence between two random variables. However, estimating these quantities in a non-parametric fashion is difficult, especially when the variables are high-dimensional, a mixture of continuous and discrete values, or both. In this paper, we propose a decision forest method, Conditional Forests (CF), to estimate these quantities. By combining quantile regression forests with honest sampling, and introducing a finite sample correction, CF improves finite sample bias in a range of settings. We demonstrate through simulations that CF achieves smaller bias and variance in both low- and high-dimensional settings for estimating posteriors, conditional entropy, and mutual information. We then use CF to estimate the amount of information between neuron class and other ceulluar feautres.


page 1

page 8


Information Potential Auto-Encoders

In this paper, we suggest a framework to make use of mutual information ...

Information-Theoretic Methods for Identifying Relationships among Climate Variables

Information-theoretic quantities, such as entropy, are used to quantify ...

A robust estimator of mutual information for deep learning interpretability

We develop the use of mutual information (MI), a well-established metric...

The Most Informative Order Statistic and its Application to Image Denoising

We consider the problem of finding the subset of order statistics that c...

Estimators of Entropy and Information via Inference in Probabilistic Models

Estimating information-theoretic quantities such as entropy and mutual i...

Ranking by Dependence - A Fair Criteria

Estimating the dependences between random variables, and ranking them ac...