DeepAI AI Chat
Log In Sign Up

Low probability states, data statistics, and entropy estimation

by   Damián G. Hernández, et al.

A fundamental problem in analysis of complex systems is getting a reliable estimate of entropy of their probability distributions over the state space. This is difficult because unsampled states can contribute substantially to the entropy, while they do not contribute to the Maximum Likelihood estimator of entropy, which replaces probabilities by the observed frequencies. Bayesian estimators overcome this obstacle by introducing a model of the low-probability tail of the probability distribution. Which statistical features of the observed data determine the model of the tail, and hence the output of such estimators, remains unclear. Here we show that well-known entropy estimators for probability distributions on discrete state spaces model the structure of the low probability tail based largely on few statistics of the data: the sample size, the Maximum Likelihood estimate, the number of coincidences among the samples, the dispersion of the coincidences. We derive approximate analytical entropy estimators for undersampled distributions based on these statistics, and we use the results to propose an intuitive understanding of how the Bayesian entropy estimators work.


page 6

page 8


Estimating the Entropy of Linguistic Distributions

Shannon entropy is often a quantity of interest to linguists studying th...

Credal Networks under Maximum Entropy

We apply the principle of maximum entropy to select a unique joint proba...

Unification of field theory and maximum entropy methods for learning probability densities

The need to estimate smooth probability distributions (a.k.a. probabilit...

Maximum Entropy Principle Analysis in Network Systems with Short-time Recordings

In many realistic systems, maximum entropy principle (MEP) analysis prov...

Maximum Entropy competes with Maximum Likelihood

Maximum entropy (MAXENT) method has a large number of applications in th...

Outperforming Good-Turing: Preliminary Report

Estimating a large alphabet probability distribution from a limited numb...

Optimal Approximate Sampling from Discrete Probability Distributions

This paper addresses a fundamental problem in random variate generation:...