DeepAI AI Chat
Log In Sign Up

Low probability states, data statistics, and entropy estimation

07/03/2022
by   Damián G. Hernández, et al.
0

A fundamental problem in analysis of complex systems is getting a reliable estimate of entropy of their probability distributions over the state space. This is difficult because unsampled states can contribute substantially to the entropy, while they do not contribute to the Maximum Likelihood estimator of entropy, which replaces probabilities by the observed frequencies. Bayesian estimators overcome this obstacle by introducing a model of the low-probability tail of the probability distribution. Which statistical features of the observed data determine the model of the tail, and hence the output of such estimators, remains unclear. Here we show that well-known entropy estimators for probability distributions on discrete state spaces model the structure of the low probability tail based largely on few statistics of the data: the sample size, the Maximum Likelihood estimate, the number of coincidences among the samples, the dispersion of the coincidences. We derive approximate analytical entropy estimators for undersampled distributions based on these statistics, and we use the results to propose an intuitive understanding of how the Bayesian entropy estimators work.

READ FULL TEXT

page 6

page 8

04/04/2022

Estimating the Entropy of Linguistic Distributions

Shannon entropy is often a quantity of interest to linguists studying th...
01/16/2013

Credal Networks under Maximum Entropy

We apply the principle of maximum entropy to select a unique joint proba...
11/19/2014

Unification of field theory and maximum entropy methods for learning probability densities

The need to estimate smooth probability distributions (a.k.a. probabilit...
08/30/2018

Maximum Entropy Principle Analysis in Network Systems with Short-time Recordings

In many realistic systems, maximum entropy principle (MEP) analysis prov...
12/17/2020

Maximum Entropy competes with Maximum Likelihood

Maximum entropy (MAXENT) method has a large number of applications in th...
07/06/2018

Outperforming Good-Turing: Preliminary Report

Estimating a large alphabet probability distribution from a limited numb...
01/13/2020

Optimal Approximate Sampling from Discrete Probability Distributions

This paper addresses a fundamental problem in random variate generation:...