DeepAI AI Chat
Log In Sign Up

Asymptotic Normality for Plug-in Estimators of Generalized Shannon's Entropy

by   Jialin Zhang, et al.

Shannon's entropy is one of the building blocks of information theory and an essential aspect of Machine Learning methods (e.g., Random Forests). Yet, it is only finitely defined for distributions with fast decaying tails on a countable alphabet. The unboundedness of Shannon's entropy over the general class of all distributions on an alphabet prevents its potential utility from being fully realized. To fill the void in the foundation of information theory, Zhang (2020) proposed generalized Shannon's entropy, which is finitely defined everywhere. The plug-in estimator, adopted in almost all entropy-based ML method packages, is one of the most popular approaches to estimating Shannon's entropy. The asymptotic distribution for Shannon's entropy's plug-in estimator was well studied in the existing literature. This paper studies the asymptotic properties for the plug-in estimator of generalized Shannon's entropy on countable alphabets. The developed asymptotic properties require no assumptions on the original distribution. The proposed asymptotic properties allow interval estimation and statistical tests with generalized Shannon's entropy.


page 1

page 2

page 3

page 4


Generalized Mutual Information

Mutual information is one of the essential building blocks of informatio...

On Generalized Schürmann Entropy Estimators

We present a new class of estimators of Shannon entropy for severely und...

Renyi and Shannon Entropies of Finite Mixtures of Multivariate Skew t-distributions

Shannon and Renyi entropies are quantitative measures of uncertainty in ...

Statistical Properties of the Entropy from Ordinal Patterns

The ultimate purpose of the statistical analysis of ordinal patterns is ...

Shannon Information Entropy in Heavy-ion Collisions

The general idea of information entropy provided by C.E. Shannon "hangs ...

Entropy-based test for generalized Gaussian distributions

In this paper, we provide the proof of L^2 consistency for the kth neare...