DeepAI AI Chat
Log In Sign Up

Asymptotic Normality for Plug-in Estimators of Generalized Shannon's Entropy

05/02/2022
by   Jialin Zhang, et al.
0

Shannon's entropy is one of the building blocks of information theory and an essential aspect of Machine Learning methods (e.g., Random Forests). Yet, it is only finitely defined for distributions with fast decaying tails on a countable alphabet. The unboundedness of Shannon's entropy over the general class of all distributions on an alphabet prevents its potential utility from being fully realized. To fill the void in the foundation of information theory, Zhang (2020) proposed generalized Shannon's entropy, which is finitely defined everywhere. The plug-in estimator, adopted in almost all entropy-based ML method packages, is one of the most popular approaches to estimating Shannon's entropy. The asymptotic distribution for Shannon's entropy's plug-in estimator was well studied in the existing literature. This paper studies the asymptotic properties for the plug-in estimator of generalized Shannon's entropy on countable alphabets. The developed asymptotic properties require no assumptions on the original distribution. The proposed asymptotic properties allow interval estimation and statistical tests with generalized Shannon's entropy.

READ FULL TEXT

page 1

page 2

page 3

page 4

07/11/2019

Generalized Mutual Information

Mutual information is one of the essential building blocks of informatio...
11/18/2021

On Generalized Schürmann Entropy Estimators

We present a new class of estimators of Shannon entropy for severely und...
01/29/2019

Renyi and Shannon Entropies of Finite Mixtures of Multivariate Skew t-distributions

Shannon and Renyi entropies are quantitative measures of uncertainty in ...
09/15/2022

Statistical Properties of the Entropy from Ordinal Patterns

The ultimate purpose of the statistical analysis of ordinal patterns is ...
01/07/2018

Shannon Information Entropy in Heavy-ion Collisions

The general idea of information entropy provided by C.E. Shannon "hangs ...
10/13/2020

Entropy-based test for generalized Gaussian distributions

In this paper, we provide the proof of L^2 consistency for the kth neare...