Measuring Information from Moments

09/01/2021
by   Wael Alghamdi, et al.
0

We investigate the problem of representing information measures in terms of the moments of the underlying random variables. First, we derive polynomial approximations of the conditional expectation operator. We then apply these approximations to bound the best mean-square error achieved by a polynomial estimator – referred to here as the PMMSE. In Gaussian channels, the PMMSE coincides with the minimum mean-square error (MMSE) if and only if the input is either Gaussian or constant, i.e., if and only if the conditional expectation of the input of the channel given the output is a polynomial of degree at most 1. By combining the PMMSE with the I-MMSE relationship, we derive new formulas for information measures (e.g., differential entropy, mutual information) that are given in terms of the moments of the underlying random variables. As an application, we introduce estimators for information measures from data via approximating the moments in our formulas by sample moments. These estimators are shown to be asymptotically consistent and possess desirable properties, e.g., invariance to affine transformations when used to estimate mutual information.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/11/2021

Polynomial Approximations of Conditional Expectations in Scalar Gaussian Channels

We consider a channel Y=X+N where X is a random variable satisfying 𝔼[|X...
research
08/29/2021

Lower Bounds for the Minimum Mean-Square Error via Neural Network-based Estimation

The minimum mean-square error (MMSE) achievable by optimal estimation of...
research
03/01/2017

Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variables

This note is concerned with accurate and computationally efficient appro...
research
11/19/2021

Some Error Analysis for the Quantum Phase Estimation Algorithms

This paper is concerned with the phase estimation algorithm in quantum c...
research
10/05/2013

Moments and Root-Mean-Square Error of the Bayesian MMSE Estimator of Classification Error in the Gaussian Model

The most important aspect of any classifier is its error rate, because t...
research
10/01/2009

Expectation Propagation on the Maximum of Correlated Normal Variables

Many inference problems involving questions of optimality ask for the ma...
research
10/18/2017

On Least Squares Linear Regression Without Second Moment

If X and Y are real valued random variables such that the first moments ...

Please sign up or login with your details

Forgot password? Click here to reset