Fisher Information and Mutual Information Constraints

02/11/2021
by   Leighton Pate Barnes, et al.
0

We consider the processing of statistical samples X∼ P_θ by a channel p(y|x), and characterize how the statistical information from the samples for estimating the parameter θ∈ℝ^d can scale with the mutual information or capacity of the channel. We show that if the statistical model has a sub-Gaussian score function, then the trace of the Fisher information matrix for estimating θ from Y can scale at most linearly with the mutual information between X and Y. We apply this result to obtain minimax lower bounds in distributed statistical estimation problems, and obtain a tight preconstant for Gaussian mean estimation. We then show how our Fisher information bound can also imply mutual information or Jensen-Shannon divergence based distributed strong data processing inequalities.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/21/2020

Fisher information under local differential privacy

We develop data processing inequalities that describe how Fisher informa...
research
09/05/2019

LSMI-Sinkhorn: Semi-supervised Squared-Loss Mutual Information Estimation with Optimal Transport

Estimating mutual information is an important machine learning and stati...
research
03/25/2021

Directed Data-Processing Inequalities for Systems with Feedback

We present novel data-processing inequalities relating the mutual inform...
research
02/28/2019

Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information

This article introduces a model-agnostic approach to study statistical s...
research
10/26/2020

Strong Privacy and Utility Guarantee: Over-the-Air Statistical Estimation

We consider the privacy problem of statistical estimation from distribut...
research
06/11/2018

Application of information-percolation method to reconstruction problems on graphs

In this paper we propose a method of proving impossibility results based...
research
01/28/2013

Generalized Bregman Divergence and Gradient of Mutual Information for Vector Poisson Channels

We investigate connections between information-theoretic and estimation-...

Please sign up or login with your details

Forgot password? Click here to reset