Generalizing Information to the Evolution of Rational Belief

11/21/2019
by   Jed A. Duersch, et al.
0

Information theory provides a mathematical foundation to measure uncertainty in belief. Belief is represented by a probability distribution that captures our understanding of an outcome's plausibility. Information measures based on Shannon's concept of entropy include realization information, Kullback-Leibler divergence, Lindley's information in experiment, cross entropy, and mutual information. We derive a general theory of information from first principles that accounts for evolving belief and recovers all of these measures. Rather than simply gauging uncertainty, information is understood in this theory to measure change in belief. We may then regard entropy as the information we expect to gain upon realization of a discrete latent random variable. This theory of information is compatible with the Bayesian paradigm in which rational belief is updated as evidence becomes available. Furthermore, this theory admits novel measures of information with well-defined properties, which we explore in both analysis and experiment. This view of information illuminates the study of machine learning by allowing us to quantify information captured by a predictive model and distinguish it from residual information contained in training data. We gain related insights regarding feature selection, anomaly detection, and novel Bayesian approaches.

READ FULL TEXT
research
12/01/2020

Fractal-based belief entropy

The total uncertainty measurement of basic probability assignment (BPA) ...
research
03/01/2020

Information Theoretic Characterization of Uncertainty Distinguishes Surprise From Accuracy Signals in the Brain

Uncertainty presents a problem for both human and machine decision-mak...
research
07/15/2019

On the Polarization of Rényi Entropy

Existing polarization theories have mostly been concerned with Shannon's...
research
03/27/2013

Combination of Evidence Using the Principle of Minimum Information Gain

One of the most important aspects in any treatment of uncertain informat...
research
01/29/2019

On the negation of a Dempster-Shafer belief structure based on maximum uncertainty allocation

Probability theory and Dempster-Shafer theory are two germane theories t...
research
03/20/2013

Formal Model of Uncertainty for Possibilistic Rules

Given a universe of discourse X-a domain of possible outcomes-an experim...
research
03/03/2021

Parsimonious Inference

Bayesian inference provides a uniquely rigorous approach to obtain princ...

Please sign up or login with your details

Forgot password? Click here to reset