A Theory of Usable Information Under Computational Constraints

02/25/2020
by   Yilun Xu, et al.
8

We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of the observer. The resulting predictive V-information encompasses mutual information and other notions of informativeness such as the coefficient of determination. Unlike Shannon's mutual information and in violation of the data processing inequality, V-information can be created through computation. This is consistent with deep neural networks extracting hierarchies of progressively more informative features in representation learning. Additionally, we show that by incorporating computational constraints, V-information can be reliably estimated from data even in high dimensions with PAC-style guarantees. Empirically, we demonstrate predictive V-information is more effective than mutual information for structure learning and fair representation learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/11/2019

Generalized Mutual Information

Mutual information is one of the essential building blocks of informatio...
research
10/04/2019

High Mutual Information in Representation Learning with Symmetric Variational Inference

We introduce the Mutual Information Machine (MIM), a novel formulation o...
research
07/21/2022

Deep Sufficient Representation Learning via Mutual Information

We propose a mutual information-based sufficient representation learning...
research
08/04/2022

Invariant Representations with Stochastically Quantized Neural Networks

Representation learning algorithms offer the opportunity to learn invari...
research
05/06/2019

FSMI: Fast computation of Shannon Mutual Information for information-theoretic mapping

Exploration tasks are embedded in many robotics applications, such as se...
research
07/04/2022

Representation Learning with Information Theory for COVID-19 Detection

Successful data representation is a fundamental factor in machine learni...
research
10/17/2011

Information, learning and falsification

There are (at least) three approaches to quantifying information. The fi...

Please sign up or login with your details

Forgot password? Click here to reset