Log In Sign Up

Information Theory Measures via Multidimensional Gaussianization

by   Valero Laparra, et al.

Information theory is an outstanding framework to measure uncertainty, dependence and relevance in data and systems. It has several desirable properties for real world applications: it naturally deals with multivariate data, it can handle heterogeneous data types, and the measures can be interpreted in physical units. However, it has not been adopted by a wider audience because obtaining information from multidimensional data is a challenging problem due to the curse of dimensionality. Here we propose an indirect way of computing information based on a multivariate Gaussianization transform. Our proposal mitigates the difficulty of multivariate density estimation by reducing it to a composition of tractable (marginal) operations and simple linear transformations, which can be interpreted as a particular deep neural network. We introduce specific Gaussianization-based methodologies to estimate total correlation, entropy, mutual information and Kullback-Leibler divergence. We compare them to recent estimators showing the accuracy on synthetic data generated from different multivariate distributions. We made the tools and datasets publicly available to provide a test-bed to analyze future methodologies. Results show that our proposal is superior to previous estimators particularly in high-dimensional scenarios; and that it leads to interesting insights in neuroscience, geoscience, computer vision, and machine learning.


page 8

page 11

page 15

page 16


Gaussianizing the Earth: Multidimensional Information Measures for Earth Data Analysis

Information theory is an excellent framework for analyzing Earth system ...

Estimators for Multivariate Information Measures in General Probability Spaces

Information theoretic quantities play an important role in various setti...

Maximizing Multivariate Information with Error-Correcting Codes

Multivariate mutual information provides a conceptual framework for char...

Information Theory in Density Destructors

Density destructors are differentiable and invertible transforms that ma...

ISDE : Independence Structure Density Estimation

Density estimation appears as a subroutine in many learning procedures, ...

Parametric Copula-GP model for analyzing multidimensional neuronal and behavioral relationships

One of the main challenges in current systems neuroscience is the analys...

KNIFE: Kernelized-Neural Differential Entropy Estimation

Mutual Information (MI) has been widely used as a loss regularizer for t...

Code Repositories


Rotation-Based Iterative Gaussianization algorithm.

view repo


Rotation-Based Iterative Gaussianization.

view repo


Gaussianization for Earth Observation Data.

view repo