Maximally Informative Hierarchical Representations of High-Dimensional Data

10/27/2014
by   Greg Ver Steeg, et al.
0

We consider a set of probabilistic functions of some input variables as a representation of the inputs. We present bounds on how informative a representation is about input data. We extend these bounds to hierarchical representations so that we can quantify the contribution of each layer towards capturing the information in the original data. The special form of these bounds leads to a simple, bottom-up optimization procedure to construct hierarchical representations that are also maximally informative about the data. This optimization has linear computational complexity and constant sample complexity in the number of variables. These results establish a new approach to unsupervised learning of deep representations that is both principled and practical. We demonstrate the usefulness of the approach on both synthetic and real-world data.

READ FULL TEXT

page 7

page 13

research
04/14/2023

On representations of real numbers and the computational complexity of converting between such representations

We study the computational complexity of converting one representation o...
research
07/08/2015

The Information Sieve

We introduce a new framework for unsupervised learning of representation...
research
11/17/2013

Unsupervised Learning of Invariant Representations in Hierarchical Architectures

The present phase of Machine Learning is characterized by supervised lea...
research
02/09/2023

Tree Learning: Optimal Algorithms and Sample Complexity

We study the problem of learning a hierarchical tree representation of d...
research
06/24/2020

Towards Understanding Hierarchical Learning: Benefits of Neural Representations

Deep neural networks can empirically perform efficient hierarchical lear...
research
08/12/2021

Probabilistic methods for approximate archetypal analysis

Archetypal analysis is an unsupervised learning method for exploratory d...
research
06/07/2016

Sifting Common Information from Many Variables

Measuring the relationship between any pair of variables is a rich and a...

Please sign up or login with your details

Forgot password? Click here to reset