Information Theoretic Lower Bounds on Negative Log Likelihood

04/12/2019
by   Luis A. Lastras, et al.
0

In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equivalently a lower bound on the negative log likelihood) that the model can assign to some data as one varies the prior and/or the likelihood function in a latent variable model. The core of our contribution is to formally show that the problem of optimizing priors in latent variable models is exactly an instance of the variational optimization problem that information theorists solve when computing rate-distortion functions, and then to use this to derive a lower bound on negative log likelihood. Moreover, we will show that if changing the prior can improve the log likelihood, then there is a way to change the likelihood function instead and attain the same log likelihood, and thus rate-distortion theory is of relevance to both optimizing priors as well as optimizing likelihood functions. We will experimentally argue for the usefulness of quantities derived from rate-distortion theory in latent variable modeling by applying them to a problem in image modeling.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/12/2018

Improving latent variable descriptiveness with AutoGen

Powerful generative models, particularly in Natural Language Modelling, ...
research
06/09/2020

Super-resolution Variational Auto-Encoders

The framework of variational autoencoders (VAEs) provides a principled m...
research
01/10/2019

Preventing Posterior Collapse with delta-VAEs

Due to the phenomenon of "posterior collapse," current latent variable g...
research
02/14/2012

Lipschitz Parametrization of Probabilistic Graphical Models

We show that the log-likelihood of several probabilistic graphical model...
research
12/23/2015

Latent Variable Modeling with Diversity-Inducing Mutual Angular Regularization

Latent Variable Models (LVMs) are a large family of machine learning mod...
research
04/15/2019

Exact Rate-Distortion in Autoencoders via Echo Noise

Compression is at the heart of effective representation learning. Howeve...
research
04/24/2018

Rate-Distortion Theory for General Sets and Measures

This paper is concerned with a rate-distortion theory for sequences of i...

Please sign up or login with your details

Forgot password? Click here to reset