DeepAI
Log In Sign Up

A note on the evaluation of generative models

11/05/2015
by   Lucas Theis, et al.
0

Probabilistic generative models can be used for compression, denoising, inpainting, texture synthesis, semi-supervised learning, unsupervised feature learning, and other tasks. Given this wide range of applications, it is not surprising that a lot of heterogeneity exists in the way these models are formulated, trained, and evaluated. As a consequence, direct comparison between models is often difficult. This article reviews mostly known but often underappreciated properties relating to the evaluation and interpretation of generative models with a focus on image models. In particular, we show that three of the currently most commonly used criteria---average log-likelihood, Parzen window estimates, and visual fidelity of samples---are largely independent of each other when the data is high-dimensional. Good performance with respect to one criterion therefore need not imply good performance with respect to the other criteria. Our results show that extrapolation from one criterion to another is not warranted and generative models need to be evaluated directly with respect to the application(s) they were intended for. In addition, we provide examples demonstrating that Parzen window estimates should generally be avoided.

READ FULL TEXT
04/03/2017

Semi-Supervised Generation with Cluster-aware Generative Models

Deep generative models trained with large amounts of unlabelled data hav...
04/20/2020

Generative Models Regression

We use recently developed techniques in generative models, specifically ...
05/01/2017

Towards well-specified semi-supervised model-based classifiers via structural adaptation

Semi-supervised learning plays an important role in large-scale machine ...
09/25/2019

Input complexity and out-of-distribution detection with likelihood-based generative models

Likelihood-based generative models are a promising resource to detect ou...
04/12/2021

Boltzmann Tuning of Generative Models

The paper focuses on the a posteriori tuning of a generative model in or...
06/02/2021

Evaluation Metrics for Graph Generative Models: Problems, Pitfalls, and Practical Solutions

Graph generative models are a highly active branch of machine learning. ...