Assessing Generative Models via Precision and Recall

05/31/2018
by   Mehdi S. M. Sajjadi, et al.
0

Recent advances in generative modeling have led to an increased interest in the study of statistical divergences as means of model comparison. Commonly used evaluation methods, such as Fréchet Inception Distance (FID), correlate well with the perceived quality of samples and are sensitive to mode dropping. However, these metrics are unable to distinguish between different failure cases since they yield one-dimensional scores. We propose a novel definition of precision and recall for distributions which disentangles the divergence into two separate dimensions. The proposed notion is intuitive, retains desirable properties, and naturally leads to an efficient algorithm that can be used to evaluate generative models. We relate this notion to total variation as well as to recent evaluation metrics such as Inception Score and FID. To demonstrate the practical utility of the proposed approach we perform an empirical study on several variants of Generative Adversarial Networks and the Variational Autoencoder. In an extensive set of experiments we show that the proposed metric is able to disentangle the quality of generated samples from the coverage of the target distribution.

READ FULL TEXT

page 8

page 12

page 14

research
11/02/2020

Toward a Generalization Metric for Deep Generative Models

Measuring the generalization capacity of Deep Generative Models (DGMs) i...
research
02/01/2023

Training Normalizing Flows with the Precision-Recall Divergence

Generative models can have distinct mode of failures like mode dropping ...
research
04/15/2019

Improved Precision and Recall Metric for Assessing Generative Models

The ability to evaluate the performance of a computational model is a vi...
research
05/14/2019

Revisiting Precision and Recall Definition for Generative Model Evaluation

In this article we revisit the definition of Precision-Recall (PR) curve...
research
05/30/2023

Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows

Achieving a balance between image quality (precision) and diversity (rec...
research
03/17/2021

Pros and Cons of GAN Evaluation Measures: New Developments

This work is an update of a previous paper on the same topic published a...
research
05/26/2019

Evaluating Generative Models Using Divergence Frontiers

Despite the tremendous progress in the estimation of generative models, ...

Please sign up or login with your details

Forgot password? Click here to reset