Towards GAN Benchmarks Which Require Generalization

01/10/2020
by   Ishaan Gulrajani, et al.
0

For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic. We clarify a necessary condition for an evaluation metric not to behave this way: estimating the function must require a large sample from the model. In search of such a metric, we turn to neural network divergences (NNDs), which are defined in terms of a neural network trained to distinguish between distributions. The resulting benchmarks cannot be "won" by training set memorization, while still being perceptually correlated and computable only from samples. We survey past work on using NNDs for evaluation and implement an example black-box metric based on these ideas. Through experimental validation we show that it can effectively measure diversity, sample quality, and generalization.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/18/2021

Optimizing Black-box Metrics with Iterative Example Weighting

We consider learning to optimize a classification metric defined by a bl...
research
11/02/2020

Toward a Generalization Metric for Deep Generative Models

Measuring the generalization capacity of Deep Generative Models (DGMs) i...
research
08/10/2020

DQI: A Guide to Benchmark Evaluation

A `state of the art' model A surpasses humans in a benchmark B, but fail...
research
02/17/2021

How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models

Devising domain- and model-agnostic evaluation metrics for generative mo...
research
02/19/2022

Diversity aware image generation

The machine learning generative algorithms such as GAN and VAE show impr...
research
08/31/2023

Unsupervised evaluation of GAN sample quality: Introducing the TTJac Score

Evaluation metrics are essential for assessing the performance of genera...
research
04/01/2022

Estimating the Jacobian matrix of an unknown multivariate function from sample values by means of a neural network

We describe, implement and test a novel method for training neural netwo...

Please sign up or login with your details

Forgot password? Click here to reset