Score-based Generative Modeling in Latent Space

by   Arash Vahdat, et al.

Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution coverage. However, they are usually applied directly in data space and often require thousands of network evaluations for sampling. Here, we propose the Latent Score-based Generative Model (LSGM), a novel approach that trains SGMs in a latent space, relying on the variational autoencoder framework. Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space, resulting in fewer network evaluations and faster sampling. To enable training LSGMs end-to-end in a scalable and stable manner, we (i) introduce a new score-matching objective suitable to the LSGM setting, (ii) propose a novel parameterization of the score function that allows SGM to focus on the mismatch of the target distribution with respect to a simple Normal one, and (iii) analytically derive multiple techniques for variance reduction of the training objective. LSGM obtains a state-of-the-art FID score of 2.10 on CIFAR-10, outperforming all existing generative results on this dataset. On CelebA-HQ-256, LSGM is on a par with previous SGMs in sample quality while outperforming them in sampling time by two orders of magnitude. In modeling binary images, LSGM achieves state-of-the-art likelihood on the binarized OMNIGLOT dataset.


page 2

page 8

page 37

page 38

page 39

page 40

page 41

page 42


Perceptual Generative Autoencoders

Modern generative models are usually designed to match target distributi...

Elucidating the Design Space of Diffusion-Based Generative Models

We argue that the theory and practice of diffusion-based generative mode...

One-element Batch Training by Moving Window

Several deep models, esp. the generative, compare the samples from two d...

Latent Translation: Crossing Modalities by Bridging Generative Models

End-to-end optimization has achieved state-of-the-art performance on man...

Fully Spiking Variational Autoencoder

Spiking neural networks (SNNs) can be run on neuromorphic devices with u...

Representation Learning in Continuous-Time Score-Based Generative Models

Score-based methods represented as stochastic differential equations on ...

Score-Guided Intermediate Layer Optimization: Fast Langevin Mixing for Inverse Problems

We prove fast mixing and characterize the stationary distribution of the...