Annealed Denoising Score Matching: Learning Energy-Based Models in High-Dimensional Spaces

10/17/2019
by   Zengyi Li, et al.
12

Energy-Based Models (EBMs) outputs unmormalized log-probability values given data samples. Such an estimation is essential in a variety of applications such as sample generation, denoising, sample restoration, outlier detection, Bayesian reasoning, and many more. However, standard maximum likelihood training is computationally expensive due to the requirement of sampling the model distribution. Score matching potentially alleviates this problem, and denoising score matching is a particularly convenient version. However, previous works do not produce models capable of high quality sample synthesis in high dimensional datasets from random initialization. We believe that is because the score is only matched over a single noise scale, which corresponds to a small set in high-dimensional space. To overcome this limitation, here we instead learn an energy function using denoising score matching over all noise scales. When sampled from random initialization using Annealed Langevin Dynamics and single-step denoising jump, our model produced high-quality samples comparable to state-of-the-art techniques such as GANs. The learned model also provide density information and set a new sample quality baseline in energy-based models. We further demonstrate that the proposed method generalizes well with an image inpainting task.

READ FULL TEXT

page 1

page 2

page 4

page 5

page 7

page 10

page 11

page 12

research
05/21/2018

Deep Energy Estimator Networks

Density estimation is a fundamental problem in statistical learning. Thi...
research
03/05/2023

How to Construct Energy for Images? Denoising Autoencoder Can Be Energy Based Model

Energy-based models parameterize the unnormalized log-probability of dat...
research
05/19/2023

Moment Matching Denoising Gibbs Sampling

Energy-Based Models (EBMs) offer a versatile framework for modeling comp...
research
12/17/2021

Heavy-tailed denoising score matching

Score-based model research in the last few years has produced state of t...
research
01/13/2021

Denoising Score Matching with Random Fourier Features

The density estimation is one of the core problems in statistics. Despit...
research
06/10/2021

Score Matching Model for Unbounded Data Score

Recent advance in score-based models incorporates the stochastic differe...
research
02/05/2023

Using Intermediate Forward Iterates for Intermediate Generator Optimization

Score-based models have recently been introduced as a richer framework t...

Please sign up or login with your details

Forgot password? Click here to reset