Heavy-tailed denoising score matching

12/17/2021
by   Jacob Deasy, et al.
17

Score-based model research in the last few years has produced state of the art generative models by employing Gaussian denoising score-matching (DSM). However, the Gaussian noise assumption has several high-dimensional limitations, motivating a more concrete route toward even higher dimension PDF estimation in future. We outline this limitation, before extending the theory to a broader family of noising distributions – namely, the generalised normal distribution. To theoretically ground this, we relax a key assumption in (denoising) score matching theory, demonstrating that distributions which are differentiable almost everywhere permit the same objective simplification as Gaussians. For noise vector length distributions, we demonstrate favourable concentration of measure in the high-dimensional spaces prevalent in deep learning. In the process, we uncover a skewed noise vector length distribution and develop an iterative noise scaling algorithm to consistently initialise the multiple levels of noise in annealed Langevin dynamics. On the practical side, our use of heavy-tailed DSM leads to improved score estimation, controllable sampling convergence, and more balanced unconditional generative performance for imbalanced datasets.

READ FULL TEXT

page 9

page 10

page 21

page 23

research
10/17/2019

Annealed Denoising Score Matching: Learning Energy-Based Models in High-Dimensional Spaces

Energy-Based Models (EBMs) outputs unmormalized log-probability values g...
research
10/21/2022

Score-based Denoising Diffusion with Non-Isotropic Gaussian Noise Models

Generative models based on denoising diffusion techniques have led to an...
research
11/07/2022

From Denoising Diffusions to Denoising Markov Models

Denoising diffusions are state-of-the-art generative models which exhibi...
research
07/12/2019

Generative Modeling by Estimating Gradients of the Data Distribution

We introduce a new generative model where samples are produced via Lange...
research
01/13/2021

Denoising Score Matching with Random Fourier Features

The density estimation is one of the core problems in statistics. Despit...
research
04/08/2021

On tuning consistent annealed sampling for denoising score matching

Score-based generative models provide state-of-the-art quality for image...
research
05/19/2023

Moment Matching Denoising Gibbs Sampling

Energy-Based Models (EBMs) offer a versatile framework for modeling comp...

Please sign up or login with your details

Forgot password? Click here to reset