Denoising Score Matching with Random Fourier Features

01/13/2021
by   Tsimboy Olga, et al.
2

The density estimation is one of the core problems in statistics. Despite this, existing techniques like maximum likelihood estimation are computationally inefficient due to the intractability of the normalizing constant. For this reason an interest to score matching has increased being independent on the normalizing constant. However, such estimator is consistent only for distributions with the full space support. One of the approaches to make it consistent is to add noise to the input data which is called Denoising Score Matching. In this work we derive analytical expression for the Denoising Score matching using the Kernel Exponential Family as a model distribution. The usage of the kernel exponential family is motivated by the richness of this class of densities. To tackle the computational complexity we use Random Fourier Features based approximation of the kernel function. The analytical expression allows to drop additional regularization terms based on the higher-order derivatives as they are already implicitly included. Moreover, the obtained expression explicitly depends on the noise variance, so the validation loss can be straightforwardly used to tune the noise level. Along with benchmark experiments, the model was tested on various synthetic distributions to study the behaviour of the model in different cases. The empirical study shows comparable quality to the competing approaches, while the proposed method being computationally faster. The latter one enables scaling up to complex high-dimensional data.

READ FULL TEXT

page 12

page 13

page 27

page 28

research
10/17/2019

Annealed Denoising Score Matching: Learning Energy-Based Models in High-Dimensional Spaces

Energy-Based Models (EBMs) outputs unmormalized log-probability values g...
research
11/20/2018

Learning deep kernels for exponential family densities

The kernel exponential family is a rich class of distributions,which can...
research
05/21/2018

Deep Energy Estimator Networks

Density estimation is a fundamental problem in statistical learning. Thi...
research
06/03/2023

Provable benefits of score matching

Score matching is an alternative to maximum likelihood (ML) for estimati...
research
11/06/2018

Kernel Exponential Family Estimation via Doubly Dual Embedding

We investigate penalized maximum log-likelihood estimation for exponenti...
research
12/17/2021

Heavy-tailed denoising score matching

Score-based model research in the last few years has produced state of t...
research
09/15/2022

Towards Healing the Blindness of Score Matching

Score-based divergences have been widely used in machine learning and st...

Please sign up or login with your details

Forgot password? Click here to reset