No MCMC for me: Amortized sampling for fast and stable training of energy-based models

10/08/2020
by   Will Grathwohl, et al.
6

Energy-Based Models (EBMs) present a flexible and appealing way to represent uncertainty. Despite recent advances, training EBMs on high-dimensional data remains a challenging problem as the state-of-the-art approaches are costly, unstable, and require considerable tuning and domain expertise to apply successfully. In this work, we present a simple method for training EBMs at scale which uses an entropy-regularized generator to amortize the MCMC sampling typically used in EBM training. We improve upon prior MCMC-based entropy regularization methods with a fast variational approximation. We demonstrate the effectiveness of our approach by using it to train tractable likelihood models. Next, we apply our estimator to the recently proposed Joint Energy Model (JEM), where we match the original performance with faster and stable training. This allows us to extend JEM models to semi-supervised classification on tabular data from a variety of continuous domains.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 22

page 23

page 24

page 25

page 26

page 27

page 28

page 29

03/20/2019

Implicit Generation and Generalization in Energy-Based Models

Energy based models (EBMs) are appealing due to their generality and sim...
03/29/2019

On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models

This study investigates the effects Markov Chain Monte Carlo (MCMC) samp...
10/25/2020

An empirical study of domain-agnostic semi-supervised learning via energy-based models: joint-training and pre-training

A class of recent semi-supervised learning (SSL) methods heavily rely on...
11/03/2021

Hamiltonian Dynamics with Non-Newtonian Momentum for Rapid Sampling

Sampling from an unnormalized probability distribution is a fundamental ...
11/26/2021

Particle Dynamics for Learning EBMs

Energy-based modeling is a promising approach to unsupervised learning, ...
10/01/2020

VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models

Energy-based models (EBMs) have recently been successful in representing...
03/17/2020

Energy-Based Processes for Exchangeable Data

Recently there has been growing interest in modeling sets with exchangea...

Code Repositories

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.