DeepAI
Log In Sign Up

Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks

07/11/2021
by   Carles Domingo Enrich, et al.
6

Energy-based models (EBMs) are generative models that are usually trained via maximum likelihood estimation. This approach becomes challenging in generic situations where the trained energy is nonconvex, due to the need to sample the Gibbs distribution associated with this energy. Using general Fenchel duality results, we derive variational principles dual to maximum likelihood EBMs with shallow overparametrized neural network energies, both in the active (aka feature-learning) and lazy regimes. In the active regime, this dual formulation leads to a training algorithm in which one updates concurrently the particles in the sample space and the neurons in the parameter space of the energy. We also consider a variant of this algorithm in which the particles are sometimes restarted at random samples drawn from the data set, and show that performing these restarts at every iteration step corresponds to score matching training. Using intermediate parameter setups in our dual algorithm thereby gives a way to interpolate between maximum likelihood and score matching training. These results are illustrated in simple numerical experiments.

READ FULL TEXT

page 1

page 2

page 3

page 4

04/15/2021

On Energy-Based Models with Overparametrized Shallow Neural Networks

Energy-based models (EBMs) are a simple yet powerful framework for gener...
10/03/2022

Statistical Efficiency of Score Matching: The View from Isoperimetry

Deep generative models parametrized up to a normalizing constant (e.g. e...
12/01/2021

Maximum Likelihood Estimation for Brownian Motion Tree Models Based on One Sample

We study the problem of maximum likelihood estimation given one data sam...
10/17/2019

Annealed Denoising Score Matching: Learning Energy-Based Models in High-Dimensional Spaces

Energy-Based Models (EBMs) outputs unmormalized log-probability values g...
06/14/2022

Energy Flows: Towards Determinant-Free Training of Normalizing Flows

Normalizing flows are a popular approach for constructing probabilistic ...
10/21/2019

On Predictive Information Sub-optimality of RNNs

Certain biological neurons demonstrate a remarkable capability to optima...
02/23/2021

EBMs Trained with Maximum Likelihood are Generator Models Trained with a Self-adverserial Loss

Maximum likelihood estimation is widely used in training Energy-based mo...