Log In Sign Up

Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks

by   Carles Domingo Enrich, et al.

Energy-based models (EBMs) are generative models that are usually trained via maximum likelihood estimation. This approach becomes challenging in generic situations where the trained energy is nonconvex, due to the need to sample the Gibbs distribution associated with this energy. Using general Fenchel duality results, we derive variational principles dual to maximum likelihood EBMs with shallow overparametrized neural network energies, both in the active (aka feature-learning) and lazy regimes. In the active regime, this dual formulation leads to a training algorithm in which one updates concurrently the particles in the sample space and the neurons in the parameter space of the energy. We also consider a variant of this algorithm in which the particles are sometimes restarted at random samples drawn from the data set, and show that performing these restarts at every iteration step corresponds to score matching training. Using intermediate parameter setups in our dual algorithm thereby gives a way to interpolate between maximum likelihood and score matching training. These results are illustrated in simple numerical experiments.


page 1

page 2

page 3

page 4


On Energy-Based Models with Overparametrized Shallow Neural Networks

Energy-based models (EBMs) are a simple yet powerful framework for gener...

Statistical Efficiency of Score Matching: The View from Isoperimetry

Deep generative models parametrized up to a normalizing constant (e.g. e...

Maximum Likelihood Estimation for Brownian Motion Tree Models Based on One Sample

We study the problem of maximum likelihood estimation given one data sam...

Annealed Denoising Score Matching: Learning Energy-Based Models in High-Dimensional Spaces

Energy-Based Models (EBMs) outputs unmormalized log-probability values g...

Energy Flows: Towards Determinant-Free Training of Normalizing Flows

Normalizing flows are a popular approach for constructing probabilistic ...

On Predictive Information Sub-optimality of RNNs

Certain biological neurons demonstrate a remarkable capability to optima...

EBMs Trained with Maximum Likelihood are Generator Models Trained with a Self-adverserial Loss

Maximum likelihood estimation is widely used in training Energy-based mo...