On Energy-Based Models with Overparametrized Shallow Neural Networks

04/15/2021
by   Carles Domingo Enrich, et al.
15

Energy-based models (EBMs) are a simple yet powerful framework for generative modeling. They are based on a trainable energy function which defines an associated Gibbs measure, and they can be trained and sampled from via well-established statistical tools, such as MCMC. Neural networks may be used as energy function approximators, providing both a rich class of expressive models as well as a flexible device to incorporate data structure. In this work we focus on shallow neural networks. Building from the incipient theory of overparametrized neural networks, we show that models trained in the so-called "active" regime provide a statistical advantage over their associated "lazy" or kernel regime, leading to improved adaptivity to hidden low-dimensional structure in the data distribution, as already observed in supervised learning. Our study covers both maximum likelihood and Stein Discrepancy estimators, and we validate our theoretical results with numerical experiments on synthetic data.

READ FULL TEXT
07/11/2021

Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks

Energy-based models (EBMs) are generative models that are usually traine...
02/03/2022

Generative Flow Networks for Discrete Probabilistic Modeling

We present energy-based generative flow networks (EB-GFN), a novel proba...
10/31/2020

Asymptotic Theory of Expectile Neural Networks

Neural networks are becoming an increasingly important tool in applicati...
06/25/2020

The Gaussian equivalence of generative models for learning with two-layer neural networks

Understanding the impact of data structure on learning in neural network...
06/01/2021

The Gaussian equivalence of generative models for learning with shallow neural networks

Understanding the impact of data structure on the computational tractabi...
09/16/2022

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study

Neural tangent kernel (NTK) is a powerful tool to analyze training dynam...
03/20/2019

Implicit Generation and Generalization in Energy-Based Models

Energy based models (EBMs) are appealing due to their generality and sim...