A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions

10/19/2020
by   Wei Deng, et al.
0

We propose an adaptively weighted stochastic gradient Langevin dynamics algorithm (SGLD), so-called contour stochastic gradient Langevin dynamics (CSGLD), for Bayesian learning in big data statistics. The proposed algorithm is essentially a scalable dynamic importance sampler, which automatically flattens the target distribution such that the simulation for a multi-modal distribution can be greatly facilitated. Theoretically, we prove a stability condition and establish the asymptotic convergence of the self-adapting parameter to a unique fixed-point, regardless of the non-convexity of the original energy function; we also present an error analysis for the weighted averaging estimators. Empirically, the CSGLD algorithm is tested on multiple benchmark datasets including CIFAR10 and CIFAR100. The numerical results indicate its superiority over the existing state-of-the-art algorithms in training deep neural networks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/20/2022

Interacting Contour Stochastic Gradient Langevin Dynamics

We propose an interacting contour stochastic gradient Langevin dynamics ...
research
10/04/2019

Nonasymptotic estimates for Stochastic Gradient Langevin Dynamics under local conditions in nonconvex optimization

Within the context of empirical risk minimization, see Raginsky, Rakhlin...
research
06/29/2017

A Fixed-Point of View on Gradient Methods for Big Data

Interpreting gradient methods as fixed-point iterations, we provide a de...
research
12/23/2015

Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks

Effective training of deep neural networks suffers from two main issues....
research
05/04/2021

On the stability of the stochastic gradient Langevin algorithm with dependent data stream

We prove, under mild conditions, that the stochastic gradient Langevin d...
research
08/16/2022

On the generalization of learning algorithms that do not converge

Generalization analyses of deep learning typically assume that the train...
research
10/24/2022

Langevin dynamics based algorithm e-THεO POULA for stochastic optimization problems with discontinuous stochastic gradient

We introduce a new Langevin dynamics based algorithm, called e-THεO POUL...

Please sign up or login with your details

Forgot password? Click here to reset