Adaptive Stochastic Gradient Langevin Dynamics: Taming Convergence and Saddle Point Escape Time

05/23/2018

∙

by   Hejian Sang, et al.

∙

0

∙

share

In this paper, we propose a new adaptive stochastic gradient Langevin dynamics (ASGLD) algorithmic framework and its two specialized versions, namely adaptive stochastic gradient (ASG) and adaptive gradient Langevin dynamics(AGLD), for non-convex optimization problems. All proposed algorithms can escape from saddle points with at most O( d) iterations, which is nearly dimension-free. Further, we show that ASGLD and ASG converge to a local minimum with at most O( d/ϵ^4) iterations. Also, ASGLD with full gradients or ASGLD with a slowly linearly increasing batch size converge to a local minimum with iterations bounded by O( d/ϵ^2), which outperforms existing first-order methods.