Stochastic Damped L-BFGS with Controlled Norm of the Hessian Approximation

by   Sanae Lotfi, et al.

We propose a new stochastic variance-reduced damped L-BFGS algorithm, where we leverage estimates of bounds on the largest and smallest eigenvalues of the Hessian approximation to balance its quality and conditioning. Our algorithm, VARCHEN, draws from previous work that proposed a novel stochastic damped L-BFGS algorithm called SdLBFGS. We establish almost sure convergence to a stationary point and a complexity bound. We empirically demonstrate that VARCHEN is more robust than SdLBFGS-VR and SVRG on a modified DavidNet problem – a highly nonconvex and ill-conditioned problem that arises in the context of deep learning, and their performance is comparable on a logistic regression problem and a nonconvex support-vector machine problem.


page 1

page 2

page 3

page 4


Structured Stochastic Quasi-Newton Methods for Large-Scale Optimization Problems

In this paper, we consider large-scale finite-sum nonconvex problems ari...

An Optimal Stochastic Algorithm for Decentralized Nonconvex Finite-sum Optimization

This paper studies the synchronized decentralized nonconvex optimization...

Sample Efficient Stochastic Variance-Reduced Cubic Regularization Method

We propose a sample efficient stochastic variance-reduced cubic regulari...

Stochastic Recursive Variance-Reduced Cubic Regularization Methods

Stochastic Variance-Reduced Cubic regularization (SVRC) algorithms have ...

Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming

In this paper, we introduce a new stochastic approximation (SA) type alg...

On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms

In this paper, we study the convergence theory of a class of gradient-ba...

Please sign up or login with your details

Forgot password? Click here to reset