Stochastic L-BFGS: Improved Convergence Rates and Practical Acceleration Strategies

04/01/2017
by   Renbo Zhao, et al.
0

We revisit the stochastic limited-memory BFGS (L-BFGS) algorithm. By proposing a new framework for the convergence analysis, we prove improved convergence rates and computational complexities of the stochastic L-BFGS algorithms compared to previous works. In addition, we propose several practical acceleration strategies to speed up the empirical performance of such algorithms. We also provide theoretical analyses for most of the strategies. Experiments on large-scale logistic and ridge regression problems demonstrate that our proposed strategies yield significant improvements vis-à-vis competing state-of-the-art algorithms.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/28/2018

A Simple Stochastic Variance Reduced Algorithm with Fast Convergence Rates

Recent years have witnessed exciting progress in the study of stochastic...
research
08/02/2023

Improved convergence rates of nonparametric penalized regression under misspecified total variation

Penalties that induce smoothness are common in nonparametric regression....
research
01/07/2021

Accelerated, Optimal, and Parallel: Some Results on Model-Based Stochastic Optimization

We extend the Approximate-Proximal Point (aProx) family of model-based m...
research
10/29/2020

Convergence of Constrained Anderson Acceleration

We prove non asymptotic linear convergence rates for the constrained And...
research
06/10/2021

A Continuized View on Nesterov Acceleration for Stochastic Gradient Descent and Randomized Gossip

We introduce the continuized Nesterov acceleration, a close variant of N...
research
12/08/2019

Histogram Transform Ensembles for Large-scale Regression

We propose a novel algorithm for large-scale regression problems named h...
research
11/24/2015

Performance Limits of Stochastic Sub-Gradient Learning, Part I: Single Agent Case

In this work and the supporting Part II, we examine the performance of s...

Please sign up or login with your details

Forgot password? Click here to reset