DeepAI AI Chat
Log In Sign Up

A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Information

by   Hardik Tankaria, et al.
Kyoto University

In this paper, we consider to improve the stochastic variance reduce gradient (SVRG) method via incorporating the curvature information of the objective function. We propose to reduce the variance of stochastic gradients using the computationally efficient Barzilai-Borwein (BB) method by incorporating it into the SVRG. We also incorporate a BB-step size as its variant. We prove its linear convergence theorem that works not only for the proposed method but also for the other existing variants of SVRG with second-order information. We conduct the numerical experiments on the benchmark datasets and show that the proposed method with constant step size performs better than the existing variance reduced methods for some test problems.


page 1

page 2

page 3

page 4


An Analysis of Stochastic Variance Reduced Gradient for Linear Inverse Problems

Stochastic variance reduced gradient (SVRG) is a popular variance reduct...

On stochastic mirror descent with interacting particles: convergence properties and variance reduction

An open problem in optimization with noisy information is the computatio...

AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods

We present an adaptive stochastic variance reduced method with an implic...

Newtonian Monte Carlo: single-site MCMC meets second-order gradient methods

Single-site Markov Chain Monte Carlo (MCMC) is a variant of MCMC in whic...

SP2: A Second Order Stochastic Polyak Method

Recently the "SP" (Stochastic Polyak step size) method has emerged as a ...

Fast Gradient Methods with Alignment for Symmetric Linear Systems without Using Cauchy Step

The performance of gradient methods has been considerably improved by th...

Learning the Step-size Policy for the Limited-Memory Broyden-Fletcher-Goldfarb-Shanno Algorithm

We consider the problem of how to learn a step-size policy for the Limit...