Random-reshuffled SARAH does not need a full gradient computations

by   Aleksandr Beznosikov, et al.

The StochAstic Recursive grAdient algoritHm (SARAH) algorithm is a variance reduced variant of the Stochastic Gradient Descent (SGD) algorithm that needs a gradient of the objective function from time to time. In this paper, we remove the necessity of a full gradient computation. This is achieved by using a randomized reshuffling strategy and aggregating stochastic gradients obtained in each epoch. The aggregated stochastic gradients serve as an estimate of a full gradient in the SARAH algorithm. We provide a theoretical analysis of the proposed approach and conclude the paper with numerical experiments that demonstrate the efficiency of this approach.


page 1

page 2

page 3

page 4


SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient

In this paper, we propose a StochAstic Recursive grAdient algoritHm (SAR...

Stochastic Variance Reduced Gradient for affine rank minimization problem

We develop an efficient stochastic variance reduced gradient descent alg...

Variance Reduction for Deep Q-Learning using Stochastic Recursive Gradient

Deep Q-learning algorithms often suffer from poor gradient estimations w...

Variance Reduction with Sparse Gradients

Variance reduction methods such as SVRG and SpiderBoost use a mixture of...

Stochastic gradient-free descents

In this paper we propose stochastic gradient-free methods and gradient-f...

Gradient Descent-Type Methods: Background and Simple Unified Convergence Analysis

In this book chapter, we briefly describe the main components that const...

Multi-kernel Passive Stochastic Gradient Algorithms

This paper develops a novel passive stochastic gradient algorithm. In pa...

Please sign up or login with your details

Forgot password? Click here to reset