DeepAI AI Chat
Log In Sign Up

AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods

by   Zheng Shi, et al.

We present an adaptive stochastic variance reduced method with an implicit approach for adaptivity. As a variant of SARAH, our method employs the stochastic recursive gradient yet adjusts step-size based on local geometry. We provide convergence guarantees for finite-sum minimization problems and show a faster convergence than SARAH can be achieved if local geometry permits. Furthermore, we propose a practical, fully adaptive variant, which does not require any knowledge of local geometry and any effort of tuning the hyper-parameters. This algorithm implicitly computes step-size and efficiently estimates local Lipschitz smoothness of stochastic functions. The numerical experiments demonstrate the algorithm's strong performance compared to its classical counterparts and other state-of-the-art first-order methods.


page 22

page 23


SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient

In this paper, we propose a StochAstic Recursive grAdient algoritHm (SAR...

A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Information

In this paper, we consider to improve the stochastic variance reduce gra...

Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimization

We propose an adaptive variance-reduction method, called AdaSpider, for ...

Adaptive Gradient Methods Converge Faster with Over-Parameterization (and you can do a line-search)

As adaptive gradient methods are typically used for training over-parame...

Implicit Temporal Differences

In reinforcement learning, the TD(λ) algorithm is a fundamental policy e...

Stochastic subgradient method converges on tame functions

This work considers the question: what convergence guarantees does the s...

Dynamical softassign and adaptive parameter tuning for graph matching

This paper studies a framework, projected fixed-point method, for graph ...

Code Repositories


Implement classical machine learning algorithm using Pytorch

view repo