Direct Acceleration of SAGA using Sampled Negative Momentum

06/28/2018
by   Kaiwen Zhou, et al.
0

Variance reduction is a simple and effective technique that accelerates convex (or non-convex) stochastic optimization. Among existing variance reduction methods, SVRG and SAGA adopt unbiased gradient estimators and have become the most popular variance reduction methods in recent years. Although various accelerated variants of SVRG (e.g., Katyusha, Acc-Prox-SVRG) have been proposed, the direct acceleration of SAGA still remains unknown. In this paper, we propose a direct accelerated variant of SAGA using Sampled Negative Momentum (SSNM), which achieves the best known oracle complexities for strongly convex problems. Consequently, our work fills the void of direct accelerated SAGA.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/07/2018

ASVRG: Accelerated Proximal SVRG

This paper proposes an accelerated proximal stochastic variance reduced ...
research
03/18/2016

Katyusha: The First Direct Acceleration of Stochastic Gradient Methods

Nesterov's momentum trick is famously known for accelerating gradient de...
research
06/10/2018

Dissipativity Theory for Accelerating Stochastic Variance Reduction: A Unified Analysis of SVRG and Katyusha Using Semidefinite Programs

Techniques for reducing the variance of gradient estimates used in stoch...
research
07/10/2017

Accelerated Stochastic Power Iteration

Principal component analysis (PCA) is one of the most powerful tools in ...
research
08/18/2020

Accelerated Zeroth-Order Momentum Methods from Mini to Minimax Optimization

In the paper, we propose a new accelerated zeroth-order momentum (Acc-ZO...
research
05/28/2019

Direct Nonlinear Acceleration

Optimization acceleration techniques such as momentum play a key role in...
research
08/30/2020

Momentum-based Accelerated Mirror Descent Stochastic Approximation for Robust Topology Optimization under Stochastic Loads

Robust topology optimization (RTO) improves the robustness of designs wi...

Please sign up or login with your details

Forgot password? Click here to reset