SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives

07/01/2014
by   Aaron Defazio, et al.
1

In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for composite objectives where a proximal operator is used on the regulariser. Unlike SDCA, SAGA supports non-strongly convex problems directly, and is adaptive to any inherent strong convexity of the problem. We give experimental results showing the effectiveness of our method.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/11/2019

General Proximal Incremental Aggregated Gradient Algorithms: Better and Novel Results under General Scheme

The incremental aggregated gradient algorithm is popular in network opti...
research
02/23/2023

A subgradient method with constant step-size for ℓ_1-composite optimization

Subgradient methods are the natural extension to the non-smooth case of ...
research
02/19/2020

A Unified Convergence Analysis for Shuffling-Type Gradient Methods

In this paper, we provide a unified convergence analysis for a class of ...
research
08/11/2023

Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reduction

The recently proposed stochastic Polyak stepsize (SPS) and stochastic li...
research
07/10/2014

Finito: A Faster, Permutable Incremental Gradient Method for Big Data Problems

Recent advances in optimization theory have shown that smooth strongly c...
research
04/20/2022

Homogenization with quasistatic Tresca's friction law: qualitative and quantitative results

The problems of frictional contacts are the key to the investigation of ...
research
01/15/2015

The Fast Convergence of Incremental PCA

We consider a situation in which we see samples in R^d drawn i.i.d. from...

Please sign up or login with your details

Forgot password? Click here to reset