SUCAG: Stochastic Unbiased Curvature-aided Gradient Method for Distributed Optimization

by   Hoi-To Wai, et al.

We propose and analyze a new stochastic gradient method, which we call Stochastic Unbiased Curvature-aided Gra- dient (SUCAG), for finite sum optimization problems. SUCAG constitutes an unbiased total gradient tracking technique that uses Hessian information to accelerate convergence. We an- alyze our method under the general asynchronous model of computation, in which functions are selected infinitely often, but with delays that can grow sublinearly. For strongly convex problems, we establish linear convergence for the SUCAG method. When the initialization point is sufficiently close to the optimal solution, the established convergence rate is only dependent on the condition number of the problem, making it strictly faster than the known rate for the SAGA method. Furthermore, we describe a Markov-driven approach of implementing the SUCAG method in a distributed asynchronous multi-agent setting, via gossiping along a random walk on the communication graph. We show that our analysis applies as long as the undirected graph is connected and, notably, establishes an asymptotic linear convergence rate that is robust to the graph topology. Numerical results demonstrate the merit of our algorithm over existing methods.


page 1

page 2

page 3

page 4


Curvature-aided Incremental Aggregated Gradient Method

We propose a new algorithm for finite sum optimization which we call the...

Minimizing Finite Sums with the Stochastic Average Gradient

We propose the stochastic average gradient (SAG) method for optimizing t...

On Curvature-aided Incremental Aggregated Gradient Methods

This paper studies an acceleration technique for incremental aggregated ...

Katyusha Acceleration for Convex Finite-Sum Compositional Optimization

Structured problems arise in many applications. To solve these problems,...

High-Dimensional Inference over Networks: Linear Convergence and Statistical Guarantees

We study sparse linear regression over a network of agents, modeled as a...

ASY-SONATA: Achieving Geometric Convergence for Distributed Asynchronous Optimization

Can one obtain a geometrically convergent algorithm for distributed asyn...

Convergence rate analysis and improved iterations for numerical radius computation

We analyze existing methods for computing the numerical radius and intro...