An optimal randomized incremental gradient method

07/08/2015
by   Guanghui Lan, et al.
0

In this paper, we consider a class of finite-sum convex optimization problems whose objective function is given by the summation of m (> 1) smooth components together with some other relatively simple terms. We first introduce a deterministic primal-dual gradient (PDG) method that can achieve the optimal black-box iteration complexity for solving these composite optimization problems using a primal-dual termination criterion. Our major contribution is to develop a randomized primal-dual gradient (RPDG) method, which needs to compute the gradient of only one randomly selected smooth component at each iteration, but can possibly achieve better complexity than PDG in terms of the total number of gradient evaluations. More specifically, we show that the total number of gradient evaluations performed by RPDG can be O (√(m)) times smaller, both in expectation and with high probability, than those performed by deterministic optimal first-order methods under favorable situations. We also show that the complexity of the RPDG method is not improvable by developing a new lower complexity bound for a general class of randomized methods for solving large-scale finite-sum convex optimization problems. Moreover, through the development of PDG and RPDG, we introduce a novel game-theoretic interpretation for these optimal methods for convex optimization.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/15/2017

Random gradient extrapolation for distributed and stochastic optimization

In this paper, we consider a class of finite-sum convex optimization pro...
research
06/30/2020

Conditional Gradient Methods for convex optimization with function constraints

Conditional gradient methods have attracted much attention in both machi...
research
04/17/2020

First-Order Methods for Optimal Experimental Design Problems with Bound Constraints

We consider a class of convex optimization problems over the simplex of ...
research
11/27/2018

Fast UAV Trajectory Optimization using Bilevel Optimization with Analytical Gradients

This paper presents an efficient optimization framework that solves traj...
research
07/19/2021

Revisiting the Primal-Dual Method of Multipliers for Optimisation over Centralised Networks

The primal-dual method of multipliers (PDMM) was originally designed for...
research
03/28/2023

Accelerated Cyclic Coordinate Dual Averaging with Extrapolation for Composite Convex Optimization

Exploiting partial first-order information in a cyclic way is arguably t...

Please sign up or login with your details

Forgot password? Click here to reset