Practical Schemes for Finding Near-Stationary Points of Convex Finite-Sums

05/25/2021
by   Kaiwen Zhou, et al.
0

The problem of finding near-stationary points in convex optimization has not been adequately studied yet, unlike other optimality measures such as minimizing function value. Even in the deterministic case, the optimal method (OGM-G, due to Kim and Fessler (2021)) has just been discovered recently. In this work, we conduct a systematic study of the algorithmic techniques in finding near-stationary points of convex finite-sums. Our main contributions are several algorithmic discoveries: (1) we discover a memory-saving variant of OGM-G based on the performance estimation problem approach (Drori and Teboulle, 2014); (2) we design a new accelerated SVRG variant that can simultaneously achieve fast rates for both minimizing gradient norm and function value; (3) we propose an adaptively regularized accelerated SVRG variant, which does not require the knowledge of some unknown initial constants and achieves near-optimal complexities. We put an emphasis on the simplicity and practicality of the new schemes, which could facilitate future developments.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/27/2019

Near-Optimal Methods for Minimizing Star-Convex Functions and Beyond

In this paper, we provide near-optimal accelerated first-order methods f...
02/07/2022

Nesterov Accelerated Shuffling Gradient Method for Convex Optimization

In this paper, we propose Nesterov Accelerated Shuffling Gradient (NASG)...
06/03/2017

Optimal Envelope Approximation in Fourier Basis with Applications in TV White Space

Lowpass envelope approximation of smooth continuous-variable signals are...
03/31/2017

Catalyst Acceleration for Gradient-Based Non-Convex Optimization

We introduce a generic scheme to solve nonconvex optimization problems u...
02/13/2019

The Complexity of Making the Gradient Small in Stochastic Convex Optimization

We give nearly matching upper and lower bounds on the oracle complexity ...
08/17/2020

A near-optimal stochastic gradient method for decentralized non-convex finite-sum optimization

This paper describes a near-optimal stochastic first-order gradient meth...
09/06/2018

Determination of Stationary Points and Their Bindings in Dataset using RBF Methods

Stationary points of multivariable function which represents some surfac...