Stochastic Nested Variance Reduction for Nonconvex Optimization

06/20/2018
by   Dongruo Zhou, et al.
0

We study finite-sum nonconvex optimization problems, where the objective function is an average of n nonconvex functions. We propose a new stochastic gradient descent algorithm based on nested variance reduction. Compared with conventional stochastic variance reduced gradient (SVRG) algorithm that uses two reference points to construct a semi-stochastic gradient with diminishing variance in each iteration, our algorithm uses K+1 nested reference points to build a semi-stochastic gradient to further reduce its variance in each iteration. For smooth nonconvex functions, the proposed algorithm converges to an ϵ-approximate first-order stationary point (i.e., ∇ F(x)_2≤ϵ) within Õ(nϵ^-2+ϵ^-3 n^1/2ϵ^-2) number of stochastic gradient evaluations. This improves the best known gradient complexity of SVRG O(n+n^2/3ϵ^-2) and that of SCSG O(nϵ^-2+ϵ^-10/3 n^2/3ϵ^-2). For gradient dominated functions, our algorithm also achieves a better gradient complexity than the state-of-the-art algorithms.

READ FULL TEXT
06/22/2018

Finding Local Minima via Stochastic Nested Variance Reduction

We propose two algorithms that can find local minima faster than the sta...
11/08/2017

Stochastic Cubic Regularization for Fast Nonconvex Optimization

This paper proposes a stochastic variant of a classic algorithm---the cu...
06/24/2019

A Stochastic Composite Gradient Method with Incremental Variance Reduction

We consider the problem of minimizing the composition of a smooth (nonco...
06/14/2022

Lazy Queries Can Reduce Variance in Zeroth-order Optimization

A major challenge of applying zeroth-order (ZO) methods is the high quer...
10/24/2019

A nonsmooth nonconvex descent algorithm

The paper presents a new descent algorithm for locally Lipschitz continu...
08/20/2020

An Optimal Hybrid Variance-Reduced Algorithm for Stochastic Composite Nonconvex Optimization

In this note we propose a new variant of the hybrid variance-reduced pro...
07/11/2020

Solving Bayesian Risk Optimization via Nested Stochastic Gradient Estimation

In this paper, we aim to solve Bayesian Risk Optimization (BRO), which i...