DeepAI AI Chat
Log In Sign Up

Faster Gradient-Free Algorithms for Nonsmooth Nonconvex Stochastic Optimization

01/16/2023
∙
by   Lesi Chen, et al.
∙
FUDAN University
∙
0
∙

We consider the optimization problem of the form min_x ∈ℝ^d f(x) ≜𝔼_ξ [F(x; ξ)], where the component F(x;ξ) is L-mean-squared Lipschitz but possibly nonconvex and nonsmooth. The recently proposed gradient-free method requires at most 𝒪( L^4 d^3/2ϵ^-4 + Δ L^3 d^3/2δ^-1ϵ^-4) stochastic zeroth-order oracle complexity to find a (δ,ϵ)-Goldstein stationary point of objective function, where Δ = f(x_0) - inf_x ∈ℝ^d f(x) and x_0 is the initial point of the algorithm. This paper proposes a more efficient algorithm using stochastic recursive gradient estimators, which improves the complexity to 𝒪(L^3 d^3/2ϵ^-3+ Δ L^2 d^3/2δ^-1ϵ^-3).

READ FULL TEXT

page 1

page 2

page 3

page 4

∙ 12/05/2022

A Simple and Efficient Stochastic Algorithm for Decentralized Nonconvex-Strongly-Concave Minimax Optimization

This paper studies the stochastic optimization for decentralized nonconv...
∙ 06/22/2022

Projection-free Constrained Stochastic Nonconvex Optimization with State-dependent Markov Data

We study a projection-free conditional gradient-type algorithm for const...
∙ 10/04/2016

A SMART Stochastic Algorithm for Nonconvex Optimization with Applications to Robust Machine Learning

In this paper, we show how to transform any optimization problem that ar...
∙ 03/06/2018

Dimensionality Reduction for Stationary Time Series via Stochastic Nonconvex Optimization

Stochastic optimization naturally arises in machine learning. Efficient ...
∙ 10/25/2022

An Optimal Stochastic Algorithm for Decentralized Nonconvex Finite-sum Optimization

This paper studies the synchronized decentralized nonconvex optimization...
∙ 06/12/2018

Sparse Stochastic Zeroth-Order Optimization with an Application to Bandit Structured Prediction

Stochastic zeroth-order (SZO), or gradient-free, optimization allows to ...
∙ 07/05/2016

Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization

In this paper we study stochastic quasi-Newton methods for nonconvex sto...