Can We Find Near-Approximately-Stationary Points of Nonsmooth Nonconvex Functions?

02/27/2020
by   Ohad Shamir, et al.
0

It is well-known that given a bounded, smooth nonconvex function, standard gradient-based methods can find ϵ-stationary points (where the gradient norm is less than ϵ) in O(1/ϵ^2) iterations. However, many important nonconvex optimization problems, such as those associated with training modern neural networks, are inherently not smooth, making these results inapplicable. Moreover, as recently pointed out in Zhang et al. [2020], it is generally impossible to provide finite-time guarantees for finding an ϵ-stationary point of nonsmooth functions. Perhaps the most natural relaxation of this is to find points which are near such ϵ-stationary points. In this paper, we show that even this relaxed goal is hard to obtain in general, given only black-box access to the function values and gradients. We conclude with a discussion of the result and its implications.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/14/2021

Oracle Complexity in Nonsmooth Nonconvex Optimization

It is well-known that given a smooth, bounded-from-below, and possibly n...
research
02/10/2020

On Complexity of Finding Stationary Points of Nonsmooth Nonconvex Functions

We provide the first non-asymptotic analysis for finding stationary poin...
research
09/06/2018

Determination of Stationary Points and Their Bindings in Dataset using RBF Methods

Stationary points of multivariable function which represents some surfac...
research
03/31/2017

Catalyst Acceleration for Gradient-Based Non-Convex Optimization

We introduce a generic scheme to solve nonconvex optimization problems u...
research
07/07/2022

The effect of smooth parametrizations on nonconvex optimization landscapes

We develop new tools to study the landscapes of nonconvex optimization p...
research
12/07/2022

Quantum Lower Bounds for Finding Stationary Points of Nonconvex Functions

Quantum algorithms for optimization problems are of general interest. De...
research
03/23/2023

Optimization Dynamics of Equivariant and Augmented Neural Networks

We investigate the optimization of multilayer perceptrons on symmetric d...

Please sign up or login with your details

Forgot password? Click here to reset