HOUDINI: Escaping from Moderately Constrained Saddles

05/27/2022
by   Dmitrii Avdiukhin, et al.
0

We give the first polynomial time algorithms for escaping from high-dimensional saddle points under a moderate number of constraints. Given gradient access to a smooth function f ℝ^d →ℝ we show that (noisy) gradient descent methods can escape from saddle points under a logarithmic number of inequality constraints. This constitutes the first tangible progress (without reliance on NP-oracles or altering the definitions to only account for certain constraints) on the main open question of the breakthrough work of Ge et al. who showed an analogous result for unconstrained and equality-constrained problems. Our results hold for both regular and stochastic gradient descent.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/29/2017

Gradient Descent Can Take Exponential Time to Escape Saddle Points

Although gradient descent (GD) almost always escapes saddle points asymp...
research
06/24/2023

Towards Understanding Gradient Approximation in Equality Constrained Deep Declarative Networks

We explore conditions for when the gradient of a deep declarative node c...
research
05/21/2021

Online Statistical Inference for Parameters Estimation with Linear-Equality Constraints

Stochastic gradient descent (SGD) and projected stochastic gradient desc...
research
09/28/2018

Efficiently testing local optimality and escaping saddles for ReLU networks

We provide a theoretical algorithm for checking local optimality and esc...
research
02/13/2019

Stochastic Gradient Descent Escapes Saddle Points Efficiently

This paper considers the perturbed stochastic gradient descent algorithm...
research
05/04/2018

Analysis of nonsmooth stochastic approximation: the differential inclusion approach

In this paper we address the convergence of stochastic approximation whe...
research
10/12/2022

Sampling in Constrained Domains with Orthogonal-Space Variational Gradient Descent

Sampling methods, as important inference and learning techniques, are ty...

Please sign up or login with your details

Forgot password? Click here to reset