DeepAI AI Chat
Log In Sign Up

A Generic Approach for Escaping Saddle points

09/05/2017
by   Sashank J Reddi, et al.
MIT
smola.org
Inria
Carnegie Mellon University
0

A central challenge to using first-order methods for optimizing nonconvex problems is the presence of saddle points. First-order methods often get stuck at saddle points, greatly deteriorating their performance. Typically, to escape from saddles one has to use second-order methods. However, most works on second-order methods rely extensively on expensive Hessian-based computations, making them impractical in large-scale settings. To tackle this challenge, we introduce a generic framework that minimizes Hessian based computations while at the same time provably converging to second-order critical points. Our framework carefully alternates between a first-order and a second-order subroutine, using the latter only close to saddle points, and yields convergence results competitive to the state-of-the-art. Empirical results suggest that our strategy also enjoys a good practical performance.

READ FULL TEXT

page 1

page 2

page 3

page 4

10/04/2022

Zeroth-Order Negative Curvature Finding: Escaping Saddle Points without Gradients

We consider escaping saddle points of nonconvex problems where only the ...
05/01/2019

Stabilized SVRG: Simple Variance Reduction for Nonconvex Optimization

Variance reduction techniques like SVRG provide simple and fast algorith...
05/23/2018

Second-Order Occlusion-Aware Volumetric Radiance Caching

We present a second-order gradient analysis of light transport in partic...
05/03/2014

Supervised Descent Method for Solving Nonlinear Least Squares Problems in Computer Vision

Many computer vision problems (e.g., camera calibration, image alignment...
02/16/2020

Distributed Averaging Methods for Randomized Second Order Optimization

We consider distributed optimization problems where forming the Hessian ...
07/02/2020

Debiasing Distributed Second Order Optimization with Surrogate Sketching and Scaled Regularization

In distributed second order optimization, a standard strategy is to aver...
02/17/2023

A Legendre-Gauss Pseudospectral Collocation Method for Trajectory Optimization in Second Order Systems

Pseudospectral collocation methods have proven to be powerful tools to s...