SPAN: A Stochastic Projected Approximate Newton Method

02/10/2020
by   Xunpeng Huang, et al.
0

Second-order optimization methods have desirable convergence properties. However, the exact Newton method requires expensive computation for the Hessian and its inverse. In this paper, we propose SPAN, a novel approximate and fast Newton method. SPAN computes the inverse of the Hessian matrix via low-rank approximation and stochastic Hessian-vector products. Our experiments on multiple benchmark datasets demonstrate that SPAN outperforms existing first-order and second-order optimization methods in terms of the convergence wall-clock time. Furthermore, we provide a theoretical analysis of the per-iteration complexity, the approximation error, and the convergence rate. Both the theoretical analysis and experimental results show that our proposed method achieves a better trade-off between the convergence rate and the per-iteration efficiency.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/27/2017

A Unifying Framework for Convergence Analysis of Approximate Newton Methods

Many machine learning models are reformulated as optimization problems. ...
research
09/15/2020

Second-order Neural Network Training Using Complex-step Directional Derivative

While the superior performance of second-order optimization methods such...
research
04/04/2020

Optimization methods for achieving high diffraction efficiency with perfect electric conducting gratings

This work presents the implementation, analysis, and convergence study o...
research
07/15/2021

Newton-LESS: Sparsification without Trade-offs for the Sketched Newton Update

In second-order optimization, a potential bottleneck can be computing th...
research
11/26/2013

Practical Inexact Proximal Quasi-Newton Method with Global Complexity Analysis

Recently several methods were proposed for sparse optimization which mak...
research
08/19/2023

Complexity Guarantees for Nonconvex Newton-MR Under Inexact Hessian Information

We consider extensions of the Newton-MR algorithm for nonconvex optimiza...
research
07/29/2015

A Gauss-Newton Method for Markov Decision Processes

Approximate Newton methods are a standard optimization tool which aim to...

Please sign up or login with your details

Forgot password? Click here to reset