DeepAI AI Chat
Log In Sign Up

SPAN: A Stochastic Projected Approximate Newton Method

by   Xunpeng Huang, et al.

Second-order optimization methods have desirable convergence properties. However, the exact Newton method requires expensive computation for the Hessian and its inverse. In this paper, we propose SPAN, a novel approximate and fast Newton method. SPAN computes the inverse of the Hessian matrix via low-rank approximation and stochastic Hessian-vector products. Our experiments on multiple benchmark datasets demonstrate that SPAN outperforms existing first-order and second-order optimization methods in terms of the convergence wall-clock time. Furthermore, we provide a theoretical analysis of the per-iteration complexity, the approximation error, and the convergence rate. Both the theoretical analysis and experimental results show that our proposed method achieves a better trade-off between the convergence rate and the per-iteration efficiency.


page 1

page 2

page 3

page 4


A Unifying Framework for Convergence Analysis of Approximate Newton Methods

Many machine learning models are reformulated as optimization problems. ...

Second-order Neural Network Training Using Complex-step Directional Derivative

While the superior performance of second-order optimization methods such...

Optimization methods for achieving high diffraction efficiency with perfect electric conducting gratings

This work presents the implementation, analysis, and convergence study o...

Newton-LESS: Sparsification without Trade-offs for the Sketched Newton Update

In second-order optimization, a potential bottleneck can be computing th...

Practical Inexact Proximal Quasi-Newton Method with Global Complexity Analysis

Recently several methods were proposed for sparse optimization which mak...

A Gauss-Newton Method for Markov Decision Processes

Approximate Newton methods are a standard optimization tool which aim to...

A Distributed Second-Order Algorithm You Can Trust

Due to the rapid growth of data and computational resources, distributed...