Oracle Complexity of Second-Order Methods for Finite-Sum Problems

11/15/2016
by   Yossi Arjevani, et al.
0

Finite-sum optimization problems are ubiquitous in machine learning, and are commonly solved using first-order methods which rely on gradient computations. Recently, there has been growing interest in second-order methods, which rely on both gradients and Hessians. In principle, second-order methods can require much fewer iterations than first-order methods, and hold the promise for more efficient algorithms. Although computing and manipulating Hessians is prohibitive for high-dimensional problems in general, the Hessians of individual functions in finite-sum problems can often be efficiently computed, e.g. because they possess a low-rank structure. Can second-order information indeed be used to solve such problems more efficiently? In this paper, we provide evidence that the answer -- perhaps surprisingly -- is negative, at least in terms of worst-case guarantees. However, we also discuss what additional assumptions and algorithmic approaches might potentially circumvent this negative result.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/12/2016

Second-Order Stochastic Optimization for Machine Learning in Linear Time

First-order stochastic methods are the state-of-the-art in large-scale m...
research
02/20/2020

Second-order Conditional Gradients

Constrained second-order convex optimization algorithms are the method o...
research
07/31/2020

LSOS: Line-search Second-Order Stochastic optimization methods

We develop a line-search second-order algorithmic framework for optimiza...
research
03/30/2021

Quadratic and Cubic Regularisation Methods with Inexact function and Random Derivatives for Finite-Sum Minimisation

This paper focuses on regularisation methods using models up to the thir...
research
12/25/2018

Second-Order Converses via Reverse Hypercontractivity

A strong converse shows that no procedure can beat the asymptotic (as bl...
research
08/29/2023

Second-order methods for quartically-regularised cubic polynomials, with applications to high-order tensor methods

There has been growing interest in high-order tensor methods for nonconv...
research
11/27/2018

Bilinear Parameterization For Differentiable Rank-Regularization

Low rank approximation is a commonly occurring problem in many computer ...

Please sign up or login with your details

Forgot password? Click here to reset