On the Oracle Complexity of Higher-Order Smooth Non-Convex Finite-Sum Optimization

03/08/2021
by   Nicolas Emmenegger, et al.
0

We prove lower bounds for higher-order methods in smooth non-convex finite-sum optimization. Our contribution is threefold: We first show that a deterministic algorithm cannot profit from the finite-sum structure of the objective, and that simulating a pth-order regularized method on the whole function by constructing exact gradient information is optimal up to constant factors. We further show lower bounds for randomized algorithms and compare them with the best known upper bounds. To address some gaps between the bounds, we propose a new second-order smoothness assumption that can be seen as an analogue of the first-order mean-squared smoothness assumption. We prove that it is sufficient to ensure state-of-the-art convergence guarantees, while allowing for a sharper lower bound.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/17/2020

Tight Lower Complexity Bounds for Strongly Convex Finite-Sum Optimization

Finite-sum optimization plays an important role in the area of machine l...
research
01/31/2019

Lower Bounds for Smooth Nonconvex Finite-Sum Optimization

Smooth finite-sum optimization has been widely studied in both convex an...
research
06/14/2020

Exploiting Higher Order Smoothness in Derivative-free Optimization and Continuous Bandits

We study the problem of zero-order optimization of a strongly convex fun...
research
06/21/2023

Optimal Algorithms for Stochastic Bilevel Optimization under Relaxed Smoothness Conditions

Stochastic Bilevel optimization usually involves minimizing an upper-lev...
research
10/02/2014

A Lower Bound for the Optimization of Finite Sums

This paper presents a lower bound for optimizing a finite sum of n funct...
research
03/15/2021

Lower Complexity Bounds of Finite-Sum Optimization Problems: The Results and Construction

The contribution of this paper includes two aspects. First, we study the...
research
04/04/2018

Tight Query Complexity Lower Bounds for PCA via Finite Sample Deformed Wigner Law

We prove a query complexity lower bound for approximating the top r dime...

Please sign up or login with your details

Forgot password? Click here to reset