A Lower Bound for the Optimization of Finite Sums

10/02/2014
by   Alekh Agarwal, et al.
0

This paper presents a lower bound for optimizing a finite sum of n functions, where each function is L-smooth and the sum is μ-strongly convex. We show that no algorithm can reach an error ϵ in minimizing all functions from this class in fewer than Ω(n + √(n(κ-1))(1/ϵ)) iterations, where κ=L/μ is a surrogate condition number. We then compare this lower bound to upper bounds for recently developed methods specializing to this setting. When the functions involved in this sum are not arbitrary, but based on i.i.d. random data, then we further contrast these complexity results with those for optimal first-order methods to directly optimize the sum. The conclusion we draw is that a lot of caution is necessary for an accurate comparison, and identify machine learning scenarios where the new methods help computationally.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/22/2019

A General Analysis Framework of Lower Complexity Bounds for Finite-Sum Optimization

This paper studies the lower bound complexity for the optimization probl...
research
10/17/2020

Tight Lower Complexity Bounds for Strongly Convex Finite-Sum Optimization

Finite-sum optimization plays an important role in the area of machine l...
research
03/08/2021

On the Oracle Complexity of Higher-Order Smooth Non-Convex Finite-Sum Optimization

We prove lower bounds for higher-order methods in smooth non-convex fini...
research
02/03/2020

A lower bound on the number of inequivalent APN functions

In this paper, we establish a lower bound on the total number of inequiv...
research
05/09/2018

Dispersion Bound for the Wyner-Ahlswede-Körner Network via Reverse Hypercontractivity on Types

Using the functional-entropic duality and the reverse hypercontractivity...
research
04/20/2018

On the Location of the Minimizer of the Sum of Two Strongly Convex Functions

The problem of finding the minimizer of a sum of convex functions is cen...
research
04/20/2018

On the Location of the Minimizer of the Sum of Strongly Convex Functions

The problem of finding the minimizer of a sum of convex functions is cen...

Please sign up or login with your details

Forgot password? Click here to reset