A General Analysis Framework of Lower Complexity Bounds for Finite-Sum Optimization

by   Guangzeng Xie, et al.

This paper studies the lower bound complexity for the optimization problem whose objective function is the average of n individual smooth convex functions. We consider the algorithm which gets access to gradient and proximal oracle for each individual component. For the strongly-convex case, we prove such an algorithm can not reach an ε-suboptimal point in fewer than Ω((n+√(κ n))(1/ε)) iterations, where κ is the condition number of the objective function. This lower bound is tighter than previous results and perfectly matches the upper bound of the existing proximal incremental first-order oracle algorithm Point-SAGA. We develop a novel construction to show the above result, which partitions the tridiagonal matrix of classical examples into n groups. This construction is friendly to the analysis of proximal oracle and also could be used to general convex and average smooth cases naturally.


page 1

page 2

page 3

page 4


Lower Complexity Bounds of Finite-Sum Optimization Problems: The Results and Construction

The contribution of this paper includes two aspects. First, we study the...

A Lower Bound for the Optimization of Finite Sums

This paper presents a lower bound for optimizing a finite sum of n funct...

DIPPA: An improved Method for Bilinear Saddle Point Problems

This paper studies bilinear saddle point problems min_xmax_y g(x) + x^⊤A...

On the Complexity of Minimizing Convex Finite Sums Without Using the Indices of the Individual Functions

Recent advances in randomized incremental methods for minimizing L-smoot...

Towards Unified Acceleration of High-Order Algorithms under Hölder Continuity and Uniform Convexity

In this paper, through a very intuitive vanilla proximal method perspec...

Thinking Inside the Ball: Near-Optimal Minimization of the Maximal Loss

We characterize the complexity of minimizing max_i∈[N] f_i(x) for convex...

On the complexity of quasiconvex integer minimization problem

In this paper, we consider the class of quasiconvex functions and its pr...