Complexity of Highly Parallel Non-Smooth Convex Optimization

06/25/2019
by   Sébastien Bubeck, et al.
0

A landmark result of non-smooth convex optimization is that gradient descent is an optimal algorithm whenever the number of computed gradients is smaller than the dimension d. In this paper we study the extension of this result to the parallel optimization setting. Namely we consider optimization algorithms interacting with a highly parallel gradient oracle, that is one that can answer poly(d) gradient queries in parallel. We show that in this case gradient descent is optimal only up to Õ(√(d)) rounds of interactions with the oracle. The lower bound improves upon a decades old construction by Nemirovski which proves optimality only up to d^1/3 rounds (as recently observed by Balkanski and Singer), and the suboptimality of gradient descent after √(d) rounds was already observed by Duchi, Bartlett and Wainwright. In the latter regime we propose a new method with improved complexity, which we conjecture to be optimal. The analysis of this new method is based upon a generalized version of the recent results on optimal acceleration for highly smooth convex optimization.

READ FULL TEXT

page 1

page 2

page 3

page 4

10/05/2020

No quantum speedup over gradient descent for non-smooth convex optimization

We study the first-order convex optimization problem, where we have blac...
01/09/2020

How to trap a gradient flow

We consider the problem of finding an ε-approximate stationary point of ...
04/19/2022

Making Progress Based on False Discoveries

We consider the question of adaptive data analysis within the framework ...
11/25/2021

Negative curvature obstructs acceleration for geodesically convex optimization, even with exact first-order oracles

Hamilton and Moitra (2021) showed that, in certain regimes, it is not po...
11/05/2018

Lower Bounds for Parallel and Randomized Convex Optimization

We study the question of whether parallelization in the exploration of t...
11/01/2017

Optimizing quantum optimization algorithms via faster quantum gradient computation

We consider a generic framework of optimization algorithms based on grad...
08/12/2018

Parallelization does not Accelerate Convex Optimization: Adaptivity Lower Bounds for Non-smooth Convex Minimization

In this paper we study the limitations of parallelization in convex opti...