Memory-Constrained Algorithms for Convex Optimization via Recursive Cutting-Planes

by   Moise Blanchard, et al.

We propose a family of recursive cutting-plane algorithms to solve feasibility problems with constrained memory, which can also be used for first-order convex optimization. Precisely, in order to find a point within a ball of radius ϵ with a separation oracle in dimension d – or to minimize 1-Lipschitz convex functions to accuracy ϵ over the unit ball – our algorithms use 𝒪(d^2/pln1/ϵ) bits of memory, and make 𝒪((Cd/pln1/ϵ)^p) oracle calls, for some universal constant C ≥ 1. The family is parametrized by p∈[d] and provides an oracle-complexity/memory trade-off in the sub-polynomial regime ln1/ϵ≫ln d. While several works gave lower-bound trade-offs (impossibility results) – we explicit here their dependence with ln1/ϵ, showing that these also hold in any sub-polynomial regime – to the best of our knowledge this is the first class of algorithms that provides a positive trade-off between gradient descent and cutting-plane methods in any regime with ϵ≤ 1/√(d). The algorithms divide the d variables into p blocks and optimize over blocks sequentially, with approximate separation vectors constructed using a variant of Vaidya's method. In the regime ϵ≤ d^-Ω(d), our algorithm with p=d achieves the information-theoretic optimal memory usage and improves the oracle-complexity of gradient descent.


page 1

page 2

page 3

page 4


Quadratic Memory is Necessary for Optimal Query Complexity in Convex Optimization: Center-of-Mass is Pareto-Optimal

We give query complexity lower bounds for convex optimization and the re...

Efficient Convex Optimization Requires Superlinear Memory

We show that any memory-constrained, first-order algorithm which minimiz...

Decomposable Non-Smooth Convex Optimization with Nearly-Linear Gradient Oracle Complexity

Many fundamental problems in machine learning can be formulated by the c...

Memory-Query Tradeoffs for Randomized Convex Optimization

We show that any randomized first-order algorithm which minimizes a d-di...

Negative curvature obstructs acceleration for geodesically convex optimization, even with exact first-order oracles

Hamilton and Moitra (2021) showed that, in certain regimes, it is not po...

An Improved Cutting Plane Method for Convex Optimization, Convex-Concave Games and its Applications

Given a separation oracle for a convex set K ⊂R^n that is contained in a...

Detecting Correlations with Little Memory and Communication

We study the problem of identifying correlations in multivariate data, u...

Please sign up or login with your details

Forgot password? Click here to reset