Breaking the Linear Iteration Cost Barrier for Some Well-known Conditional Gradient Methods Using MaxIP Data-structures

11/30/2021
by   Anshumali Shrivastava, et al.
0

Conditional gradient methods (CGM) are widely used in modern machine learning. CGM's overall running time usually consists of two parts: the number of iterations and the cost of each iteration. Most efforts focus on reducing the number of iterations as a means to reduce the overall running time. In this work, we focus on improving the per iteration cost of CGM. The bottleneck step in most CGM is maximum inner product search (MaxIP), which requires a linear scan over the parameters. In practice, approximate MaxIP data-structures are found to be helpful heuristics. However, theoretically, nothing is known about the combination of approximate MaxIP data-structures and CGM. In this work, we answer this question positively by providing a formal framework to combine the locality sensitive hashing type approximate MaxIP data-structures with CGM algorithms. As a result, we show the first algorithm, where the cost per iteration is sublinear in the number of parameters, for many fundamental optimization algorithms, e.g., Frank-Wolfe, Herding algorithm, and policy gradient.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/19/2022

Accelerating Frank-Wolfe Algorithm using Low-Dimensional and Adaptive Data Structures

In this paper, we study the problem of speeding up a type of optimizatio...
research
05/18/2021

Sublinear Least-Squares Value Iteration via Locality Sensitive Hashing

We present the first provable Least-Squares Value Iteration (LSVI) algor...
research
04/07/2022

Speeding Up Sparsification using Inner Product Search Data Structures

We present a general framework that utilizes different efficient data st...
research
05/10/2022

Programming Data Structures for Large-Scale Desktop Simulations of Complex Systems

Studying complex systems requires running large-scale simulations over m...
research
05/13/2023

Efficient Asynchronize Stochastic Gradient Algorithm with Structured Data

Deep learning has achieved impressive success in a variety of fields bec...
research
03/09/2020

Adaptive Fibonacci and Pairing Heaps

This brief note presents two adaptive heap data structures and conjectur...
research
08/12/2021

Ethereum Data Structures

Ethereum platform operates with rich spectrum of data structures and has...

Please sign up or login with your details

Forgot password? Click here to reset