Accelerating Frank-Wolfe via Averaging Step Directions

05/24/2022
by   Zhaoyue Chen, et al.
0

The Frank-Wolfe method is a popular method in sparse constrained optimization, due to its fast per-iteration complexity. However, the tradeoff is that its worst case global convergence is comparatively slow, and importantly, is fundamentally slower than its flow rate–that is to say, the convergence rate is throttled by discretization error. In this work, we consider a modified Frank-Wolfe where the step direction is a simple weighted average of past oracle calls. This method requires very little memory and computational overhead, and provably decays this discretization error term. Numerically, we show that this method improves the convergence rate over several problems, especially after the sparse manifold has been detected. Theoretically, we show the method has an overall global convergence rate of O(1/k^p), where 0< p < 1; after manifold identification, this rate speeds to O(1/k^3p/2). We also observe that the method achieves this accelerated rate from a very early stage, suggesting a promising mode of acceleration for this family of methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/04/2023

Reducing Discretization Error in the Frank-Wolfe Method

The Frank-Wolfe algorithm is a popular method in structurally constraine...
research
10/14/2022

A Multistep Frank-Wolfe Method

The Frank-Wolfe algorithm has regained much interest in its use in struc...
research
12/10/2012

A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method

In this note, we present a new averaging technique for the projected sto...
research
04/18/2017

Accelerated Distributed Dual Averaging over Evolving Networks of Growing Connectivity

We consider the problem of accelerating distributed optimization in mult...
research
12/11/2018

On the Curved Geometry of Accelerated Optimization

In this work we propose a differential geometric motivation for Nesterov...
research
12/17/2018

Accelerating Multigrid Optimization via SESOP

A merger of two optimization frameworks is introduced: SEquential Subspa...
research
10/12/2021

Global Convergence of Triangularized Orthogonalization-free Method

This paper proves the global convergence of a triangularized orthogonali...

Please sign up or login with your details

Forgot password? Click here to reset