On the Global Linear Convergence of Frank-Wolfe Optimization Variants

11/18/2015
by   Simon Lacoste-Julien, et al.
0

The Frank-Wolfe (FW) optimization algorithm has lately re-gained popularity thanks in particular to its ability to nicely handle the structured constraints appearing in machine learning applications. However, its convergence rate is known to be slow (sublinear) when the solution lies at the boundary. A simple less-known fix is to add the possibility to take 'away steps' during optimization, an operation that importantly does not require a feasibility oracle. In this paper, we highlight and clarify several variants of the Frank-Wolfe optimization algorithm that have been successfully applied in practice: away-steps FW, pairwise FW, fully-corrective FW and Wolfe's minimum norm point algorithm, and prove for the first time that they all enjoy global linear convergence, under a weaker condition than strong convexity of the objective. The constant in the convergence rate has an elegant interpretation as the product of the (classical) condition number of the function with a novel geometric quantity that plays the role of a 'condition number' of the constraint set. We provide pointers to where these algorithms have made a difference in practice, in particular with the flow polytope, the marginal polytope and the base polytope for submodular optimization.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/03/2013

A Novel Frank-Wolfe Algorithm. Analysis and Applications to Large-Scale SVM Training

Recently, there has been a renewed interest in the machine learning comm...
research
03/20/2022

Convergence rates of the stochastic alternating algorithm for bi-objective optimization

Stochastic alternating algorithms for bi-objective optimization are cons...
research
03/20/2018

Frank-Wolfe with Subsampling Oracle

We analyze two novel randomized variants of the Frank-Wolfe (FW) or cond...
research
01/27/2019

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the O(1/T) Convergence Rate

Stochastic approximation (SA) is a classical approach for stochastic con...
research
02/11/2020

Self-concordant analysis of Frank-Wolfe algorithms

Projection-free optimization via different variants of the Frank-Wolfe (...
research
04/28/2021

FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizer by Strong Convexity

The AdaBelief algorithm demonstrates superior generalization ability to ...
research
11/25/2019

Dragonfly Algorithm and its Applications in Applied Science – Survey

One of the most recently developed heuristic optimization algorithms is ...

Please sign up or login with your details

Forgot password? Click here to reset