Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Features

04/23/2023
by   Aleksandr Beznosikov, et al.
0

The Frank-Wolfe (FW) method is a popular approach for solving optimization problems with structured constraints that arise in machine learning applications. In recent years, stochastic versions of FW have gained popularity, motivated by large datasets for which the computation of the full gradient is prohibitively expensive. In this paper, we present two new variants of the FW algorithms for stochastic finite-sum minimization. Our algorithms have the best convergence guarantees of existing stochastic FW approaches for both convex and non-convex objective functions. Our methods do not have the issue of permanently collecting large batches, which is common to many stochastic projection-free approaches. Moreover, our second approach does not require either large batches or full deterministic gradients, which is a typical weakness of many techniques for finite-sum problems. The faster theoretical rates of our approaches are confirmed experimentally.

READ FULL TEXT

page 6

page 7

page 9

page 16

page 22

research
07/27/2016

Stochastic Frank-Wolfe Methods for Nonconvex Optimization

We study Frank-Wolfe methods for nonconvex stochastic and finite-sum opt...
research
08/30/2022

Using Taylor-Approximated Gradients to Improve the Frank-Wolfe Method for Empirical Risk Minimization

The Frank-Wolfe method has become increasingly useful in statistical and...
research
02/26/2022

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

We propose a stochastic conditional gradient method (CGM) for minimizing...
research
07/14/2021

Zeroth and First Order Stochastic Frank-Wolfe Algorithms for Constrained Optimization

This paper considers stochastic convex optimization problems with two se...
research
03/17/2022

Learning Distributionally Robust Models at Scale via Composite Optimization

To train machine learning models that are robust to distribution shifts ...
research
10/03/2019

Best-first Search Algorithm for Non-convex Sparse Minimization

Non-convex sparse minimization (NSM), or ℓ_0-constrained minimization of...
research
10/21/2020

Efficient Projection-Free Algorithms for Saddle Point Problems

The Frank-Wolfe algorithm is a classic method for constrained optimizati...

Please sign up or login with your details

Forgot password? Click here to reset