Stochastic Frank-Wolfe Methods for Nonconvex Optimization

07/27/2016
by   Sashank J Reddi, et al.
0

We study Frank-Wolfe methods for nonconvex stochastic and finite-sum optimization problems. Frank-Wolfe methods (in the convex case) have gained tremendous recent interest in machine learning and optimization communities due to their projection-free property and their ability to exploit structured constraints. However, our understanding of these algorithms in the nonconvex setting is fairly limited. In this paper, we propose nonconvex stochastic Frank-Wolfe methods and analyze their convergence properties. For objective functions that decompose into a finite-sum, we leverage ideas from variance reduction techniques for convex optimization to obtain new variance reduced nonconvex Frank-Wolfe methods that have provably faster convergence than the classical Frank-Wolfe method. Finally, we show that the faster convergence rates of our variance reduced methods also translate into improved convergence rates for the stochastic setting.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/19/2016

Stochastic Variance Reduction for Nonconvex Optimization

We study nonconvex finite-sum problems and analyze stochastic variance r...
research
07/07/2020

An Accelerated DFO Algorithm for Finite-sum Convex Functions

Derivative-free optimization (DFO) has recently gained a lot of momentum...
research
02/12/2021

Stochastic Gradient Langevin Dynamics with Variance Reduction

Stochastic gradient Langevin dynamics (SGLD) has gained the attention of...
research
04/23/2023

Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Features

The Frank-Wolfe (FW) method is a popular approach for solving optimizati...
research
12/26/2020

Variance Reduction on Adaptive Stochastic Mirror Descent

We study the idea of variance reduction applied to adaptive stochastic m...
research
11/20/2018

Sampling Can Be Faster Than Optimization

Optimization algorithms and Monte Carlo sampling algorithms have provide...
research
02/12/2018

Katyusha X: Practical Momentum Method for Stochastic Sum-of-Nonconvex Optimization

The problem of minimizing sum-of-nonconvex functions (i.e., convex funct...

Please sign up or login with your details

Forgot password? Click here to reset