Optimization with First-Order Surrogate Functions

by   Julien Mairal, et al.

In this paper, we study optimization methods consisting of iteratively minimizing surrogates of an objective function. By proposing several algorithmic variants and simple convergence analyses, we make two main contributions. First, we provide a unified viewpoint for several first-order optimization techniques such as accelerated proximal gradient, block coordinate descent, or Frank-Wolfe algorithms. Second, we introduce a new incremental scheme that experimentally matches or outperforms state-of-the-art solvers for large-scale optimization problems typically arising in machine learning.


page 1

page 2

page 3

page 4


Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization

Majorization-minimization algorithms consist of iteratively minimizing a...

Fast and Scalable Lasso via Stochastic Frank-Wolfe Methods with a Convergence Guarantee

Frank-Wolfe (FW) algorithms have been often proposed over the last few y...

Catalyst Acceleration for First-order Convex Optimization: from Theory to Practice

We introduce a generic scheme for accelerating gradient-based optimizati...

Estimate Sequences for Stochastic Composite Optimization: Variance Reduction, Acceleration, and Robustness to Noise

In this paper, we propose a unified view of gradient-based algorithms fo...

Optimization Methods for Large-Scale Machine Learning

This paper provides a review and commentary on the past, present, and fu...

Combinatorial Preconditioners for Proximal Algorithms on Graphs

We present a novel preconditioning technique for proximal optimization m...