Optimization with First-Order Surrogate Functions

05/14/2013 ∙ by Julien Mairal, et al. ∙ 0 ∙ share

In this paper, we study optimization methods consisting of iteratively minimizing surrogates of an objective function. By proposing several algorithmic variants and simple convergence analyses, we make two main contributions. First, we provide a unified viewpoint for several first-order optimization techniques such as accelerated proximal gradient, block coordinate descent, or Frank-Wolfe algorithms. Second, we introduce a new incremental scheme that experimentally matches or outperforms state-of-the-art solvers for large-scale optimization problems typically arising in machine learning.

READ FULL TEXT

Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization

Majorization-minimization algorithms consist of iteratively minimizing a...
06/19/2013 ∙ by Julien Mairal, et al. ∙ 0 ∙ share

read it
Fast and Scalable Lasso via Stochastic Frank-Wolfe Methods with a Convergence Guarantee

Frank-Wolfe (FW) algorithms have been often proposed over the last few y...
10/24/2015 ∙ by Emanuele Frandi, et al. ∙ 0 ∙ share

read it
Optimization Methods for Large-Scale Machine Learning

This paper provides a review and commentary on the past, present, and fu...
06/15/2016 ∙ by Leon Bottou, et al. ∙ 0 ∙ share

read it
Catalyst Acceleration for First-order Convex Optimization: from Theory to Practice

We introduce a generic scheme for accelerating gradient-based optimizati...
12/15/2017 ∙ by Hongzhou Lin, et al. ∙ 0 ∙ share

read it
Randomized Similar Triangles Method: A Unifying Framework for Accelerated Randomized Optimization Methods (Coordinate Descent, Directional Search, Derivative-Free Method)

In this paper, we consider smooth convex optimization problems with simp...
07/26/2017 ∙ by Pavel Dvurechensky, et al. ∙ 0 ∙ share

read it
Asynchronous Delay-Aware Accelerated Proximal Coordinate Descent for Nonconvex Nonsmooth Problems

Nonconvex and nonsmooth problems have recently attracted considerable at...
02/05/2019 ∙ by Ehsan Kazemi, et al. ∙ 0 ∙ share

read it
Block-Cyclic Stochastic Coordinate Descent for Deep Neural Networks

We present a stochastic first-order optimization algorithm, named BCSC, ...
11/20/2017 ∙ by Kensuke Nakamura, et al. ∙ 0 ∙ share

read it

Optimization with First-Order Surrogate Functions

Authors

Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization

Fast and Scalable Lasso via Stochastic Frank-Wolfe Methods with a Convergence Guarantee

Optimization Methods for Large-Scale Machine Learning

Catalyst Acceleration for First-order Convex Optimization: from Theory to Practice

Randomized Similar Triangles Method: A Unifying Framework for Accelerated Randomized Optimization Methods (Coordinate Descent, Directional Search, Derivative-Free Method)

Asynchronous Delay-Aware Accelerated Proximal Coordinate Descent for Nonconvex Nonsmooth Problems

Block-Cyclic Stochastic Coordinate Descent for Deep Neural Networks

This week in AI