DeepAI AI Chat
Log In Sign Up

Convergence Rate Analysis of Accelerated Forward-Backward Algorithm with Generalized Nesterov Momentum Scheme

12/11/2021
by   Yizun Lin, et al.
JNU
Guangdong University of Technology
NetEase, Inc
0

Nesterov's accelerated forward-backward algorithm (AFBA) is an efficient algorithm for solving a class of two-term convex optimization models consisting of a differentiable function with a Lipschitz continuous gradient plus a nondifferentiable function with a closed form of its proximity operator. It has been shown that the iterative sequence generated by AFBA with a modified Nesterov's momentum scheme converges to a minimizer of the objective function with an o(1/k^2) convergence rate in terms of the function value (FV-convergence rate) and an o(1/k) convergence rate in terms of the distance between consecutive iterates (DCI-convergence rate). In this paper, we propose a more general momentum scheme with an introduced power parameter ω∈(0,1] and show that AFBA with the proposed momentum scheme converges to a minimizer of the objective function with an o(1/k^2ω) FV-convergence rate and an o(1/k^ω) DCI-convergence rate. The generality of the proposed momentum scheme provides us a variety of parameter selections for different scenarios, which makes the resulting algorithm more flexible to achieve better performance. We then employ AFBA with the proposed momentum scheme to solve the smoothed hinge loss ℓ_1-support vector machine model. Numerical results demonstrate that the proposed generalized momentum scheme outperforms two existing momentum schemes.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/07/2023

Convergence rates for momentum stochastic gradient descent with noise of machine learning type

We consider the momentum stochastic gradient descent scheme (MSGD) and i...
06/18/2020

Improving the Convergence Rate of One-Point Zeroth-Order Optimization using Residual Feedback

Many existing zeroth-order optimization (ZO) algorithms adopt two-point ...
08/20/2021

Practical and Fast Momentum-Based Power Methods

The power method is a classical algorithm with broad applications in mac...
10/24/2019

Efficient Computation of Kubo Conductivity for Incommensurate 2D Heterostructures

Here we introduce a numerical method for computing conductivity via the ...
02/28/2020

Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectives

We analyze the convergence rate of various momentum-based optimization a...
10/22/2021

GPU-Accelerated Forward-Backward algorithm with Application to Lattice-Free MMI

We propose to express the forward-backward algorithm in terms of operati...