Zeroth-Order Online Alternating Direction Method of Multipliers: Convergence Analysis and Applications

10/21/2017
by   Sijia Liu, et al.
0

In this paper, we design and analyze a new zeroth-order online algorithm, namely, the zeroth-order online alternating direction method of multipliers (ZOO-ADMM), which enjoys dual advantages of being gradient-free operation and employing the ADMM to accommodate complex structured regularizers. Compared to the first-order gradient-based online algorithm, we show that ZOO-ADMM requires √(m) times more iterations, leading to a convergence rate of O(√(m)/√(T)), where m is the number of optimization variables, and T is the number of iterations. To accelerate ZOO-ADMM, we propose two minibatch strategies: gradient sample averaging and observation averaging, resulting in an improved convergence rate of O(√(1+q^-1m)/√(T)), where q is the minibatch size. In addition to convergence analysis, we also demonstrate ZOO-ADMM to applications in signal processing, statistics, and machine learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/29/2019

Zeroth-Order Stochastic Alternating Direction Method of Multipliers for Nonconvex Nonsmooth Optimization

Alternating direction method of multipliers (ADMM) is a popular optimiza...
research
09/12/2018

A Fast Globally Linearly Convergent Algorithm for the Computation of Wasserstein Barycenters

In this paper, we consider the problem of computing a Wasserstein baryce...
research
09/04/2019

Parameter Estimation with the Ordered ℓ_2 Regularization via an Alternating Direction Method of Multipliers

Regularization is a popular technique in machine learning for model esti...
research
09/05/2020

Distributed Optimization, Averaging via ADMM, and Network Topology

There has been an increasing necessity for scalable optimization methods...
research
09/24/2019

On the Convergence of ADMM with Task Adaption and Beyond

Along with the development of learning and vision, Alternating Direction...
research
02/13/2022

Quasistatic cohesive fracture with an alternating direction method of multipliers

A method for quasistatic cohesive fracture is introduced that uses an al...
research
06/10/2020

ADMMiRNN: Training RNN with Stable Convergence via An Efficient ADMM Approach

It is hard to train Recurrent Neural Network (RNN) with stable convergen...

Please sign up or login with your details

Forgot password? Click here to reset