Motivated by the need to solve large-scale optimization problems and increasing capabilities in parallel computing, block coordinate update (BCU) methods have become particularly popular in recent years due to their low per-update computational complexity, low memory requirements, and their potentials in a distributive computing environment. In the context of optimization, BCU first appeared in the form of block coordinate descent (BCD) type of algorithms which can be applied to solve unconstrained smooth problems or those with separable nonsmooth terms in the objective (possibly with separable constraints). More recently, it has been developed for solving problems with nonseparable nonsmooth terms and/or constraint in a primal-dual framework.
In this paper, we consider the following linearly constrained multi-block structured optimization model:
where is partitioned into disjoint blocks , is a smooth convex function with Lipschitz continuous gradient, and each is proper closed convex and possibly non-differentiable. Note that can include an indicator function of a convex set , and thus (1) can implicitly include certain separable block constraints in addition to the nonseparable linear constraint.
Many applications arising in statistical and machine learning, image processing, and finance can be formulated in the form of (1) including the basis pursuit , constrained regression 
, support vector machine in its dual form, portfolio optimization , just to name a few.
Towards finding a solution for (1), we will first present an accelerated proximal Jacobian alternating direction method of multipliers (Algorithm 1), and then we generalize it to an accelerated randomized primal-dual block coordinate update method (Algorithm 2). Assuming strong convexity on the objective function, we will establish convergence rate results of the proposed algorithms by adaptively setting the parameters, where is the total number of iterations. In addition, if further assuming smoothness and the full-rankness we then obtain linear convergence of a modified method (Algorithm 3).
1.1 Related methods
Our algorithms are closely related to randomized coordinate descent methods, primal-dual coordinate update methods, and accelerated primal-dual methods. In this subsection, let us briefly review the three classes of methods and discuss their relations to our algorithms.
Randomized coordinate descent methods
In the absence of linear constraint, Algorithm 2 specializes to randomized coordinate descent (RCD), which was first proposed in  for smooth problems and later generalized in [38, 27] to nonsmooth problems. It was shown that RCD converges sublinearly with rate , which can be accelerated to for convex problems and achieves a linear rate for strongly convex problems. By choosing multiple block variables at each iteration,  proposed to parallelize the RCD method and showed the same convergence results for parallelized RCD. This is similar to setting in Algorithm 2, allowing parallel updates on the selected -blocks.
Primal-dual coordinate update methods
In the presence of linear constraints, coordinate descent methods may fail to converge to a solution of the problem because fixing all but one block, the selected block variable may be uniquely determined by the linear constraint. To perform coordinate update to the linearly constrained problem (1), one effective approach is to update both primal and dual variables. Under this framework, the alternating direction method of multipliers (ADMM) is one popular choice. Originally, ADMM [17, 14] was proposed for solving two-block structured problems with separable objective (by setting and in (1)), for which its convergence and also convergence rate have been well-established (see e.g. [2, 13, 22, 29]). However, directly extending ADMM to the multi-block setting such as (1) may fail to converge; see  for a divergence example of the ADMM even for solving a linear system of equations. Lots of efforts have been spent on establishing the convergence of multi-block ADMM under stronger assumptions (see e.g. [4, 6, 25, 26, 16]) such as strong convexity or orthogonality conditions on the linear constraint. Without additional assumptions, modification is necessary for the ADMM applied to multi-block problems to be convergent; see [12, 19, 20, 39] for example. Very recently,  proposed a randomized primal-dual coordinate (RPDC) update method, whose asynchronous parallel version was then studied in . Applied to (1), RPDC is a special case of Algorithm 2 with fixed parameters. It was shown that RPDC converges with rate under convexity assumption. More general than solving an optimization problem, primal-dual coordinate (PDC) update methods have also appeared in solving fixed-point or monotone inclusion problems [36, 35, 9, 34]
. However, for these problems, the PDC methods are only shown to converge but no convergence rate estimates are known unless additional assumptions are made such as the strong monotonicity condition.
Accelerated primal-dual methods
It is possible to accelerate the rate of convergence from to for gradient type methods. The first acceleration result was shown by Nesterov  for solving smooth unconstrained problems. The technique has been generalized to accelerate gradient-type methods on possibly nonsmooth convex programs [1, 32]. Primal-dual methods on solving linearly constrained problems can also be accelerated by similar techniques. Under convexity assumption, the augmented Lagrangian method (ALM) is accelerated in  from convergence rate to by using a similar technique as that in  to the multiplier update, and  accelerates the linearized ALM using a technique similar to that in . Assuming strong convexity on the objective,  accelerates the ADMM method, and the assumption is weakened in  to assuming the strong convexity for one component of the objective function. On solving bilinear saddle-point problems, various primal-dual methods can be accelerated if either primal or dual problem is strongly convex [5, 11, 3]. Without strong convexity, partial acceleration is still possible in terms of the rate depending on some other quantities; see e.g. [33, 8].
1.2 Contributions of this paper
We accelerate the proximal Jacobian ADMM  and also generalize it to an accelerated primal-dual coordinate updating method for linearly constrained multi-block structured convex program, where in the objective there is a nonseparable smooth function. With parameters fixed during all iterations, the generalized method reduces to that in  and enjoys convergence rate under mere convexity assumption. By adaptively setting the parameters at different iterations, we show that the accelerated method has convergence rate if the objective is strongly convex. In addition, if there is one block variable that is independent of all others in the objective (but coupled in the linear constraint) and also the corresponding component function is smooth, we modify the algorithm by treating that independent variable in a different way and establish a linear convergence result. Numerically, we test the accelerated method on quadratic programming and compare it to the (nonaccelerated) RPDC method in . The results demonstrate that the accelerated method performs efficiently and stably with the parameters automatically set in accordance of the analysis, while the RPDC method needs to tune its parameters for different data in order to have a comparable performance.
1.3 Nomenclature and basic facts
Notations. For a positive integer , we denote as . We let denote the subvector of with blocks indexed by . Namely, if , then . Similarly, denotes the submatrix of with columns indexed by , and denotes the sum of component functions indicated by . We use for the partial gradient of with respect to at and with respect to . For a nondifferentiable function , denotes a subgradient of at . We reserve
for the identity matrix and usefor Euclidean norm. Given a symmetric positive semidefinite (PSD) matrix , for any vector of appropriate size, we define , and
If , we simply use . Also, we denote
Preparations. A point is called a Karush-Kuhn-Tucker (KKT) point of (1) if
For convex programs, the conditions in (4) are sufficient for to be an optimal solution of (1), and they are also necessary if a certain qualification condition holds (e.g., the Slater condition: there is in the interior of the domain of such that ). Together with the convexity of , (4) implies
We will use the following lemmas as basic facts. The first lemma is straightforward to verify from the definition of ; the second one is similar to Lemma 3.3 in ; the third one is from Lemma 3.5 in .
For any vectors and symmetric PSD matrix of appropriate sizes, it holds that
Given a function , for a given and a random vector , if for any (that may depend on ) it holds then for any , we have
Proof. Let if , and otherwise. Then
In addition, since , we have and thus . Hence, we have the desired result from .
where satisfies the optimality conditions in (4), and we assume .
Outline. The rest of the paper is organized as follows. Section 2 presents the accelerated proximal Jacobian ADMM and its convergence results. In section 3, we propose an accelerated primal-dual block coordinate update method with convergence analysis. Section 4 assumes more structure on the problem (1) and modifies the algorithm in section 3 to have linear convergence. Numerical results are provided in section 5. Finally, section 6 concludes the paper.
2 Accelerated proximal Jacobian ADMM
In this section, we propose an accelerated proximal Jacobian ADMM for solving (1). At each iteration, the algorithm updates all block variables in parallel by minimizing a linearized proximal approximation of the augmented Lagrangian function, and then it renews the multiplier. Specifically, it iteratively performs the following updates:
where and are scalar parameters, is an block diagonal matrix with as its -th diagonal block for , and denotes the residual. Note that (7a) consists of independent subproblems, and they can be solved in parallel.
Algorithm 1 summarizes the proposed method. It reduces to the proximal Jacobian ADMM in  if and are fixed for all and there is no nonseparable function . We will show that adapting the parameters as the iteration progresses can accelerate the convergence of the algorithm.
2.1 Technical assumptions
Throughout the analysis in this section, we make the following assumptions.
There exists satisfying the KKT conditions in (4).
is Lipschitz continuous with modulus .
The function is strongly convex with modulus .
The first two assumptions are standard, and the third one is for showing convergence rate of , where is the number of iterations. Note that if is strongly convex with modulus , we can let and . This way, we have a convex function and a strongly convex function . Hence, Assumption 3 is without loss of generality. With only convexity, Algorithm 1 can be shown to converge at the rate with parameters fixed for all iterations, and the order is optimal as shown in the very recent work .
2.2 Convergence results
Lemma 2.1 (One-iteration analysis)
Using the above lemma, we are able to prove the following theorem.
In the next theorem, we provide a set of parameters that satisfy the conditions in Theorem 2.2 and establish the convergence rate result.
3 Accelerating randomized primal-dual block coordinate updates
In this section, we generalize Algorithm 1 to a randomized setting where the user may choose to update a subset of blocks at each iteration. Instead of updating all block variables, we randomly choose a subset of them to renew at each iteration. Depending on the number of processors (nodes, or cores), we can choose a single or multiple block variables for each update.
3.1 The algorithm
Our algorithm is an accelerated version of the randomized primal-dual coordinate update method recently proposed in , for which we shall use RPDC as its acronym.111In fact,  presents a more general algorithmic framework. It assumes two groups of variables, and each has multi-block structure. Our method in Algorithm 2 is an accelerated version of one special case of Algorithm 1 in . At each iteration, it performs a block proximal gradient update to a subset of randomly selected primal variables while keeping the remaining ones fixed, followed by an update to the multipliers. Specifically, at iteration , it selects an index set with cardinality and performs the following updates:
where and are algorithm parameters, and their values will be determined later. Note that we use in (17a) for simplicity. It can be replaced by a PSD matrix weighted norm square term as in (7a), and our convergence results still hold.
Algorithm 2 summarizes the above method. If the parameters and are fixed during all the iterations, i.e., constant parameters, the algorithm reduces to a special case of the RPDC method in . Adapting these parameters to the iterations, we will show that Algorithm 2 enjoys faster convergence rate than RPDC if the problem is strongly convex.
3.2 Convergence results
For any with , is Lipschitz continuous with a uniform constant .
Note that if is Lipschitz continuous with constant , then and . In addition, if and only differ on a set with cardinality , then
Lemma 3.1 (One iteration analysis)
When (i.e., (1) is convex), Algorithm 2 has convergence rate with fixed . This can be shown from (19), and a similar result in slightly different form has been established in [15, Theorem 3.6]. For completeness, we provide its proof in the appendix.
Theorem 3.2 (Un-accelerated convergence)
When is strongly convex, the above convergence rate can be accelerated to by adaptively changing the parameters at each iteration. The following theorem is our main result. It shows an convergence result under certain conditions on the parameters. Based on this theorem, we will give a set of parameters that satisfy these conditions, thus providing a specific scheme to choose the paramenters.
The following parameters satisfy all conditions in (21):
4 Linearly convergent primal-dual method
In this section, we assume some more structure on (1) and show that a linear rate of convergence is possible. If there is no linear constraint, Algorithm 2 reduces to the RCD method proposed in . It is well-known that RCD converges linearly if the objective is strongly convex. However, with the presence of linear constraints, mere strong convexity of the objective of the primal problem only ensures the smoothness of its Lagrangian dual function, but not its strong concavity. Hence, in general, we do not expect linear convergence by only assuming strong convexity on the primal objective function. To ensure linear convergence on both the primal and dual variables, we need additional assumptions.
Throughout this section, we suppose that there is at least one block variable being absent in the nonseparable part of the objective, namely . For convenience, we rename this block variable to be , and the corresponding component function and constraint coefficient matrix as and . Specifically, we consider the following problem
One example of (26) is the problem that appears while computing a point on the central path of a convex program. Suppose we are interested in solving
Let and use the log-barrier function. We have the log-barrier approximation of (27) as follows:
where is the all-one vector. As decreases, the approximation becomes more accurate.
in every iteration but instead update it in probability. Hence, roughly speaking, and variables are updated in the same frequency. The method is summarized in Algorithm 3.
4.1 Technical assumptions
There exists satisfying the KKT conditions in (32).
The function is strongly convex with modulus , and its gradient is Lipschitz continuous with constant .
The strong convexity of and implies