A Distributed Frank-Wolfe Algorithm for Communication-Efficient Sparse Learning

Learning sparse combinations is a frequent theme in machine learning. In this paper, we study its associated optimization problem in the distributed setting where the elements to be combined are not centrally located but spread over a network. We address the key challenges of balancing communication costs and optimization errors. To this end, we propose a distributed Frank-Wolfe (dFW) algorithm. We obtain theoretical guarantees on the optimization error ϵ and communication cost that do not depend on the total number of combining elements. We further show that the communication cost of dFW is optimal by deriving a lower-bound on the communication cost required to construct an ϵ-approximate solution. We validate our theoretical analysis with empirical studies on synthetic and real-world data, which demonstrate that dFW outperforms both baselines and competing methods. We also study the performance of dFW when the conditions of our analysis are relaxed, and show that dFW is fairly robust.



There are no comments yet.


page 1

page 2

page 3

page 4


Graph Partitioning via Parallel Submodular Approximation to Accelerate Distributed Machine Learning

Distributed computing excels at processing large scale data, but the com...

Quantized Frank-Wolfe: Communication-Efficient Distributed Optimization

How can we efficiently mitigate the overhead of gradient communications ...

AIDE: Fast and Communication Efficient Distributed Optimization

In this paper, we present two new communication-efficient methods for di...

DEED: A General Quantization Scheme for Communication Efficiency in Bits

In distributed optimization, a popular technique to reduce communication...

Distributed Graph Learning with Smooth Data Priors

Graph learning is often a necessary step in processing or representing s...

SLSGD: Secure and Efficient Distributed On-device Machine Learning

We consider distributed on-device learning with limited communication an...

Optimal Random Sampling from Distributed Streams Revisited

We give an improved algorithm for drawing a random sample from a large d...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Most machine learning algorithms are designed to be executed on a single computer that has access to the full training set. However, in recent years, large-scale data has become increasingly distributed across several machines (nodes) connected through a network, for storage purposes or because it is collected at different physical locations, as in cloud computing, mobile devices or wireless sensor networks (Lewis, 2004; Armbrust et al., 2010). In these situations, communication between machines is often the main bottleneck and the naive approach of sending the entire dataset to a coordinator node is impractical. This distributed setting triggers many interesting research questions. From a fundamental perspective, studying the tradeoff between communication complexity (Yao, 1979; Abelson, 1980) and learning/optimization error has recently attracted some interest (Balcan et al., 2012; Daumé III et al., 2012; Shamir, 2014). From a more practical view, a lot of research has gone into developing scalable algorithms with small communication and synchronization overhead (see for instance Lazarevic and Obradovic, 2002; Forero et al., 2010; Boyd et al., 2011; Duchi et al., 2012; Dekel et al., 2012; Balcan et al., 2013; Yang, 2013) and references therein).

In this paper, we present both theoretical and practical results for the problem of learning a sparse combination of elements (atoms

) that are spread over a network. This has numerous applications in machine learning, such as LASSO regression

(Tibshirani, 1996)

, support vector machines

(Cortes and Vapnik, 1995), multiple kernel learning (Bach et al., 2004), -Adaboost (Shen and Li, 2010) and sparse coding (Lee et al., 2006). Formally, let be a finite set of -dimensional atoms (one per column). Here, atoms broadly refer to data points, features, dictionary elements, basis functions, etc, depending on the application. We wish to find a weight vector ( giving the weight assigned to atom ) that minimizes some cost function under a sparsity constraint:111Our results also hold for sparsity constraints other than the norm, such as simplex constraints.


where the cost function is convex and continuously differentiable and .

We propose a distributed Frank-Wolfe (dFW) algorithm, a novel approach to solve such distributed sparse learning problems based on an adaptation of its centralized counterpart (Frank and Wolfe, 1956; Clarkson, 2010; Jaggi, 2013). Intuitively, dFW is able to determine, in a communication-efficient way, which atoms are needed to reach an -approximate solution, and only those are broadcasted through the network. We also introduce an approximate variant of dFW that can be used to balance the amount of computation across the distributed machines to reduce synchronization costs. In terms of theoretical guarantees, we show that the total amount of communication required by dFW is upper-bounded by a quantity that does not depend on but only on , and the topology of the network. We further establish that this dependency on and is worst-case optimal for the family of problems of interest by proving a matching lower-bound of communication for any deterministic algorithm to construct an -approximation. Simple to implement, scalable and parameter-free, dFW can be applied to many distributed learning problems. We illustrate its practical performance on two learning tasks: LASSO regression with distributed features and Kernel SVM with distributed examples. Our main experimental findings are as follows: (i) dFW outperforms baseline strategies that select atoms using a local criterion, (ii) when the data and/or the solution is sparse, dFW requires less communication than ADMM, a popular competing method, and (iii) in a real-world distributed architecture, dFW achieves significant speedups over the centralized version, and is robust to random communication drops and asynchrony in the updates.


Section 2 reviews the Frank-Wolfe algorithm in the centralized setting. In Section 3, we introduce our proposed dFW algorithm and an approximate variant of practical interest, as well as examples of applications. The theoretical analysis of dFW, including our matching communication lower bound, is presented in Section 4. Section 5 reviews some related work, and experimental results are presented in Section 6. Finally, we conclude in Section 7.


We denote scalars by lightface letters, vectors by bold lower case letters and matrices by bold upper case letters. For , we use to denote the set . For , is the all zeros vector except 1 in the entry. For a vector , the support of is defined as . The gradient of a differentiable function at is denoted by .

2 The Frank-Wolfe Algorithm

1:  Let
2:  for  do
4:     , where or obtained via line-search.
5:  end for
6:  stopping criterion:
Algorithm 1 FW algorithm on general domain
1:  if constraint
4:  else if simplex constraint
Algorithm 2 Compute for / simplex

The Frank-Wolfe (FW) algorithm (Frank and Wolfe, 1956; Clarkson, 2010; Jaggi, 2013), also known as Conditional Gradient (Levitin and Polyak, 1966), is a procedure to solve convex constrained optimization problems of the form


where is convex and continuously differentiable, and is a compact convex subset of any vector space (say ). The basic FW algorithm is shown in Algorithm 1. At each iteration, it moves towards a feasible point that minimizes a linearization of at the current iterate. The stopping criterion uses a surrogate optimality gap that can be easily computed as a by-product of the algorithm. Let be an optimal solution of (2). Theorem 1 shows that FW finds an -approximate solution in iterations.

Theorem 1 (Jaggi, 2013).

Let be the curvature constant of .222The assumption of bounded is closely related to that of -Lipschitz continuity of with respect to an arbitrary chosen norm . In particular, we have (Jaggi, 2013). Algorithm 1 terminates after at most iterations and outputs a feasible point which satisfies .

Solving the subproblems

The key step in FW is to solve a linear minimization subproblem on (step 3 in Algorithm 1). When the extreme points of have a special structure (e.g., when they are sparse), it can be exploited to solve this subproblem efficiently. In particular, when is an norm constraint as in problem (1), a single coordinate (corresponding to the largest entry in the magnitude of the current gradient) is greedily added to the solution found so far (Algorithm 2). Taking , we get . Combining this observation with Theorem 1 shows that Algorithm 1 always finds an -approximate solution to problem (1) with nonzero entries, which is sometimes called an -coreset of size (Shalev-Shwartz et al., 2010; Jaggi, 2011). It turns out that this is worst-case optimal (up to constant terms) for the family of problems (1), as evidenced by the derivation of a matching lower bound of for the sparsity of an -approximate solution (Jaggi, 2011).333Note that the dependency on can be improved to for strongly convex functions by using so-called away-steps (see for instance (Lacoste-Julien and Jaggi, 2013)). However, when is large, this improvement might not be desirable since it introduces a dependency on .

Another domain of interest is the unit simplex . The solution to the linear subproblem, shown in Algorithm 2, also ensures the sparsity of the iterates and leads to an -approximation with nonzero entries, which is again optimal (Clarkson, 2010; Jaggi, 2013).

For both domains, the number of nonzero entries depends only on , and not on , which is particularly useful for tackling large-scale problems. These results constitute the basis for our design of a distributed algorithm with small communication overhead.

3 Distributed Frank-Wolfe (dFW) Algorithm

1:  on all nodes:
2:  for  do
3:     on each node :
  • [noitemsep,topsep=2pt,itemindent=-10pt]

  •      // find largest entry of the local gradient in absolute value

  •                       // compute partial sum for stopping criterion

  • broadcast and

4:     on each node :
  • [noitemsep,topsep=2pt,itemindent=-10pt]

  •                  // compute index of node with largest overall gradient

  • if  then broadcast and end if  // send atom and index to other nodes

5:     on all nodes: , where or line-search
6:  end for
7:  stopping criterion:                                                                 
Algorithm 3 Distributed Frank-Wolfe (dFW) algorithm for problem (1)

We now focus on the problem of solving (1) in the distributed setting. We consider a set of local nodes which communicate according to an undirected connected graph where is a set of edges. An edge indicates that and can communicate with each other. To simplify the analysis, we assume no latency in the network and synchronous updates — our experiments in Section 6.3 nonetheless suggest that these simplifying assumptions could be relaxed.

The atom matrix is partitioned column-wise across . Formally, for , let the local data of , denoted by , be the set of column indices associated with node , where and for . The local gradient of is the gradient of restricted to indices in . We measure the communication cost in number of real values transmitted.

3.1 Basic Algorithm

Our communication-efficient procedure to solve (1) in the distributed setting is shown in Algorithm 3.444Note that it can be straightforwardly adapted to deal with a simplex constraint based on Algorithm 2. The algorithm is parameter-free and each iteration goes as follows: (i) each node first identifies the largest component of its local gradient (in absolute value) and broadcasts it to the other nodes (step 3), (ii) the node with the overall largest value broadcasts the corresponding atom, which will be needed by other nodes to compute their local gradients in subsequent iterations (step 4), and (iii) all nodes perform a FW update (step 5). We have the following result.

Theorem 2.

Algorithm 3 terminates after rounds and total communication, where is an upper bound on the cost of broadcasting a real number to all nodes in the network . At termination, all nodes hold the same with optimality gap at most and the atoms corresponding to its nonzero entries.


We prove this by showing that dFW performs Frank-Wolfe updates and that the nodes have enough information to execute the algorithm. The key observation is that the local gradient only depends on local data and atoms received so far. Details can be found in Appendix A. ∎

Theorem 2 shows that dFW allows a trade-off between communication and optimization error without dependence on the total number of atoms, which makes the algorithm very appealing for large-scale problems. The communication cost depends only linearly on , and the network topology.

3.2 An Approximate Variant

1:  repeat times
2:   , where with
Algorithm 4 GreedySelection() // add centers from to
1:  on each node : ;
2:  for  do
3:     perform the same steps as in Algorithm 3, except that is selected from
4:     on each node :
5:  end for
Algorithm 5 Approximate dFW algorithm

Besides its strong theoretical guarantees on optimization error and communication cost, dFW is very scalable, as the local computational cost of an iteration typically scales linearly in the number of atoms on the node. However, if these local costs are misaligned, the algorithm can suffer from significant wait time overhead due to the synchronized updates. To address this issue, we propose an approximate variant of dFW which can be used to balance or reduce the effective number of atoms on each node. The algorithm is as follows. First, each node clusters its local dataset into groups using the classic greedy -center algorithm of Gonzalez (1985), which repeatedly adds a new center that is farthest from the current set of centers (Algorithm 4). Then, dFW is simply run on the resulting centers. Intuitively, each original atom has a nearby center, thus selecting only from the set of centers is optimal up to small additive error that only leads to small error in the final solution. Optionally, we can also gradually add more centers so that the additive error scales as , in which case the error essentially does not affect the quality of the final solution. The details are given in Algorithm 5, and we have the following result.

Lemma 1.

Let denote the optimal -radius of partitioning the local data indexed by into clusters, and let . Let . Then, Algorithm 5 computes a feasible solution with optimality gap at most after iterations. Furthermore, if , then the optimality gap is at most .


By the analysis in Gonzalez (1985), the maximum cluster radius on node is at most at time , hence the maximum -radius over all clusters is at most . For any , there exists such that

Thus the selected atom is optimal up to a additive error. Then the first claim follows from the analysis of Freund and Grigas (2013).

Similarly, when , the additive error is . The second claim then follows from the analysis of Jaggi (2013). ∎

Algorithm 5 has the potential to improve runtime over exact dFW in various practical situations. For instance, when dealing with heterogenous local nodes, can pick to be proportional to its computational power so as to reduce the overall waiting time. Another situation where this variant can be useful is when the distribution of atoms across nodes is highly unbalanced. We illustrate the practical benefits of this approximate variant on large-scale Kernel SVM in Section 6.

3.3 Illustrative Examples

We give three examples of interesting problems that dFW can solve efficiently: LASSO regression, Kernel SVM and -Adaboost. Others applications include sparse coding Lee et al. (2006) and multiple kernel learning Bach et al. (2004).

LASSO regression with distributed features

LASSO (Tibshirani, 1996)

is a linear regression method which aims at approximating a target vector

by a sparse combination of features:


where and are the target value and feature for training point , respectively. Our dFW algorithm applies to problem (3) in the context of distributed features (each local node holds a subset of the features for all training points), as common in distributed sensor networks (see e.g. Christoudias et al., 2008)

. It applies in a similar manner to sparse logistic regression

(Shevade and Keerthi, 2003) and, perhaps more interestingly, to Group-LASSO (Yuan and Lin, 2006)

. For the latter, suppose each group of features is located on the same node, for instance when combining features coming from the same source in multi-view learning or when using dummy variables that encode categorical features. In this case, each atom is a group of features and dFW selects a single group at each iteration

(see Jaggi, 2013, for the Frank-Wolfe update in this case).

Support Vector Machines with distributed examples

Let be a training sample with and . Let be a PSD kernel. The dual problem of L2-loss SVM with kernel is as follows (Tsang et al., 2005; Ouyang and Gray, 2010):


with where is the “augmented kernel”.

It is easy to see that with . Thus where is the atom matrix. Notice however that atoms may be of very high (or even infinite) dimension since they lie in kernel space. Fortunately, the gradient only depends on , thus we can broadcast the training point instead of (assuming all nodes know the kernel function ). Note that dFW can be seen as a distributed version of Core Vector Machines (Tsang et al., 2005), a large-scale SVM algorithm based on Frank-Wolfe, and that it applies in a similar manner to a direct multi-class L2-SVM (Asharaf et al., 2007) and to Structural SVM (see the recent work of Lacoste-Julien et al., 2013).

Boosting with distributed base classifiers


be a class of base classifiers to be combined. Given a training sample

, let where . We consider the following -Adaboost formulation (Shen and Li, 2010):


where is a tuning parameter. Our algorithm thus straightforwardly applies to (5) when base classifiers are distributed across nodes. The gradient of (5) is as follows:

The Frank-Wolfe update thus corresponds to adding the base classifier that performs best on the training sample weighted by , where defines a distribution that favors points that are currently misclassified. This can be done efficiently even for a large (potentially infinite) family of classifiers when a weak learner for is available.555A weak learner for returns the base classifier in that does best on the weighted sample defined by . In particular, when is the class of decision stumps and the features are distributed, each node can call the weak learner on its local features to determine its best local base classifier.

4 Theoretical Analysis

Figure 1: Illustration of network topologies.

4.1 Communication Cost of dFW under Different Network Topology

We derive the communication cost of dFW for various types of network graphs, depicted in Figure 1.

Star network

In a star network, there is a coordinator node connected to all local nodes . Broadcasting a number to all nodes has cost , so the communication cost is . This can be improved to : instead of broadcasting and , we can send these quantities only to which selects the maximum and computes the sum of .

Rooted tree

Broadcasting on a tree costs , but this can be improved again by avoiding broadcast of and . To select the maximum , each node sends to its parent the maximum value among its local and the values received from its children. The root gets the maximum and can send it back to all the nodes. A similar trick works for computing the sum of . Hence, only and the selected atom need to be broadcasted, and the total communication is .

General connected graph

If the nodes can agree on a spanning tree, then this reduces to the previous case. We further consider the case when nodes operate without using any information beyond that in their local neighborhood and cannot agree on a spanning tree (this is known as the fully distributed setting, Mosk-Aoyama et al., 2010). In this case, broadcasting can be done through a message-passing procedure similar to the one used in Balcan et al. (2013). The cost of broadcasting a number is , and thus the total communication cost is .

4.2 Lower Bound on Communication Cost

In this section, we derive a lower bound on the communication required by any deterministic algorithm to construct an -approximation to problem (1) for either the -ball or the simplex constraint. We assume that at the end of the algorithm, at least one of the nodes has all the selected atoms. This is not a restrictive assumption since in many applications, knowledge of the selected atoms is required in order to use the learned model (for instance in kernel SVM).

Our proof is based on designing an optimization problem that meets the desired minimum amount of communication. First, we consider a problem for which any -approximation solution must have selected atoms. We split the data across two nodes and so that these atoms must be almost evenly split across the two nodes. Then, we show that for any fix dataset on one node, there are different instances of the dataset on the other node, so that in any two such instances, the sets of selected atoms are different. Therefore, for any node to figure out the atom set, it needs bits. The proof is completed by showing .

4.2.1 Setup

Consider the problem of minimizing over the unit simplex , where is a orthonormal matrix (i.e., ). Suppose the first columns of (denoted as ) are on the node and the other half (denoted as ) are on the node . Restrict to be block diagonal as follows: vectors in only have non-zeros entries in coordinates and vectors in only have non-zeros entries in coordinates .

For any deterministic algorithm , let denote the set of atoms selected given input and , and let denote the minimum value of when and has only non-zero entries on . We only consider deterministic algorithms that lead to provably good approximations: satisfies for any . Furthermore, at the end of the algorithm, at least one of the nodes has all the selected atoms.

In practice, bounded precision representations are used for real values. Study of communication in such case requires algebraic assumptions (Abelson, 1980; Tsitsiklis and Luo, 1987). For our problem, we allow the columns in to be orthogonal and of unit length only approximately. More precisely, we say that two vectors and are near-orthogonal if , and a vector has near-unit length if . The notion of subspace is then defined based on near-orthogonality and near-unit length. We make the following assumption about the representation precision of vectors in subspaces.

Assumption 1.

There exists a constant s.t. for any -dimensional subspace, , the number of different -dimensional near-unit vectors in the subspace is .

4.2.2 Analysis

Let and (for simplicity, suppose is an integer). We first show that the atoms in any good approximation solution must be almost evenly split across the two nodes.

Claim 1.

Any with has more than non-zero entries. Consequently, for .


First note that . When using exact real values, , thus the problem is equivalent to minimizing over the unit simplex. When using bounded precision representation, the exact equivalence does not hold. Still, by our definitions of near-unit vectors and near-orthogonal vectors, where each entry in has absolute value at most . So for any , .

Then, any with satisfies that where is the optimal value for . To see this, let denote the optimal solution for and denote the optimal solution for . We have:

which leads to . By Lemma 3 of Jaggi (2013), we have that for ,

So . Hence , which leads to the claim. ∎

Next, we show that for any fix dataset on one node, there are many different instances of the dataset on the other node, so that any two different instances lead to different selected atoms.

Claim 2.

For any instance of , there exist different instances of (denoted as ) such that the set of atoms selected from are different for all , i.e., for any ,


Let denote the number of different sets of near-unit vectors in dimension that are near-orthogonal to each other. By Claim 1, if , then . Then there are at most different so that are all the same. Since there are different instances of , we have

Next, we observe that . We first choose a near-unit vector in the -dimensional subspace, and then choose a near-unit vector in the -dimensional subspace orthogonal to the first chosen vector, and continue in this way until near-unit vectors are selected. By Assumption 1, there are different choices, and there are repetitions, which leads to the bound on . The bound on then follows from . ∎

Distinguishing between the instances requires communication, resulting in our lower bound.

Theorem 3.

Suppose a deterministic algorithm satisfies for any . Under Assumption 1, the communication cost of is lower-bounded by bits.


Suppose the algorithm uses less than bits. At the end of the algorithm, if node has the selected atoms, then must have determined these atoms based on and less than bits. By Claim 2, there are different instances of with different . For at least two of these instances, will output the same set of atoms, which is contradictory. A similar argument holds if node has the selected atoms. Therefore, the algorithm uses at least bits. ∎

4.2.3 Remarks

The same lower bound holds for -constrained problems by reduction to the simplex case. The analysis also extends to networks with more than two nodes, in which case the diameter of the network is a multiplicative factor in the bound. Details can be found in Appendix B.

The upper bound given by dFW in Theorem 2 matches the lower bound of Theorem 3 in its dependency on and . This leads to the important result that the communication cost of dFW is worst-case optimal in these two quantities.

5 Related Work

5.1 Distributed Examples

An important body of work considers the distributed minimization of a function where is convex and only known to node . This typically corresponds to the risk minimization setting with distributed training examples, where gives the loss for the data points located on node . These methods are based on subgradient descent (SGD) (Duchi et al., 2012; Dekel et al., 2012, and references therein), the Alternating Direction Methods of Multipliers (ADMM) (Boyd et al., 2011; Wei and Ozdaglar, 2012) and Stochastic Dual Coordinate Ascent (SDCA) (Yang, 2013). Their rate of convergence (and thus their communication cost) for general convex functions is in or , sometimes with a dependency on , the total number of data points.

Note that for Kernel SVM (Section 3.3), none of the above methods can be applied efficiently, either because they rely on the primal (GD, ADMM) or because they require broadcasting at each iteration a number of data points that is at least the number of nodes in the network (SDCA). In contrast, dFW only needs to broadcast a single point per iteration and still converges in , with no dependency on . We illustrate its performance on this task in the experiments.

5.2 Distributed Features

In the case of distributed features, ADMM is often the method of choice due to its wide applicability and good practical performance (Boyd et al., 2011).666Note that to the best of our knowledge, no convergence rate is known for ADMM with distributed features. Given nodes, the parameter vector is partitioned as with , where . The matrix is partitioned conformably as and the optimization problem has the form


where the loss is convex, the regularizer is separable across nodes and convex, and . Each node iteratively solves a local subproblem and sends its local prediction to a coordinator node, which then broadcasts the current global prediction . ADMM can be slow to converge to high accuracy, but typically converges to modest accuracy within a few tens of iterations (Boyd et al., 2011).

Both ADMM and dFW can deal with LASSO regression with distributed features. When features are dense, dFW and ADMM have a similar iteration cost but dFW is typically slower to converge as it only adds one feature to the solution at each iteration. On the other hand, the case of sparse features creates an interesting tradeoff between the number of iterations and the communication cost per iteration. Indeed, an iteration of dFW becomes much cheaper in communication than an iteration of ADMM since local/global predictions are typically dense. This tradeoff is studied in the experiments.

5.3 Relation to Matching Pursuit

Finally, note that for the -constrained case, matching pursuit (Tropp, 2004) greedily selects atoms in the same way as FW, although the updates are different (Jaggi, 2011). The proposed communication schemes thus readily apply to a distributed matching pursuit algorithm.

6 Experiments

We have shown that dFW enjoys strong theoretical guarantees on the communication cost. In this section, we validate our analysis with a series of experiments on two tasks: LASSO regression with distributed features, and Kernel SVM with distributed examples. In Section 6.1, we compare dFW to two baseline strategies. Section 6.2 studies the communication tradeoff between dFW and ADMM, a popular distributed optimization technique. Finally, Section 6.3 proposes an evaluation of the performance of (exact and approximate) dFW in a real-world distributed environment.

6.1 Comparison to Baselines

We first compare dFW to some baseline strategies for selecting a subset of atoms locally. We investigate two methods: (i) the random strategy, where each node selects a fixed-size subset of its atoms uniformly at random, and (ii) the local FW strategy, where each node runs the FW algorithm locally to select a fixed-sized subset as proposed by Lodi et al. (2010). The objective function on the union of the selected subsets is then optimized with a batch solver. We compare the methods based on the objective value they achieve for a given communication cost. In these experiments, we use a star graph with nodes and each atom is assigned to a node uniformly at random. Results are averaged over 5 runs.

(a) Kernel SVM, Adult dataset
(b) LASSO, Dorothea dataset
Figure 2: Comparison with baseline strategies (best seen in color).
Kernel SVM with distributed examples

We conduct experiments on the UCI Adult dataset, a census income database with about 32,000 examples and 123 features, so each node holds about 320 examples. We use the RBF kernel and set its bandwidth based on the averaged distance among examples and fix SVM parameter to . Figure 2(a) shows that the training examples selected by dFW lead to significantly smaller objective value than those selected by both baselines. Note that the local FW does much better than the random strategy but performs worse than dFW due to its local selection criterion.

LASSO regression with distributed features

We also conduct experiments on Dorothea, a drug discovery dataset from the NIPS 2003 Feature Selection Challenge

(Guyon et al., 2004). It has 1,150 examples and 100,000 binary features (half of which are irrelevant to the problem), so each node holds about 1,000 features. Figure 2(b) shows the results for (other values of do not change the relative performance of the methods significantly). Again, dFW significantly outperforms both baselines. On this dataset, these local strategies decrease the objective at a slow rate due the selection of irrelevant features.

6.2 Comparison to ADMM

In this section, we compare dFW with distributed ADMM (Boyd et al., 2011), a popular competing method, on the problem of LASSO with distributed features, both on synthetic and real data. As in the previous experiment, we use

nodes and a uniform distribution of atoms, with results averaged over 5 runs.

Synthetic Data
Figure 3: Comparison with ADMM on LASSO using synthetic data of varying sparsity levels. The plots show MSE versus communication.

The purpose of this experiment is to study the tradeoff between dFW and ADMM discussed in Section 5. To do so, we generate synthetic data following the protocol in (Boyd et al., 2011) but with varying density of the data and the solution. Specifically, we create a data matrix has examples and features (atoms) with density and its nonzero entries are drawn from . The true solution has density with each nonzero entry drawn from , and the target vector is computed as , where . We use , thereby creating a set of 9 problems. We set the regularization parameter to , where is the value above which the solution becomes the zero vector.777This choice results in a solution with a number of nonzero entries in the same order of magnitude as . The corresponding value of for the constrained problem is obtained from the norm of the solution to the -regularized problem. For ADMM, we try several values of its penalty and relaxation parameters and , as done in (Boyd et al., 2011).

Figure 3 shows the MSE versus communication. These results confirm that dFW requires much less communication when the data and/or the solution are sparse (with no parameter to tune), while ADMM has the edge in the dense case (if its parameters are properly tuned). The rule of thumb seems to be that they perform similarly when , which is consistent with their communication cost per iteration and practical convergence speed.

Real data
Figure 4: Comparison with ADMM on LASSO, Dorothea dataset (best seen in color).

We also conduct experiments on the Dorothea dataset used in Section 6.1. We compare dFW to ADMM for different values of , which affect the sparsity of the solution. The results shown for ADMM are the best over its penalty and relaxation parameters.

The results in terms of communication cost shown in Figure 4 (left plot) are consistent with those obtained on synthetic data. Indeed, dFW is better at finding sparse solutions with very little communication overhead, even though many features in this dataset are irrelevant. On the other hand, as the target model becomes more dense (), ADMM catches up with dFW.

Figure 4 (middle plot) shows that ADMM generally converges in fewer iterations than dFW. However, an ADMM iteration is computationally more expensive as it requires to solve a LASSO problem on each node. In contrast, a dFW iteration amounts to computing the local gradients and is typically much faster. To give a rough idea of the difference, we show the time spent on sequentially solving the local subproblems on a single 2.5GHz CPU core in Figure 4 (right plot).888Both algorithms are implemented in Matlab. For ADMM, LASSO subproblems are solved using a proximal gradient method. This shows that on this dataset, dFW decreases the objective faster than ADMM by 1 to 2 orders of magnitude despite its larger number of iterations.

6.3 Large-scale Distributed Experiments

In this section, we evaluate the performance of dFW on large-scale kernel SVM in real-world distributed conditions. Note that dFW is computationally very efficient if implemented carefully. In particular, the local gradient of a node can be recursively updated using only the kernel values between its points and the point received at the previous iteration. Thus the algorithm only needs memory and the computational cost of an iteration is . We implemented dFW in C++ with openMPI and experiment with a fully connected network with nodes which communicate over a 56.6-gigabit infrastructure. Each node is a single 2.4GHz CPU core of a separate host. We use a speech dataset of 8.7M examples with 120 features and 42 class labels (the task is majority class versus rest).999Data is from IARPA Babel Program, Cantonese language collection release IARPA-babel101b-v0.4c limited data pack. In all experiments, we use the RBF kernel and set its bandwidth based on the averaged distance among examples. The parameter of SVM is set to .

(a) Exact dFW on ideal distribution
(b) Approx. dFW to balance costs
(c) dFW under asynchrony
Figure 5: Empirical evaluation of our theoretical analysis on distributed Kernel SVM (best seen in color).
Exact dFW

We first investigate how dFW scales with the number of nodes in the network. For this experiment, training points are assigned to nodes uniformly at random. Figure 5(a) shows the runtime results for different values of . We obtain near-linear speedup, showing that synchronization costs are not a significant issue when the local computation is well-balanced across nodes.

Approximate variant

When the local costs are misaligned, the exact dFW will suffer from large wait time. As we have seen in Section 3.2, we can use the approximate variant of dFW to balance these costs. To illustrate this, we set and assign about 50% of the data to a single node while others node share the rest uniformly at random. We use the greedy clustering algorithm on the large node to generate a number of centers approximately equal to the number of atoms on the other nodes, thus balancing local computation.101010For kernel SVM with RBF kernel, the difference between gradient entries of can be directly bounded by performing clustering in the augmented kernel space. This is because . Figure 5(b) shows that the approximate dFW reduces the runtime dramatically over exact dFW on the unbalanced distribution. As often the case in large-scale datasets, training examples cluster well, so the additive error is negligible in this case.

Asynchronous updates

Another way to reduce waiting times and synchronization costs is to use asynchronous updates. To simulate this, we randomly drop the communication messages with some probability

. This means that at any given iteration: (i) the selected atom may be suboptimal, and (ii) some nodes may not update their iterate. As a result, the iterates across nodes are not synchronized anymore. Figure 5(c) shows the objective value averaged over all the nodes at each iteration. We can see that dFW is robust to this asynchronous setting and converges properly (albeit a bit slower), even with 40% random communication drops. This suggests that an asynchronous version of dFW could lead to significant speedups in practice. A theoretical analysis of the algorithm in this challenging setting is an exciting direction for future work.

7 Conclusion

We studied the problem of finding sparse combinations of distributed elements, focusing on minimizing the communication overhead. To this end, we proposed a distributed Frank-Wolfe algorithm with favorable properties in communication cost. We showed a lower-bound to the communication cost for the family of problems we consider. This established that the proposed dFW is optimal in its dependency on the approximation quality . Experiments confirmed the practical utility of the approach.


This research is partially supported by NSF grants CCF-0953192 and CCF-1101215, AFOSR grant FA9550-09-1-0538, ONR grant N00014-09-1-0751, a Google Research Award, a Microsoft Research Faculty Fellowship and the IARPA via DoD/ARL contract # W911NF-12-C-0012. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoD/ARL, or the U.S. Government.


  • Abelson (1980) H. Abelson. Lower bounds on information transfer in distributed computations. Journal of the ACM, 27(2):384–392, 1980.
  • Armbrust et al. (2010) M. Armbrust, A. Fox, R. Griffith, A. D. J., R. H. Katz, A. Konwinski, G. Lee, D. A. Patterson, A. Rabkin, I. Stoica, and M. Zaharia. A view of cloud computing. Communications of the ACM, 53(4):50–58, 2010.
  • Asharaf et al. (2007) S. Asharaf, M. N. Murty, and S. K. Shevade. Multiclass core vector machine. In ICML, pages 41–48, 2007.
  • Bach et al. (2004) F. R. Bach, G. R. G. Lanckriet, and M. I. Jordan. Multiple kernel learning, conic duality, and the SMO algorithm. In ICML, 2004.
  • Balcan et al. (2012) M.-F. Balcan, A. Blum, S. Fine, and Y. Mansour. Distributed Learning, Communication Complexity and Privacy. In COLT, 2012.
  • Balcan et al. (2013) M.-F. Balcan, S. Ehrlich, and Y. Liang.

    Distributed k-means and k-median clustering on general communication topologies.

    In NIPS, 2013.
  • Boyd et al. (2011) S. P. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. FTML, 3(1):1–122, 2011.
  • Christoudias et al. (2008) C. M. Christoudias, R. Urtasun, and T. Darrell. Unsupervised feature selection via distributed coding for multi-view object recognition. In CVPR, 2008.
  • Clarkson (2010) K. L. Clarkson. Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm. ACM Transactions on Algorithms, 6(4):1–30, 2010.
  • Cortes and Vapnik (1995) C. Cortes and V. Vapnik. Support-Vector Networks. Machine Learning, 20(3):273–297, 1995.
  • Daumé III et al. (2012) H. Daumé III, J. M. Phillips, A. Saha, and S. Venkatasubramanian. Protocols for learning classifiers on distributed data. In AISTATS, pages 282–290, 2012.
  • Dekel et al. (2012) Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal Distributed Online Prediction Using Mini-Batches. JMLR, 13:165–202, 2012.
  • Duchi et al. (2012) J. C. Duchi, A. Agarwal, and M. J. Wainwright. Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling. IEEE Transactions on Automatic Control, 57(3):592–606, 2012.
  • Forero et al. (2010) P. A. Forero, A. Cano, and G. B. Giannakis. Consensus-Based Distributed Support Vector Machines. JMLR, 11:1663–1707, 2010.
  • Frank and Wolfe (1956) M. Frank and P. Wolfe. An algorithm for quadratic programming. Naval Research Logistics Quarterly, 3(1-2):95–110, 1956.
  • Freund and Grigas (2013) Robert M. Freund and Paul Grigas. New Analysis and Results for the Conditional Gradient Method. Technical report, arXiv:1307.0873, 2013.
  • Gonzalez (1985) Teofilo F Gonzalez. Clustering to minimize the maximum intercluster distance. Theoretical Computer Science, 38:293–306, 1985.
  • Guyon et al. (2004) I. Guyon, S. R. Gunn, A. Ben-Hur, and G. Dror. Result Analysis of the NIPS 2003 Feature Selection Challenge. In NIPS, 2004.
  • Jaggi (2011) M. Jaggi. Sparse Convex Optimization Methods for Machine Learning. PhD thesis, ETH Zurich, 2011.
  • Jaggi (2013) M. Jaggi. Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization. In ICML, 2013.
  • Lacoste-Julien and Jaggi (2013) S. Lacoste-Julien and M. Jaggi. An Affine Invariant Linear Convergence Analysis for Frank-Wolfe Algorithms. Technical report, arXiv:1312.7864, 2013.
  • Lacoste-Julien et al. (2013) S. Lacoste-Julien, M. Jaggi, M. Schmidt, and P. Pletscher. Block-Coordinate Frank-Wolfe Optimization for Structural SVMs. In ICML, 2013.
  • Lazarevic and Obradovic (2002) A. Lazarevic and Z. Obradovic. Boosting Algorithms for Parallel and Distributed Learning. Distributed and Parallel Databases, 11(2):203–229, 2002.
  • Lee et al. (2006) H. Lee, A. Battle, R. Raina, and A. Y. Ng. Efficient sparse coding algorithms. In NIPS, pages 801–808, 2006.
  • Levitin and Polyak (1966) Evgeny S. Levitin and Boris T. Polyak. Constrained minimization methods. USSR Computational Mathematics and Mathematical Physics, 6(5):1–50, 1966.
  • Lewis (2004) F. L. Lewis. Wireless sensor networks. Smart environments: technologies, protocols, and applications, pages 11–46, 2004.
  • Lodi et al. (2010) S. Lodi, R. Ñanculef, and C. Sartori. Single-Pass Distributed Learning of Multi-class SVMs Using Core-Sets. In SDM, pages 257–268, 2010.
  • Mosk-Aoyama et al. (2010) D. Mosk-Aoyama, T. Roughgarden, and D. Shah. Fully distributed algorithms for convex optimization problems. SIAM Journal on Optimization, 20(6):3260–3279, 2010.
  • Ouyang and Gray (2010) H. Ouyang and A. G. Gray. Fast Stochastic Frank-Wolfe Algorithms for Nonlinear SVMs. In SDM, pages 245–256, 2010.
  • Shalev-Shwartz et al. (2010) S. Shalev-Shwartz, N. Srebro, and T. Zhang. Trading Accuracy for Sparsity in Optimization Problems with Sparsity Constraints. SIAM Journal on Optimization, 20(6):2807–2832, 2010.
  • Shamir (2014) Ohad Shamir.

    Fundamental Limits of Online and Distributed Algorithms for Statistical Learning and Estimation.

    In NIPS, 2014.
  • Shen and Li (2010) C. Shen and H. Li. On the Dual Formulation of Boosting Algorithms. TPAMI, 32(12):2216–2231, 2010.
  • Shevade and Keerthi (2003) S. K. Shevade and S. S. Keerthi. A simple and efficient algorithm for gene selection using sparse logistic regression. Bioinformatics, 19(17):2246–2253, 2003.
  • Tibshirani (1996) R. Tibshirani. Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B, 58(1):267–288, 1996.
  • Tropp (2004) J. A. Tropp. Greed is good: algorithmic results for sparse approximation. IEEE Transactions on Information Theory, 50(10):2231–2242, 2004.
  • Tsang et al. (2005) I. W. Tsang, J. T. Kwok, and P.-M. Cheung. Core Vector Machines: Fast SVM Training on Very Large Data Sets. JMLR, 6:363–392, 2005.
  • Tsitsiklis and Luo (1987) J. N. Tsitsiklis and Z.-Q. Luo. Communication complexity of convex optimization. Journal of Complexity, 3(3):231–243, 1987.
  • Wei and Ozdaglar (2012) E. Wei and A. E. Ozdaglar. Distributed Alternating Direction Method of Multipliers. In CDC, pages 5445–5450, 2012.
  • Yang (2013) T. Yang. Trading Computation for Communication: Distributed Stochastic Dual Coordinate Ascent. In NIPS, 2013.
  • Yao (1979) A. C.-C. Yao. Some complexity questions related to distributive computing. In STOC, pages 209–213, 1979.
  • Yuan and Lin (2006) M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. B, 68:49–67, 2006.

Appendix A Proof of Theorem 2

We first prove the following claim.

Claim 3.

At each iteration of dFW, each node has enough information to execute the algorithm. The communication cost per iteration is , where is an upper bound on the cost of broadcasting a real number to all nodes in the network .


Let and . We show that has enough information to compute its local gradient at iteration . The entry of its local gradient is given by

where . Notice that is a linear combination of , hence

Therefore, only depends on atoms which were broadcasted at iterations (step 4) and are thus available to at iteration . Furthermore, atom belongs to the local dataset of for all . It follows that has enough information to compute its local gradient at iteration and thus to execute step 3. Moreover, it is easy to see that can execute steps 4, 5 and 6 based on information broadcasted at earlier steps of the same iteration, which completes the proof of the first part of the claim.

At iteration , dFW requires node to broadcast the atom , and every node to broadcast a constant number of real values. Thus the communication cost of iteration is . ∎

Using Claim 3, we prove Theorem 2.


We prove this by showing that dFW is a Frank-Wolfe algorithm, in the sense that its updates are equivalent to those of Algorithm 1. For , at iteration of dFW, we have:

where the last equality comes from the fact that . Moreover, . Therefore, the update (step 5) of dFW is equivalent to the Frank-Wolfe update (Algorithm 1) for the constraint (Algorithm 2). The stopping criterion is also equivalent since

and . Consequently, dFW terminates after iterations and outputs a feasible that satisfies . Combining this result with Claim 3 establishes the total communication cost. The second part of the theorem follows directly from the fact that dFW uses Frank-Wolfe updates. ∎

Appendix B Extensions of the Lower Bound

In this section, we extend the communication lower bound (Theorem 3) to two important cases.

b.1 Problems with Constraint

The lower bound also applies for the case when the constraint is . Choose and consider the following problem

where is a orthonormal matrix. After scaling, it is equivalent to

Since , taking absolute values of the components in does not change the objective value. The problem is then equivalent to optimizing over the unit simplex, and the lower bound follows.

b.2 Larger Number of Nodes

The analysis also extends to the case where the network has more than two nodes. Select two nodes that are farthest apart in the network, put the data and on these two nodes, and by the same argument, the communication is lower bounded by bits, where is the diameter of the graph.