1 Introduction
Most machine learning algorithms are designed to be executed on a single computer that has access to the full training set. However, in recent years, largescale data has become increasingly distributed across several machines (nodes) connected through a network, for storage purposes or because it is collected at different physical locations, as in cloud computing, mobile devices or wireless sensor networks (Lewis, 2004; Armbrust et al., 2010). In these situations, communication between machines is often the main bottleneck and the naive approach of sending the entire dataset to a coordinator node is impractical. This distributed setting triggers many interesting research questions. From a fundamental perspective, studying the tradeoff between communication complexity (Yao, 1979; Abelson, 1980) and learning/optimization error has recently attracted some interest (Balcan et al., 2012; Daumé III et al., 2012; Shamir, 2014). From a more practical view, a lot of research has gone into developing scalable algorithms with small communication and synchronization overhead (see for instance Lazarevic and Obradovic, 2002; Forero et al., 2010; Boyd et al., 2011; Duchi et al., 2012; Dekel et al., 2012; Balcan et al., 2013; Yang, 2013) and references therein).
In this paper, we present both theoretical and practical results for the problem of learning a sparse combination of elements (atoms
) that are spread over a network. This has numerous applications in machine learning, such as LASSO regression
(Tibshirani, 1996)(Cortes and Vapnik, 1995), multiple kernel learning (Bach et al., 2004), Adaboost (Shen and Li, 2010) and sparse coding (Lee et al., 2006). Formally, let be a finite set of dimensional atoms (one per column). Here, atoms broadly refer to data points, features, dictionary elements, basis functions, etc, depending on the application. We wish to find a weight vector ( giving the weight assigned to atom ) that minimizes some cost function under a sparsity constraint:^{1}^{1}1Our results also hold for sparsity constraints other than the norm, such as simplex constraints.(1) 
where the cost function is convex and continuously differentiable and .
We propose a distributed FrankWolfe (dFW) algorithm, a novel approach to solve such distributed sparse learning problems based on an adaptation of its centralized counterpart (Frank and Wolfe, 1956; Clarkson, 2010; Jaggi, 2013). Intuitively, dFW is able to determine, in a communicationefficient way, which atoms are needed to reach an approximate solution, and only those are broadcasted through the network. We also introduce an approximate variant of dFW that can be used to balance the amount of computation across the distributed machines to reduce synchronization costs. In terms of theoretical guarantees, we show that the total amount of communication required by dFW is upperbounded by a quantity that does not depend on but only on , and the topology of the network. We further establish that this dependency on and is worstcase optimal for the family of problems of interest by proving a matching lowerbound of communication for any deterministic algorithm to construct an approximation. Simple to implement, scalable and parameterfree, dFW can be applied to many distributed learning problems. We illustrate its practical performance on two learning tasks: LASSO regression with distributed features and Kernel SVM with distributed examples. Our main experimental findings are as follows: (i) dFW outperforms baseline strategies that select atoms using a local criterion, (ii) when the data and/or the solution is sparse, dFW requires less communication than ADMM, a popular competing method, and (iii) in a realworld distributed architecture, dFW achieves significant speedups over the centralized version, and is robust to random communication drops and asynchrony in the updates.
Outline
Section 2 reviews the FrankWolfe algorithm in the centralized setting. In Section 3, we introduce our proposed dFW algorithm and an approximate variant of practical interest, as well as examples of applications. The theoretical analysis of dFW, including our matching communication lower bound, is presented in Section 4. Section 5 reviews some related work, and experimental results are presented in Section 6. Finally, we conclude in Section 7.
Notations
We denote scalars by lightface letters, vectors by bold lower case letters and matrices by bold upper case letters. For , we use to denote the set . For , is the all zeros vector except 1 in the entry. For a vector , the support of is defined as . The gradient of a differentiable function at is denoted by .
2 The FrankWolfe Algorithm
The FrankWolfe (FW) algorithm (Frank and Wolfe, 1956; Clarkson, 2010; Jaggi, 2013), also known as Conditional Gradient (Levitin and Polyak, 1966), is a procedure to solve convex constrained optimization problems of the form
(2) 
where is convex and continuously differentiable, and is a compact convex subset of any vector space (say ). The basic FW algorithm is shown in Algorithm 1. At each iteration, it moves towards a feasible point that minimizes a linearization of at the current iterate. The stopping criterion uses a surrogate optimality gap that can be easily computed as a byproduct of the algorithm. Let be an optimal solution of (2). Theorem 1 shows that FW finds an approximate solution in iterations.
Theorem 1 (Jaggi, 2013).
Solving the subproblems
The key step in FW is to solve a linear minimization subproblem on (step 3 in Algorithm 1). When the extreme points of have a special structure (e.g., when they are sparse), it can be exploited to solve this subproblem efficiently. In particular, when is an norm constraint as in problem (1), a single coordinate (corresponding to the largest entry in the magnitude of the current gradient) is greedily added to the solution found so far (Algorithm 2). Taking , we get . Combining this observation with Theorem 1 shows that Algorithm 1 always finds an approximate solution to problem (1) with nonzero entries, which is sometimes called an coreset of size (ShalevShwartz et al., 2010; Jaggi, 2011). It turns out that this is worstcase optimal (up to constant terms) for the family of problems (1), as evidenced by the derivation of a matching lower bound of for the sparsity of an approximate solution (Jaggi, 2011).^{3}^{3}3Note that the dependency on can be improved to for strongly convex functions by using socalled awaysteps (see for instance (LacosteJulien and Jaggi, 2013)). However, when is large, this improvement might not be desirable since it introduces a dependency on .
Another domain of interest is the unit simplex . The solution to the linear subproblem, shown in Algorithm 2, also ensures the sparsity of the iterates and leads to an approximation with nonzero entries, which is again optimal (Clarkson, 2010; Jaggi, 2013).
For both domains, the number of nonzero entries depends only on , and not on , which is particularly useful for tackling largescale problems. These results constitute the basis for our design of a distributed algorithm with small communication overhead.
3 Distributed FrankWolfe (dFW) Algorithm
We now focus on the problem of solving (1) in the distributed setting. We consider a set of local nodes which communicate according to an undirected connected graph where is a set of edges. An edge indicates that and can communicate with each other. To simplify the analysis, we assume no latency in the network and synchronous updates — our experiments in Section 6.3 nonetheless suggest that these simplifying assumptions could be relaxed.
The atom matrix is partitioned columnwise across . Formally, for , let the local data of , denoted by , be the set of column indices associated with node , where and for . The local gradient of is the gradient of restricted to indices in . We measure the communication cost in number of real values transmitted.
3.1 Basic Algorithm
Our communicationefficient procedure to solve (1) in the distributed setting is shown in Algorithm 3.^{4}^{4}4Note that it can be straightforwardly adapted to deal with a simplex constraint based on Algorithm 2. The algorithm is parameterfree and each iteration goes as follows: (i) each node first identifies the largest component of its local gradient (in absolute value) and broadcasts it to the other nodes (step 3), (ii) the node with the overall largest value broadcasts the corresponding atom, which will be needed by other nodes to compute their local gradients in subsequent iterations (step 4), and (iii) all nodes perform a FW update (step 5). We have the following result.
Theorem 2.
Algorithm 3 terminates after rounds and total communication, where is an upper bound on the cost of broadcasting a real number to all nodes in the network . At termination, all nodes hold the same with optimality gap at most and the atoms corresponding to its nonzero entries.
Proof.
We prove this by showing that dFW performs FrankWolfe updates and that the nodes have enough information to execute the algorithm. The key observation is that the local gradient only depends on local data and atoms received so far. Details can be found in Appendix A. ∎
Theorem 2 shows that dFW allows a tradeoff between communication and optimization error without dependence on the total number of atoms, which makes the algorithm very appealing for largescale problems. The communication cost depends only linearly on , and the network topology.
3.2 An Approximate Variant
Besides its strong theoretical guarantees on optimization error and communication cost, dFW is very scalable, as the local computational cost of an iteration typically scales linearly in the number of atoms on the node. However, if these local costs are misaligned, the algorithm can suffer from significant wait time overhead due to the synchronized updates. To address this issue, we propose an approximate variant of dFW which can be used to balance or reduce the effective number of atoms on each node. The algorithm is as follows. First, each node clusters its local dataset into groups using the classic greedy center algorithm of Gonzalez (1985), which repeatedly adds a new center that is farthest from the current set of centers (Algorithm 4). Then, dFW is simply run on the resulting centers. Intuitively, each original atom has a nearby center, thus selecting only from the set of centers is optimal up to small additive error that only leads to small error in the final solution. Optionally, we can also gradually add more centers so that the additive error scales as , in which case the error essentially does not affect the quality of the final solution. The details are given in Algorithm 5, and we have the following result.
Lemma 1.
Let denote the optimal radius of partitioning the local data indexed by into clusters, and let . Let . Then, Algorithm 5 computes a feasible solution with optimality gap at most after iterations. Furthermore, if , then the optimality gap is at most .
Proof.
By the analysis in Gonzalez (1985), the maximum cluster radius on node is at most at time , hence the maximum radius over all clusters is at most . For any , there exists such that
Thus the selected atom is optimal up to a additive error. Then the first claim follows from the analysis of Freund and Grigas (2013).
Similarly, when , the additive error is . The second claim then follows from the analysis of Jaggi (2013). ∎
Algorithm 5 has the potential to improve runtime over exact dFW in various practical situations. For instance, when dealing with heterogenous local nodes, can pick to be proportional to its computational power so as to reduce the overall waiting time. Another situation where this variant can be useful is when the distribution of atoms across nodes is highly unbalanced. We illustrate the practical benefits of this approximate variant on largescale Kernel SVM in Section 6.
3.3 Illustrative Examples
We give three examples of interesting problems that dFW can solve efficiently: LASSO regression, Kernel SVM and Adaboost. Others applications include sparse coding Lee et al. (2006) and multiple kernel learning Bach et al. (2004).
LASSO regression with distributed features
LASSO (Tibshirani, 1996)
is a linear regression method which aims at approximating a target vector
by a sparse combination of features:(3) 
where and are the target value and feature for training point , respectively. Our dFW algorithm applies to problem (3) in the context of distributed features (each local node holds a subset of the features for all training points), as common in distributed sensor networks (see e.g. Christoudias et al., 2008)
. It applies in a similar manner to sparse logistic regression
(Shevade and Keerthi, 2003) and, perhaps more interestingly, to GroupLASSO (Yuan and Lin, 2006). For the latter, suppose each group of features is located on the same node, for instance when combining features coming from the same source in multiview learning or when using dummy variables that encode categorical features. In this case, each atom is a group of features and dFW selects a single group at each iteration
(see Jaggi, 2013, for the FrankWolfe update in this case).Support Vector Machines with distributed examples
Let be a training sample with and . Let be a PSD kernel. The dual problem of L2loss SVM with kernel is as follows (Tsang et al., 2005; Ouyang and Gray, 2010):
(4) 
with where is the “augmented kernel”.
It is easy to see that with . Thus where is the atom matrix. Notice however that atoms may be of very high (or even infinite) dimension since they lie in kernel space. Fortunately, the gradient only depends on , thus we can broadcast the training point instead of (assuming all nodes know the kernel function ). Note that dFW can be seen as a distributed version of Core Vector Machines (Tsang et al., 2005), a largescale SVM algorithm based on FrankWolfe, and that it applies in a similar manner to a direct multiclass L2SVM (Asharaf et al., 2007) and to Structural SVM (see the recent work of LacosteJulien et al., 2013).
Boosting with distributed base classifiers
Let
be a class of base classifiers to be combined. Given a training sample
, let where . We consider the following Adaboost formulation (Shen and Li, 2010):(5) 
where is a tuning parameter. Our algorithm thus straightforwardly applies to (5) when base classifiers are distributed across nodes. The gradient of (5) is as follows:
The FrankWolfe update thus corresponds to adding the base classifier that performs best on the training sample weighted by , where defines a distribution that favors points that are currently misclassified. This can be done efficiently even for a large (potentially infinite) family of classifiers when a weak learner for is available.^{5}^{5}5A weak learner for returns the base classifier in that does best on the weighted sample defined by . In particular, when is the class of decision stumps and the features are distributed, each node can call the weak learner on its local features to determine its best local base classifier.
4 Theoretical Analysis
4.1 Communication Cost of dFW under Different Network Topology
We derive the communication cost of dFW for various types of network graphs, depicted in Figure 1.
Star network
In a star network, there is a coordinator node connected to all local nodes . Broadcasting a number to all nodes has cost , so the communication cost is . This can be improved to : instead of broadcasting and , we can send these quantities only to which selects the maximum and computes the sum of .
Rooted tree
Broadcasting on a tree costs , but this can be improved again by avoiding broadcast of and . To select the maximum , each node sends to its parent the maximum value among its local and the values received from its children. The root gets the maximum and can send it back to all the nodes. A similar trick works for computing the sum of . Hence, only and the selected atom need to be broadcasted, and the total communication is .
General connected graph
If the nodes can agree on a spanning tree, then this reduces to the previous case. We further consider the case when nodes operate without using any information beyond that in their local neighborhood and cannot agree on a spanning tree (this is known as the fully distributed setting, MoskAoyama et al., 2010). In this case, broadcasting can be done through a messagepassing procedure similar to the one used in Balcan et al. (2013). The cost of broadcasting a number is , and thus the total communication cost is .
4.2 Lower Bound on Communication Cost
In this section, we derive a lower bound on the communication required by any deterministic algorithm to construct an approximation to problem (1) for either the ball or the simplex constraint. We assume that at the end of the algorithm, at least one of the nodes has all the selected atoms. This is not a restrictive assumption since in many applications, knowledge of the selected atoms is required in order to use the learned model (for instance in kernel SVM).
Our proof is based on designing an optimization problem that meets the desired minimum amount of communication. First, we consider a problem for which any approximation solution must have selected atoms. We split the data across two nodes and so that these atoms must be almost evenly split across the two nodes. Then, we show that for any fix dataset on one node, there are different instances of the dataset on the other node, so that in any two such instances, the sets of selected atoms are different. Therefore, for any node to figure out the atom set, it needs bits. The proof is completed by showing .
4.2.1 Setup
Consider the problem of minimizing over the unit simplex , where is a orthonormal matrix (i.e., ). Suppose the first columns of (denoted as ) are on the node and the other half (denoted as ) are on the node . Restrict to be block diagonal as follows: vectors in only have nonzeros entries in coordinates and vectors in only have nonzeros entries in coordinates .
For any deterministic algorithm , let denote the set of atoms selected given input and , and let denote the minimum value of when and has only nonzero entries on . We only consider deterministic algorithms that lead to provably good approximations: satisfies for any . Furthermore, at the end of the algorithm, at least one of the nodes has all the selected atoms.
In practice, bounded precision representations are used for real values. Study of communication in such case requires algebraic assumptions (Abelson, 1980; Tsitsiklis and Luo, 1987). For our problem, we allow the columns in to be orthogonal and of unit length only approximately. More precisely, we say that two vectors and are nearorthogonal if , and a vector has nearunit length if . The notion of subspace is then defined based on nearorthogonality and nearunit length. We make the following assumption about the representation precision of vectors in subspaces.
Assumption 1.
There exists a constant s.t. for any dimensional subspace, , the number of different dimensional nearunit vectors in the subspace is .
4.2.2 Analysis
Let and (for simplicity, suppose is an integer). We first show that the atoms in any good approximation solution must be almost evenly split across the two nodes.
Claim 1.
Any with has more than nonzero entries. Consequently, for .
Proof.
First note that . When using exact real values, , thus the problem is equivalent to minimizing over the unit simplex. When using bounded precision representation, the exact equivalence does not hold. Still, by our definitions of nearunit vectors and nearorthogonal vectors, where each entry in has absolute value at most . So for any , .
Then, any with satisfies that where is the optimal value for . To see this, let denote the optimal solution for and denote the optimal solution for . We have:
which leads to . By Lemma 3 of Jaggi (2013), we have that for ,
So . Hence , which leads to the claim. ∎
Next, we show that for any fix dataset on one node, there are many different instances of the dataset on the other node, so that any two different instances lead to different selected atoms.
Claim 2.
For any instance of , there exist different instances of (denoted as ) such that the set of atoms selected from are different for all , i.e., for any ,
Proof.
Let denote the number of different sets of nearunit vectors in dimension that are nearorthogonal to each other. By Claim 1, if , then . Then there are at most different so that are all the same. Since there are different instances of , we have
Next, we observe that . We first choose a nearunit vector in the dimensional subspace, and then choose a nearunit vector in the dimensional subspace orthogonal to the first chosen vector, and continue in this way until nearunit vectors are selected. By Assumption 1, there are different choices, and there are repetitions, which leads to the bound on . The bound on then follows from . ∎
Distinguishing between the instances requires communication, resulting in our lower bound.
Theorem 3.
Suppose a deterministic algorithm satisfies for any . Under Assumption 1, the communication cost of is lowerbounded by bits.
Proof.
Suppose the algorithm uses less than bits. At the end of the algorithm, if node has the selected atoms, then must have determined these atoms based on and less than bits. By Claim 2, there are different instances of with different . For at least two of these instances, will output the same set of atoms, which is contradictory. A similar argument holds if node has the selected atoms. Therefore, the algorithm uses at least bits. ∎
4.2.3 Remarks
The same lower bound holds for constrained problems by reduction to the simplex case. The analysis also extends to networks with more than two nodes, in which case the diameter of the network is a multiplicative factor in the bound. Details can be found in Appendix B.
5 Related Work
5.1 Distributed Examples
An important body of work considers the distributed minimization of a function where is convex and only known to node . This typically corresponds to the risk minimization setting with distributed training examples, where gives the loss for the data points located on node . These methods are based on subgradient descent (SGD) (Duchi et al., 2012; Dekel et al., 2012, and references therein), the Alternating Direction Methods of Multipliers (ADMM) (Boyd et al., 2011; Wei and Ozdaglar, 2012) and Stochastic Dual Coordinate Ascent (SDCA) (Yang, 2013). Their rate of convergence (and thus their communication cost) for general convex functions is in or , sometimes with a dependency on , the total number of data points.
Note that for Kernel SVM (Section 3.3), none of the above methods can be applied efficiently, either because they rely on the primal (GD, ADMM) or because they require broadcasting at each iteration a number of data points that is at least the number of nodes in the network (SDCA). In contrast, dFW only needs to broadcast a single point per iteration and still converges in , with no dependency on . We illustrate its performance on this task in the experiments.
5.2 Distributed Features
In the case of distributed features, ADMM is often the method of choice due to its wide applicability and good practical performance (Boyd et al., 2011).^{6}^{6}6Note that to the best of our knowledge, no convergence rate is known for ADMM with distributed features. Given nodes, the parameter vector is partitioned as with , where . The matrix is partitioned conformably as and the optimization problem has the form
(6) 
where the loss is convex, the regularizer is separable across nodes and convex, and . Each node iteratively solves a local subproblem and sends its local prediction to a coordinator node, which then broadcasts the current global prediction . ADMM can be slow to converge to high accuracy, but typically converges to modest accuracy within a few tens of iterations (Boyd et al., 2011).
Both ADMM and dFW can deal with LASSO regression with distributed features. When features are dense, dFW and ADMM have a similar iteration cost but dFW is typically slower to converge as it only adds one feature to the solution at each iteration. On the other hand, the case of sparse features creates an interesting tradeoff between the number of iterations and the communication cost per iteration. Indeed, an iteration of dFW becomes much cheaper in communication than an iteration of ADMM since local/global predictions are typically dense. This tradeoff is studied in the experiments.
5.3 Relation to Matching Pursuit
6 Experiments
We have shown that dFW enjoys strong theoretical guarantees on the communication cost. In this section, we validate our analysis with a series of experiments on two tasks: LASSO regression with distributed features, and Kernel SVM with distributed examples. In Section 6.1, we compare dFW to two baseline strategies. Section 6.2 studies the communication tradeoff between dFW and ADMM, a popular distributed optimization technique. Finally, Section 6.3 proposes an evaluation of the performance of (exact and approximate) dFW in a realworld distributed environment.
6.1 Comparison to Baselines
We first compare dFW to some baseline strategies for selecting a subset of atoms locally. We investigate two methods: (i) the random strategy, where each node selects a fixedsize subset of its atoms uniformly at random, and (ii) the local FW strategy, where each node runs the FW algorithm locally to select a fixedsized subset as proposed by Lodi et al. (2010). The objective function on the union of the selected subsets is then optimized with a batch solver. We compare the methods based on the objective value they achieve for a given communication cost. In these experiments, we use a star graph with nodes and each atom is assigned to a node uniformly at random. Results are averaged over 5 runs.
Kernel SVM with distributed examples
We conduct experiments on the UCI Adult dataset, a census income database with about 32,000 examples and 123 features, so each node holds about 320 examples. We use the RBF kernel and set its bandwidth based on the averaged distance among examples and fix SVM parameter to . Figure 2(a) shows that the training examples selected by dFW lead to significantly smaller objective value than those selected by both baselines. Note that the local FW does much better than the random strategy but performs worse than dFW due to its local selection criterion.
LASSO regression with distributed features
We also conduct experiments on Dorothea, a drug discovery dataset from the NIPS 2003 Feature Selection Challenge
(Guyon et al., 2004). It has 1,150 examples and 100,000 binary features (half of which are irrelevant to the problem), so each node holds about 1,000 features. Figure 2(b) shows the results for (other values of do not change the relative performance of the methods significantly). Again, dFW significantly outperforms both baselines. On this dataset, these local strategies decrease the objective at a slow rate due the selection of irrelevant features.6.2 Comparison to ADMM
In this section, we compare dFW with distributed ADMM (Boyd et al., 2011), a popular competing method, on the problem of LASSO with distributed features, both on synthetic and real data. As in the previous experiment, we use
nodes and a uniform distribution of atoms, with results averaged over 5 runs.
Synthetic Data
The purpose of this experiment is to study the tradeoff between dFW and ADMM discussed in Section 5. To do so, we generate synthetic data following the protocol in (Boyd et al., 2011) but with varying density of the data and the solution. Specifically, we create a data matrix has examples and features (atoms) with density and its nonzero entries are drawn from . The true solution has density with each nonzero entry drawn from , and the target vector is computed as , where . We use , thereby creating a set of 9 problems. We set the regularization parameter to , where is the value above which the solution becomes the zero vector.^{7}^{7}7This choice results in a solution with a number of nonzero entries in the same order of magnitude as . The corresponding value of for the constrained problem is obtained from the norm of the solution to the regularized problem. For ADMM, we try several values of its penalty and relaxation parameters and , as done in (Boyd et al., 2011).
Figure 3 shows the MSE versus communication. These results confirm that dFW requires much less communication when the data and/or the solution are sparse (with no parameter to tune), while ADMM has the edge in the dense case (if its parameters are properly tuned). The rule of thumb seems to be that they perform similarly when , which is consistent with their communication cost per iteration and practical convergence speed.
Real data
We also conduct experiments on the Dorothea dataset used in Section 6.1. We compare dFW to ADMM for different values of , which affect the sparsity of the solution. The results shown for ADMM are the best over its penalty and relaxation parameters.
The results in terms of communication cost shown in Figure 4 (left plot) are consistent with those obtained on synthetic data. Indeed, dFW is better at finding sparse solutions with very little communication overhead, even though many features in this dataset are irrelevant. On the other hand, as the target model becomes more dense (), ADMM catches up with dFW.
Figure 4 (middle plot) shows that ADMM generally converges in fewer iterations than dFW. However, an ADMM iteration is computationally more expensive as it requires to solve a LASSO problem on each node. In contrast, a dFW iteration amounts to computing the local gradients and is typically much faster. To give a rough idea of the difference, we show the time spent on sequentially solving the local subproblems on a single 2.5GHz CPU core in Figure 4 (right plot).^{8}^{8}8Both algorithms are implemented in Matlab. For ADMM, LASSO subproblems are solved using a proximal gradient method. This shows that on this dataset, dFW decreases the objective faster than ADMM by 1 to 2 orders of magnitude despite its larger number of iterations.
6.3 Largescale Distributed Experiments
In this section, we evaluate the performance of dFW on largescale kernel SVM in realworld distributed conditions. Note that dFW is computationally very efficient if implemented carefully. In particular, the local gradient of a node can be recursively updated using only the kernel values between its points and the point received at the previous iteration. Thus the algorithm only needs memory and the computational cost of an iteration is . We implemented dFW in C++ with openMPI and experiment with a fully connected network with nodes which communicate over a 56.6gigabit infrastructure. Each node is a single 2.4GHz CPU core of a separate host. We use a speech dataset of 8.7M examples with 120 features and 42 class labels (the task is majority class versus rest).^{9}^{9}9Data is from IARPA Babel Program, Cantonese language collection release IARPAbabel101bv0.4c limited data pack. In all experiments, we use the RBF kernel and set its bandwidth based on the averaged distance among examples. The parameter of SVM is set to .
Exact dFW
We first investigate how dFW scales with the number of nodes in the network. For this experiment, training points are assigned to nodes uniformly at random. Figure 5(a) shows the runtime results for different values of . We obtain nearlinear speedup, showing that synchronization costs are not a significant issue when the local computation is wellbalanced across nodes.
Approximate variant
When the local costs are misaligned, the exact dFW will suffer from large wait time. As we have seen in Section 3.2, we can use the approximate variant of dFW to balance these costs. To illustrate this, we set and assign about 50% of the data to a single node while others node share the rest uniformly at random. We use the greedy clustering algorithm on the large node to generate a number of centers approximately equal to the number of atoms on the other nodes, thus balancing local computation.^{10}^{10}10For kernel SVM with RBF kernel, the difference between gradient entries of can be directly bounded by performing clustering in the augmented kernel space. This is because . Figure 5(b) shows that the approximate dFW reduces the runtime dramatically over exact dFW on the unbalanced distribution. As often the case in largescale datasets, training examples cluster well, so the additive error is negligible in this case.
Asynchronous updates
Another way to reduce waiting times and synchronization costs is to use asynchronous updates. To simulate this, we randomly drop the communication messages with some probability
. This means that at any given iteration: (i) the selected atom may be suboptimal, and (ii) some nodes may not update their iterate. As a result, the iterates across nodes are not synchronized anymore. Figure 5(c) shows the objective value averaged over all the nodes at each iteration. We can see that dFW is robust to this asynchronous setting and converges properly (albeit a bit slower), even with 40% random communication drops. This suggests that an asynchronous version of dFW could lead to significant speedups in practice. A theoretical analysis of the algorithm in this challenging setting is an exciting direction for future work.7 Conclusion
We studied the problem of finding sparse combinations of distributed elements, focusing on minimizing the communication overhead. To this end, we proposed a distributed FrankWolfe algorithm with favorable properties in communication cost. We showed a lowerbound to the communication cost for the family of problems we consider. This established that the proposed dFW is optimal in its dependency on the approximation quality . Experiments confirmed the practical utility of the approach.
Acknowledgments
This research is partially supported by NSF grants CCF0953192 and CCF1101215, AFOSR grant FA95500910538, ONR grant N000140910751, a Google Research Award, a Microsoft Research Faculty Fellowship and the IARPA via DoD/ARL contract # W911NF12C0012. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoD/ARL, or the U.S. Government.
References
 Abelson (1980) H. Abelson. Lower bounds on information transfer in distributed computations. Journal of the ACM, 27(2):384–392, 1980.
 Armbrust et al. (2010) M. Armbrust, A. Fox, R. Griffith, A. D. J., R. H. Katz, A. Konwinski, G. Lee, D. A. Patterson, A. Rabkin, I. Stoica, and M. Zaharia. A view of cloud computing. Communications of the ACM, 53(4):50–58, 2010.
 Asharaf et al. (2007) S. Asharaf, M. N. Murty, and S. K. Shevade. Multiclass core vector machine. In ICML, pages 41–48, 2007.
 Bach et al. (2004) F. R. Bach, G. R. G. Lanckriet, and M. I. Jordan. Multiple kernel learning, conic duality, and the SMO algorithm. In ICML, 2004.
 Balcan et al. (2012) M.F. Balcan, A. Blum, S. Fine, and Y. Mansour. Distributed Learning, Communication Complexity and Privacy. In COLT, 2012.

Balcan et al. (2013)
M.F. Balcan, S. Ehrlich, and Y. Liang.
Distributed kmeans and kmedian clustering on general communication topologies.
In NIPS, 2013.  Boyd et al. (2011) S. P. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. FTML, 3(1):1–122, 2011.
 Christoudias et al. (2008) C. M. Christoudias, R. Urtasun, and T. Darrell. Unsupervised feature selection via distributed coding for multiview object recognition. In CVPR, 2008.
 Clarkson (2010) K. L. Clarkson. Coresets, sparse greedy approximation, and the FrankWolfe algorithm. ACM Transactions on Algorithms, 6(4):1–30, 2010.
 Cortes and Vapnik (1995) C. Cortes and V. Vapnik. SupportVector Networks. Machine Learning, 20(3):273–297, 1995.
 Daumé III et al. (2012) H. Daumé III, J. M. Phillips, A. Saha, and S. Venkatasubramanian. Protocols for learning classifiers on distributed data. In AISTATS, pages 282–290, 2012.
 Dekel et al. (2012) Ofer Dekel, Ran GiladBachrach, Ohad Shamir, and Lin Xiao. Optimal Distributed Online Prediction Using MiniBatches. JMLR, 13:165–202, 2012.
 Duchi et al. (2012) J. C. Duchi, A. Agarwal, and M. J. Wainwright. Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling. IEEE Transactions on Automatic Control, 57(3):592–606, 2012.
 Forero et al. (2010) P. A. Forero, A. Cano, and G. B. Giannakis. ConsensusBased Distributed Support Vector Machines. JMLR, 11:1663–1707, 2010.
 Frank and Wolfe (1956) M. Frank and P. Wolfe. An algorithm for quadratic programming. Naval Research Logistics Quarterly, 3(12):95–110, 1956.
 Freund and Grigas (2013) Robert M. Freund and Paul Grigas. New Analysis and Results for the Conditional Gradient Method. Technical report, arXiv:1307.0873, 2013.
 Gonzalez (1985) Teofilo F Gonzalez. Clustering to minimize the maximum intercluster distance. Theoretical Computer Science, 38:293–306, 1985.
 Guyon et al. (2004) I. Guyon, S. R. Gunn, A. BenHur, and G. Dror. Result Analysis of the NIPS 2003 Feature Selection Challenge. In NIPS, 2004.
 Jaggi (2011) M. Jaggi. Sparse Convex Optimization Methods for Machine Learning. PhD thesis, ETH Zurich, 2011.
 Jaggi (2013) M. Jaggi. Revisiting FrankWolfe: ProjectionFree Sparse Convex Optimization. In ICML, 2013.
 LacosteJulien and Jaggi (2013) S. LacosteJulien and M. Jaggi. An Affine Invariant Linear Convergence Analysis for FrankWolfe Algorithms. Technical report, arXiv:1312.7864, 2013.
 LacosteJulien et al. (2013) S. LacosteJulien, M. Jaggi, M. Schmidt, and P. Pletscher. BlockCoordinate FrankWolfe Optimization for Structural SVMs. In ICML, 2013.
 Lazarevic and Obradovic (2002) A. Lazarevic and Z. Obradovic. Boosting Algorithms for Parallel and Distributed Learning. Distributed and Parallel Databases, 11(2):203–229, 2002.
 Lee et al. (2006) H. Lee, A. Battle, R. Raina, and A. Y. Ng. Efficient sparse coding algorithms. In NIPS, pages 801–808, 2006.
 Levitin and Polyak (1966) Evgeny S. Levitin and Boris T. Polyak. Constrained minimization methods. USSR Computational Mathematics and Mathematical Physics, 6(5):1–50, 1966.
 Lewis (2004) F. L. Lewis. Wireless sensor networks. Smart environments: technologies, protocols, and applications, pages 11–46, 2004.
 Lodi et al. (2010) S. Lodi, R. Ñanculef, and C. Sartori. SinglePass Distributed Learning of Multiclass SVMs Using CoreSets. In SDM, pages 257–268, 2010.
 MoskAoyama et al. (2010) D. MoskAoyama, T. Roughgarden, and D. Shah. Fully distributed algorithms for convex optimization problems. SIAM Journal on Optimization, 20(6):3260–3279, 2010.
 Ouyang and Gray (2010) H. Ouyang and A. G. Gray. Fast Stochastic FrankWolfe Algorithms for Nonlinear SVMs. In SDM, pages 245–256, 2010.
 ShalevShwartz et al. (2010) S. ShalevShwartz, N. Srebro, and T. Zhang. Trading Accuracy for Sparsity in Optimization Problems with Sparsity Constraints. SIAM Journal on Optimization, 20(6):2807–2832, 2010.

Shamir (2014)
Ohad Shamir.
Fundamental Limits of Online and Distributed Algorithms for Statistical Learning and Estimation.
In NIPS, 2014.  Shen and Li (2010) C. Shen and H. Li. On the Dual Formulation of Boosting Algorithms. TPAMI, 32(12):2216–2231, 2010.
 Shevade and Keerthi (2003) S. K. Shevade and S. S. Keerthi. A simple and efficient algorithm for gene selection using sparse logistic regression. Bioinformatics, 19(17):2246–2253, 2003.
 Tibshirani (1996) R. Tibshirani. Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B, 58(1):267–288, 1996.
 Tropp (2004) J. A. Tropp. Greed is good: algorithmic results for sparse approximation. IEEE Transactions on Information Theory, 50(10):2231–2242, 2004.
 Tsang et al. (2005) I. W. Tsang, J. T. Kwok, and P.M. Cheung. Core Vector Machines: Fast SVM Training on Very Large Data Sets. JMLR, 6:363–392, 2005.
 Tsitsiklis and Luo (1987) J. N. Tsitsiklis and Z.Q. Luo. Communication complexity of convex optimization. Journal of Complexity, 3(3):231–243, 1987.
 Wei and Ozdaglar (2012) E. Wei and A. E. Ozdaglar. Distributed Alternating Direction Method of Multipliers. In CDC, pages 5445–5450, 2012.
 Yang (2013) T. Yang. Trading Computation for Communication: Distributed Stochastic Dual Coordinate Ascent. In NIPS, 2013.
 Yao (1979) A. C.C. Yao. Some complexity questions related to distributive computing. In STOC, pages 209–213, 1979.
 Yuan and Lin (2006) M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. B, 68:49–67, 2006.
Appendix A Proof of Theorem 2
We first prove the following claim.
Claim 3.
At each iteration of dFW, each node has enough information to execute the algorithm. The communication cost per iteration is , where is an upper bound on the cost of broadcasting a real number to all nodes in the network .
Proof.
Let and . We show that has enough information to compute its local gradient at iteration . The entry of its local gradient is given by
where . Notice that is a linear combination of , hence
Therefore, only depends on atoms which were broadcasted at iterations (step 4) and are thus available to at iteration . Furthermore, atom belongs to the local dataset of for all . It follows that has enough information to compute its local gradient at iteration and thus to execute step 3. Moreover, it is easy to see that can execute steps 4, 5 and 6 based on information broadcasted at earlier steps of the same iteration, which completes the proof of the first part of the claim.
At iteration , dFW requires node to broadcast the atom , and every node to broadcast a constant number of real values. Thus the communication cost of iteration is . ∎
Proof.
We prove this by showing that dFW is a FrankWolfe algorithm, in the sense that its updates are equivalent to those of Algorithm 1. For , at iteration of dFW, we have:
where the last equality comes from the fact that . Moreover, . Therefore, the update (step 5) of dFW is equivalent to the FrankWolfe update (Algorithm 1) for the constraint (Algorithm 2). The stopping criterion is also equivalent since
and . Consequently, dFW terminates after iterations and outputs a feasible that satisfies . Combining this result with Claim 3 establishes the total communication cost. The second part of the theorem follows directly from the fact that dFW uses FrankWolfe updates. ∎
Appendix B Extensions of the Lower Bound
In this section, we extend the communication lower bound (Theorem 3) to two important cases.
b.1 Problems with Constraint
The lower bound also applies for the case when the constraint is . Choose and consider the following problem
where is a orthonormal matrix. After scaling, it is equivalent to
Since , taking absolute values of the components in does not change the objective value. The problem is then equivalent to optimizing over the unit simplex, and the lower bound follows.
b.2 Larger Number of Nodes
The analysis also extends to the case where the network has more than two nodes. Select two nodes that are farthest apart in the network, put the data and on these two nodes, and by the same argument, the communication is lower bounded by bits, where is the diameter of the graph.
Comments
There are no comments yet.