I Introduction
Largescale data often contains noise, statistical dependencies, and complex structure. To extract value from such data, we need both flexible, expressive models and scalable algorithms to perform reasoning over these models. In this paper, we show how a general class of distributed optimization techniques can be implemented efficiently on graphparallel abstractions frameworks.
Consensus optimization using alternating method of multipliers (ADMM), is a recently popularized general method for distributed solution of largescale complex optimization problems. The optimization problem is decomposed into simple subproblems to be solved in parallel. The combination of decomposability of dual ascent and the fast convergence of method of multipliers makes it suitable for many problems including distributed signal processing [2], inference in graphical models [3, 4]
, and popular machine learning algorithms
[5].Vertex programming is an efficient graphparallel abstraction for distributed graph computation. Pregel [6], Giraph [7] and GraphLab [8] are recently proposed frameworks for parallelizing graphintensive computation. These frameworks adopt a vertexcentric model to define independent programs on each vertex. Experiments show it outperforms the MapReduce abstraction by one to two orders of magnitude in machine learning and data mining algorithms [9].
In this paper, we investigate the bipartite topology of general ADMMbased consensus optimization and develop its vertexprogramming formulation. We also propose a novel partitioning scheme that utilizes the characteristics of computation graph based on a hypergraph interpretation of the bipartite data graph. Our partitioning can reduce the number of replicated vertices by an order of magnitude over the current stateoftheart partitioning scheme, reducing communication cost accordingly.
Our contributions include the following:

We develop a scalable, parallel algorithm for ADMMbased consensus optimization using the vertex programming abstraction. The underlying computation graph takes on a bipartite structure.

We propose a partitioning method that treats the bipartite graph as a hypergraph and performs a hypergraph cut to distribute the nodes across machines. This strategy enables our replication factor to outperform the stateoftheart graph partitioning scheme [8]. The running time of ADMMbased consensus optimization using our partition strategy is approximately half of the time compared to current stateoftheart methods. Our partitioning strategy is of independent interest, since it can be used for any problem that decomposes into a bipartite computation graph, such as factorgraph belief propagation [10].

We implement our algorithm in GraphLab and evaluate our partitioning scheme on a social network analysis problem, demonstrating that we can perform largescale probabilistic inference on a modest number of machines, even when the graph has the heavy degree skew often found in real networks.
Ii Motivation
Before describing our proposed algorithm (Section IV), we begin with a simple illustrative example of the kind of problems that can be solved using ADMM. We consider the task of social network analysis on a network of individuals connected by various social relationships. The goal of the analysis is to predict the voting preferences of various individuals in the social network, using the relationships and some observed voting preferences of other individuals in the network. The problem can be cast as a probabilistic inference problem, and the approach that we take here is to define the model using probabilistic soft logic (PSL) [11]
. PSL is a generalpurpose language for describing largescale probabilistic models over continuousvalued random variables using weighted logical rules.
A PSL program consists of a set of logical rules with conjunctive bodies and disjunctive heads (negations allowed). Rules are labeled with nonnegative weights. The following program, based on an example from Bach et al. [4], encodes a simple model to predict voter behavior using information about a voter (voter registration) and their social network described by two types of links indicating Friend and Spouse relationships:
1.1:  RegisteredAs(A, P)  
0.5:  
1.8: 
Consider any constants for persons, and and party instantiating logical terms , , and respectively. The first rule encodes the correlation between voter registration and party preferences, which tend to be aligned but are not always. The next rule states that if is a friend of and votes for party , there is a chance that votes for party as well, whereas the second makes the same statement for spouses. The rule weights indicate that spouses are more likely to vote for the same party than friends. The resulting probabilistic model will combine all of these influences and include the implied structured dependencies. We can also consider more rules and relationship types, leading to a full program in Figure 1. PSL can also include constraints on logical atoms, such as mutual exclusivity of voting preferences Votes.
0.5:  RegisteredAs(A, P)  
0.3:  Votes(A, P)  
0.1:  Votes(A, P)  
0.05:  Votes(A, P)  
0.1:  Votes(A, P)  
0.7:  Votes(A, P) 
The engine behind PSL compiles the logical program into a continuousvariable representation known as a
hingeloss Markov random field (HLMRF) [4, 12]. Like many probabilistic graphical models, inference in HLMRFs can be distributed and solved using consensus optimization. In HLMRFs, inference of the mostprobable explanation (MPE) is a convex optimization, since the logical rule are converted into hingeloss potentials and constraints such as mutual exclusivity can be relaxed to linear equalities. Since inference is a convex optimization, HLMRFs are particularly wellsuited for consensus optimization. We defer to previous papers for the mathematical formalisms of HLMRFs, and here mainly discuss the general implementation of ADMMbased consensus optimization, which has many applications beyond PSL.
Iii Preliminaries
Iiia ADMMBased Consensus Optimization
Consensus optimization simplifies solution of a global objective by decomposing a complex objective into simpler subproblems over local copies of the variables and constraining each local copy to be equal to a global consensus variable. The general form of the consensus optimization is:
(1)  
subject to 
where with dimension
is the local variable vector on which the
th subproblem depends and is the objective function for the th subproblem. For ease of notation, let denote the global consensus variable vector that the local variable should equal.To solve the consensus optimization, ADMM relaxes the global equality constraints using an augmented Lagrangian and solves the dual objective. The ADMMbased solution procedure is [1]:
(2) 
where superscript represents the iteration, is the number of local copies of the th entry of global consensus variable, is the vector of Lagrange multipliers for th subproblem, is the corresponding global consensus entry for th dimension of local variable , denotes the th entry of global consensus variable, and is a step size parameter. The update of can be viewed as averaging local copies in subproblems.
Equation IIIA shows that subproblems can be solved independently. This general form of consensus optimization defines a bipartite graph structure where denotes the set of subproblems containing local variables , represents the set of consensus variable entries, and expresses the dependencies as shown in Figure 2. Each subproblem is connected to its dependent consensus variables, while each consensus variable is connected to subproblems containing its local copies.
As described above, the computation required for solving a subproblem depends on only a subset of the consensus variable nodes, while the computation needed for a consensus variable requires information about only a subset of the local variables. This dependency relationship makes ADMM computation wellsuited for vertexprocessing parallelization.
IiiB Vertex Programming Frameworks
Recent development of vertex programming frameworks, such as Pregel [6] and GraphLab [9], are aimed at improving the scalability of graph processing. Vertexcentric models in these systems execute userdefined functions on each vertex independently and define the order of execution of vertices. Pregel and GraphLab have superior computational performance over MapReduce for many data mining and machine learning algorithms, such as belief propagation, Gibbs sampling, and PageRank [6, 9, 13].
In this paper, we implement our synchronous vertex programs in GraphLab.
IiiC GAS Vertex Programming API
Gonzalez et al. [8] propose the gatherapplyscatter (GAS) abstraction that describes common structures of various vertex programming frameworks. In the vertex programming setting, the user defines a data graph with data structures representing vertices, edges, and messages. The user provides a vertex program associated with each vertex. The GAS model abstracts the program into three conceptual phases of execution on each vertex. In the gather phase, each vertex is able to aggregate neighborhood information, which could be pushed or pulled from adjacent nodes. The aggregation during this phase is userdefined, but must be commutative and associative. In the apply phase, each vertex can use its aggregated value to update its own associated data. Finally in the scatter phase, each vertex either sends messages to its neighbors or updates other vertices or edges in its neighborhood via global state variables.
Iv Distributed ADMMbased consensus optimization implementation (aco)
In order to implement a graph algorithm using vertex programming, one needs to define the data graph structure including the vertex, edge and message data types, as well as a vertex program that defines the computation.
Iva Data Graph and Data Types
Recall that in ADMMbased consensus optimization, the computation graph is based on the dependencies between subproblems and consensus variables described by the bipartite dependency graph (Fig. 2). In this graph, any consensus variable has a degree of at least two, since any local variables that only appear in one subproblem do not need consensus nodes.
For each node in the bipartite graph, we construct a vertex in the vertex program. We use different data types for subproblem vertices and consensusvariable vertices, denoted and , respectively. For each subproblem vertex , we maintain the involved local variables in and associated Lagrange multiplier , both of which are dimension vectors. Each also stores a vector of dimensionality for holding the dependent consensus variable values. For each consensus variable node , we only store its current value.
IvB aco Vertex Program
We use the GAS abstraction introduced in Section IIIC to describe our ADMMbased consensus optimization implementation, aco, shown in Alg. 1. In each iteration, we define a temporary consensusvariable keyvalue table consensus_var, where the key is a consensus variable’s global unique id, i.e., consensus_var. We also define a program variable local_copy_sum to aggregate the sum of each local copy of a consensus variable.
As shown in Alg.1, in the gather, apply and scatter stages, we alternate computation on the subproblem nodes and the consensus nodes . We describe the computation and communication for each node type below.
IvB1 Subproblem Nodes
In the gather phase of the iteration, each subproblem node reads the consensus variables updated in iteration in its neighborhood. We store each consensus variable in the keyvalue table consensus_var. The commutative and associative aggregation function here combines the keyvalue tables. After getting the updated consensusvariable table, we use it to solve the optimization subproblem in line 13, and the vector is updated to the solution. Note that the subproblem solver is applicationspecific and is defined by the user. In the scatter phase, the subproblem notifies dependent consensus nodes if was updated.
IvB2 Consensus Variable Node
The consensus variable node behaves differently in the vertex program. It aggregates all local copies of it from subproblem nodes using summation in gather phase, then update itself with the average value in apply method. In the scatter phase, convergence conditions are used to determine whether related sub problems need to be scheduled to run again.
IvB3 Termination Conditions
One possible criterion for convergence is the global primal and dual residual of all consensus variables and their local copies. At the superstep, an aggregator can be used to aggregate residuals across all consensus variables. If both primal and dual residuals are small enough, then we have reached global convergence and the program stops. If not, all subproblems will be scheduled again in the next iteration. However, this global convergence criterion has two disadvantages: first, the use of aggregator will bring overhead as it needs to aggregate information from all distributed machines; second, some consensus variables and their corresponding local copies do not change much and their subproblem counterparts are still scheduled to run, wasting computation resources.
Instead, our proposed convergence criterion measures local convergence. In this local criterion, we check the primal residual and dual residual for local copies of a consensus variable only. Each consensus vertex calculates both primal residual and dual residual using its dependent local copies. If both of them are small, subproblems connecting to this consensus variable will not be notified in the following iteration. The notifications to a particular subproblems come from all consensus variables connecting to that particular subproblem vertex. If none of the connected consensus variables notify it, the subproblem skips the following iteration, thus saving computation. A skipped subproblem node will be notified again if its dependent consensus variable is updated and the convergence criterion is not met.
V Hypergraph partitioning
In this section, we present our new hypergraphbased partitioning scheme (Hyper) that is better suited for aco than current stateoftheart approaches. The primary factors for efficient implementation of distributed graph algorithms are load balancing and communication cost. Vertex programming frameworks ensure good load balancing using a balanced way cut of the graph. Significant communication costs result from vertices whose neighbors are assigned to different machines. Pregel [6] uses edge cut, mirrors vertices, and proposes message combiners to reduce communication, while GraphLab uses vertexcut. An edgecut has been proven to be convertible to a vertexcut with less communication overhead [8]. Therefore, we focus on vertex partitioning in the rest of our discussion in this section.
Va Problem Definition and Notation
Let be a general graph, be a parameter determining the imbalance. For any , let denote the subset of machines that vertex is assigned to. Then the balanced way vertex cut problem for machines is defined as
(3)  
subject to 
where . The objective corresponds to the replication factor, or how many copies of each node exist across all machines, and the constraint corresponds to a limit on the edges that can be assigned to any one machine.
VB Intuition and StateoftheArt
The current stateoftheart strategy used in GraphLab is a sequential greedy heuristic algorithm
[8], which we refer to as Greedy. Multiple machines process sets of edges one by one and place each of them into a machine, where the placement is maintained across multiple machines. When a machine places an edge , the Greedy strategy follows heuristic rules: if both and are , edge is placed on the machine with the fewest assigned edges; if only one of and is not , say , then is put in one machine in ; if , then is assigned to one of the machines in the intersection; the last case is both and are not , but , then is assigned to one of the machines from the vertex with the most unassigned edges.As we will show, the Greedy strategy does not work well with the aco bipartite graphs. In aco and other similar problem structures, subproblem nodes tend to have much lower degree than consensus nodes. Because the last heuristic in the greedy scheme is biased to large degree nodes, Greedy places a large number of subproblem nodes onto different machines.
In practice, largescale aco involves millions of consensus variables, so computing a highquality partitioning is more important than fast sequential partitioning. Once partitioned, the same topology may be reused multiple times, for example when performing parameter optimization. In the rest of our discussion, we investigate properties of the aco bipartite graph and propose a novel and efficient partitioning scheme.
VC Specific Properties of Consensus Optimization
We assume the bipartite graph of the aco exhibits four characteristics:

The consensus variable nodes have a powerlaw degree distribution , where is a shape parameter.

The subproblem degree distribution is centered around some small number. We uses a Poisson distribution for the simplicity of analysis, i.e.,
. 
For largescale optimizations, the number of subproblems is larger than the number of consensus variables, i.e., .

The algorithm strictly follows the bipartite structure, i.e., the computation of only depends on and the computation of only depends on .
We briefly justify each of these characteristics. The first characteristic above results from natural powerlaw degree distributions found in realdata applications. Such degree distributions are common in largescale aco problems when the variables involved in subproblems correspond to objects in the real world, especially for applications on social and natural networks.
The second characteristic is a standard requirement for the utility of the ADMM decomposition, where the original optimization problem decomposes into small subproblems that are each easy to solve. Thus the degree of each subproblem node (the number of variables it involves) is small. The Poisson distribution is commonly used to model the number of events occurring during a fixed interval or space. Therefore, it is suitable to describe the number of variables that are involved in a subproblem. The parameter describes the average number of variables in a subproblem and should be small.
The third characteristic similarly corresponds to the utility of consensus optimization. The rich models we aim to reason over typically include overlapping interactions among variables. These overlapping interactions make the distribution complex and thus make direct optimization cumbersome. By decomposing the problem into many subproblems that share a smaller set of variables, inference becomes easier. In practice, the number of subproblems is roughly an order of magnitude greater than the number of variables.
The fourth characteristic, that the computation of nodes in depends only on the values of a subset of and the computation of nodes in depends only on the values of a subset of , exhibits itself in many factorbased representations of probabilistic models. This characteristic provides us with extra information about the structure of computation and, when combined with the third characteristic, motivates partitioning only the consensus variables instead of partitioning over the whole set of nodes , which inevitably introduces expensive redundancy.
VD Analysis of aco
We first analyze Random over all nodes in and show its inefficiency in bipartite graphs with the four characteristics mentioned above, then we present our novel partitioning strategy. If we use random vertexcut without considering the data dependency structure, the expected number of replications of a random vertexcut on the bipartite graph is
where , is the set of machines on which variable is located, is the total number of machines, is a consensus variable’s degree (powerlawdistributed), and is a subproblem’s degree (Poissondistributed). This analysis is direct extension of that in [8]. The factors affecting the expected replication factor are the number of machines, the parameter of powerlaw distribution , and the parameter of Poisson distribution . We evaluate how these parameters affect the replication factor on synthetic data in our experiments (Section VI).
If we further assume that each subproblem itself is not very complex and involves just a constant number of variables, the expected number of replications is:
Thus the expected replication factor increases linearly with the number of subproblems .
VE Hypergraphbased Bipartite Graph Partitioning
Since the random strategy is agnostic to the graph structure, it should be possible to exploit the known bipartite graph structure to improve efficiency. In particular, the bipartite structure suggests that we should aim to partition the consensus variables only. The objective function of the vertex partition over a bipartite graph is defined as
(4)  
subject to  
When assigning a consensus variable node a machine, we also assign the associated subproblems to that same machine. Because we are cutting only the consensus variable nodes, for each edge located on a machine, the workload added to that machine is the load of the corresponding subproblems plus the workload of consensus variable, which only involves a simple averaging. Therefore, the edge balance in the equation above is equivalent to subproblem node balance, which can be reduced to the following constrained optimization:
(5)  
subject to  
where is the assignment of consensus variables to machines, is the mapping from subproblems to machines when the assignment is given, and is the imbalance factor.
We perform vertex cut only on consensus nodes by treating it as a hyperedge partitioning in the hypergraph view of the bipartite graph. The bipartite graph is converted into a hypergraph as follows: each node is a also a node in ; each node is a hyperedge connecting the set of nodes in corresponding to ’s neighbors in . We thus denote the hypergraph as where is the hyperedge set. Immediately, one can see that the vertex cut of only consensus nodes in reduces to Hyper in .
The hyperedge partitioning problem has been wellstudied and there are various packages that can perform hyperedge partitioning efficiently. In our experiments, we use hMETIS [14]. The objective function is to minimize the sum of external degrees: , where is the set of subproblems that are assigned to the th machine, is the external degree of partition , i.e., the number of hyperedges that are incident but not fully inside . Because consensus nodes in have a onetoone mapping to hyperedges in , minimizing the sum of external degrees in is equivalent to minimizing the replication factor in .
Vi Experiments
In this section, we first compare our hypergraph partitioning vertexcut technique Hyper with the greedy vertexcut algorithm Greedy and the hashbased random partitioning Random introduced in Section V. We then present the evaluation of our aco implementation using GraphLab on the largescale social network analysis problem introduced in Section II. In all of our experiments, we use hMetis [15] with unbalanced factor and use the sum of external degree objective to perform the hyperedge cut. Code and data for all experiments will be made available at http://linqs.cs.umd.edu/admm.
Via Evaluation of Partitioning Strategies
Replication factor  

Hyper  Greedy  Random  
2.0  1.5  1,254,452  1,811,449  11.54  1.10  1.44  2.24 
2.0  1,015,092  1,661,788  9.15  1.14  1.64  2.61  
2.5  799,850  1,389,912  7.00  1.17  1.74  3.00  
3.0  662,938  1,247,468  5.63  1.25  1.88  3.41  
3.5  578,983  1,142,410  4.79  1.28  1.97  3.78  
2.2  1.5  647,396  1,051,772  5.47  1.18  1.62  2.44 
2.0  514,906  902,526  4.15  1.25  1.75  2.79  
2.5  409,645  792,021  3.10  1.37  1.93  3.16  
3.0  363,194  756,398  2.63  1.45  2.08  3.48  
3.5  319,539  708,340  2.20  1.57  2.22  3.80  
2.4  1.5  450,976  704,064  3.51  1.24  1.56  2.53 
2.0  356,921  614,450  2.57  1.34  1.72  2.84  
2.5  303,164  559,470  2.03  1.45  1.85  3.13  
3.0  271,035  541,912  1.71  1.55  2.00  3.40  
3.5  249,170  522,232  1.49  1.63  2.10  3.65  
2.6  1.5  375,734  580,372  2.76  1.25  1.54  2.53 
2.0  308,738  515,364  2.09  1.34  1.67  2.79  
2.5  265,934  474,523  1.66  1.44  1.78  3.04  
3.0  237,711  451,492  1.38  1.53  1.90  3.27  
3.5  218,271  437,161  1.18  1.62  2.00  3.49  
2.8  1.5  335,411  507,033  2.35  1.24  1.51  2.50 
2.0  278,068  450,796  1.78  1.33  1.62  2.73  
2.5  242,126  420,322  1.42  1.42  1.74  2.96  
3.0  218,350  400,794  1.18  1.50  1.84  3.14  
3.5  200,716  385,167  1.01  1.58  1.92  3.33 
ViA1 Dataset Description
We begin by studying the effect of different partitioning strategies on the replication factor of the vertex program. To produce a dataset that embodies the rich characteristics discussed in Section V, we generate graphs with a powerlaw degree distribution for consensus variable vertices with parameter and a Poisson distribution over the degree of subproblems with parameter . We use the random bipartite generator in the python package NetworkX [16], which generates random bipartite graphs from two given degree sequences.
We fix the number of consensus variable vertices to and generate two degree sequences with varying and . Since in natural graphs is roughly 2.2 [17] and subproblems from real problems tend to be small, we vary from 2.0 to 2.8, and vary from 1.5 to 3.5. We use rejection sampling to remove samples with any nodes of degree less than 2. Because the sum of degrees for and should be equal, their proportion () can be derived from , , and . We list the properties of generated datasets in Table I, where is the number of vertices and shows number of edges.
ViA2 Replication Factor Results with Synthetic Data
Given the dataset listed in Table I, we vary the number of machines (partitions) and measure the replication factor of each scheme, denoted as Hyper, Greedy, and Random accordingly in Table I. The parameters’ default values are set to: , , . We list the results in the last three columns of Table I and in Fig. 3. In general, in all generated datasets, Hyper always has a smaller replication factor than Greedy and Random. In the worst case, Greedyreplicates around more vertices than Hyper (), and always replicates more ().
In Fig. 3(a), we vary to show how replication factor grows when the number of machines increases. The results show that Hyper is less sensitive to the number of machines than the other schemes and scales better in practice.
Next we fix , vary and in Fig. 3(b) and Fig. 3(c) to study the partitioning performance based on different bipartite graph topologies. Recall that parameter determines the powerlaw shape. As shown in Fig. 3(b), the larger is, the smaller the maximum degree of the consensus nodes become, and difference between and is smaller, e.g., when in the plot, is only . In this case, cutting the consensus nodes in Hyper provides less improvement over Greedy. On the other hand, when increases, each subproblem has more variables, and the number of subproblems decreases. Hyper tends to cut more subproblem nodes, as shown in Fig. 3(c).
Finally, in Fig. 3(d), we plot the relationship between replication factor and the proportion between and . When the proportion is small (), Hyper and Greedy scale similarly, but Hyper is still better. As the proportion increases, the advantage of Hyper much more pronounced.
In summary, our proposed hypergraphbased vertexcut scheme outperforms the stateoftheart Greedyscheme provided in GraphLab implementation [8, 18] for realistic bipartite graph settings. Especially when the two types of nodes in the bipartite graph are imbalanced, which is typically the case in consensus optimization, Hyper can generate much higher quality partitions.
ViB Performance of aco for PSL Voter Model
Next we compare the performance of our proposed aco vertexprogramming algorithm empirically on an MPI 2 cluster using OpenMPI 1.4.3 consisting of eight Intel Core2 Quad CPU 2.66GHz machines with 4GB RAM running Ubuntu 12.04 Linux. We implement our algorithm using GraphLab 2 (v2.1.4245) [18]. For each machine in the cluster, we start only one process with 4 threads (ncpus). We use the synchronous engine provided by GraphLab 2, which is explained in detail in [8]. Our proposed approach can be applied to other vertex programming frameworks easily since it does not use any special features of GraphLab beyond the synchronous GAS framework.
ViB1 Voter Network Dataset Description
Name  

3,307,971  1,102,498  6,011,257  3.00  
6,656,775  2,101,072  12,107,131  3.17  
9,962,627  3,149,103  18,113,119  3.16  
13,349,751  4,203,703  24,288,223  3.18 
We generate social voter networks using the synthetic generator of Broecheler et al. [19, 4] and create a probabilistic model using the PSL program in Section II. The details of the datasets are listed in Table II. The smallest one has 6 million edges and 4.4 million vertices and fits in 4GB memory on a single machine when loaded in GraphLab; the rest of the datasets do not fit on a single machine. The fifth column in Table II shows that the proportion between two sets in our data graph is around 3. In the voter PSL model, the variables corresponding to the truth of the Votes(person,party) predicate are consensus variables, and each initialized rule maps to a subproblem. Each Votes(person, party) appears in at most eight rules. In practice, PSL programs can be far more complex and many more subproblems can be grounded, thus the proportion may be even larger. In such cases, acoHyper partitioning will even further reduce communication cost.
ViB2 Performance Results with PSL Inference
To evaluate the performance of our algorithm, we use a GraphLab vertex program that implements our aco algorithm described in Section IV and vary the partitioning technique. We use the method described in [4] to solve the quadratic subproblems defined by voter PSL program. We consider performance of aco under two settings: full convergence and early stopping when one considers computation time budgets. It is important to consider the earlystopped setting since aco is known to have very fast initial convergence and then slow convergence toward the final optimum [1]. In practice, one can stop early when the majority of variables have converged and quickly obtain a highquality approximate solution. As shown in Fig. 5, inference in the PSL voter model quickly converges on 99% of the consensus variables, taking 1,000 iterations on all four datasets.
Full convergence: As shown in Fig. 4(a) and 4(b), we first vary to show the running time and speedup under the full convergence setting. Because is able to fit into memory on one machine, communication cost overwhelms extra computation resources, and prevents distributed computation from performing better than single machine. In Fig. 4(a), acoGreedy and acoRand perform ( to ) worse than the single machine setting. acoHyper has similar running times to those of a single machine and is better than acoGreedy.
On larger data sets that cannot fit on a single machine, our approach is approximately twice as fast as acoGreedy (Fig. 4(b) and 4(c)). However, the speedup for the full convergence setting is not as significant because some consensus variables take many iterations to converge. Fewer than 1% of the consensus variables are still active after 1,000 iterations, but as long as any one variable has not converged, increasing the number of machines will not produce speedup in terms of computation time. In Fig. 4(c), we evaluate weak scaling by increasing machine and dataset together. Both Hyper and Greedy scale well on large datasets that cannot fit into one machine.
Performance under early stopping: As shown in Fig. 5, the majority of nodes converge quickly. Since modern computing models often include a payasyougo cost, one may not benefit from waiting for the last few variables to converge. For instance, the last 1% of vertices in take of the total time for full convergence. Motivated by this reasoning, in the following experiments, we measure the running time to complete 1,000 iterations of acoHyper and acoGreedy, regardless of the convergence status at the end of the last iteration.
In Fig. 6(a), we show the accumulated running time of each iteration. Note because we use synchronous setting, both algorithms have the same state at the end of each iteration. In Fig. 6(a), we show that acoHyper performs to better than the acoGreedy because of the reduction in communication cost. In Fig. 6(b), we vary the number of machines for the same dataset to show the speed up. There are diminishing returns on increasing the number of machines, due to the communication overhead incurred by adding machines, but our hypergraph partitioning produces overall faster computation.
Vii Related Work
The ADMM algorithm, recently popularized by Boyd et al. [1], has been used for many applications such as distributed signal processing [2] and inference in graphical models [3]. In particular, many largescale distributed optimization problems can be cast as consensus optimization and use ADMM to solve them [5, 4, 12].
Tziratas et al. [3] proposed a general messagingpassing scheme for MRF optimization based on dual decomposition. Their solution has a masterslave MRF structure that is analogous to our bipartite topology. Building on this work, dual decomposition has similarly been proposed to perform distributed inference in Markov Logic Networks (MLNs) [20]. This work showed that combining MRFlevel partitioning and programlevel partitioning produces superior performance compared with just MRFlevel partitioning. MRFlevel partitioning treats the grounded MLN as a collection of trees, which is different from our partitioning objective.
There are several contributions which discuss implementations of ADMMrelated distributed algorithms. Boyd et al. [1] discusses the implementation of ADMM in MapReduce with global consensus variables, which is a special case of our setup. GraphLab [21] contains implementations of the MRF dual decomposition from [3]. Also related, a largescale implementation of the “accuracy of the top” algorithm [22]
proposes methods for speeding up convergence of the top quartile, including tuning the communication topology, for an ADMM consensus optimization algorithm written in Pregel.
Viii Conclusion
In this paper, we introduce a vertex programming algorithm for distributed ADMMbased consensus optimization. To mitigate the communication overhead of distributed computation, we provide a novel partitioning strategy that converts the bipartite computation graph into a hypergraph and uses a wellstudied hypergraph cut algorithm to assign nodes to machines. This combination of the ADMM vertex program and hypergraph partitioning enables distributed optimization over largescale data. Our experiments on probabilistic inference over largescale, synthetic social networks demonstrate that our contributions lead to a significant improvement in scalability. Additionally, the partitioning scheme is of independent interest to researchers and practitioners, since many other graph algorithms also have a bipartite computation structure and will similarly benefit from the reduced communication overhead induced by hypergraph partitioning.
References
 [1] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends in Machine Learning, vol. 3, no. 1, 2011.
 [2] E. Esser, “Applications of Lagrangianbased alternating direction methods and connections to split Bregman,” in UCLA Technical report, 2009.

[3]
N. Komodakis, N. Paragios, and G. Tziritas, “MRF optimization via dual
decomposition: Messagepassing revisited,” in
IEEE International Conference on Computer Vision
, 2007.  [4] S. Bach, M. Broecheler, L. Getoor, and D. O’Leary, “Scaling MPE inference for constrained continuous markov random fields with consensus optimization,” in Advances in Neural Information Processing Systems (NIPS), 2012.

[5]
P. Ferero and A. Cano, “Consensusbased distributed support vector machines,”
Machine Learning Research, vol. 11, pp. 1663–1707, 2010.  [6] G. Malewicz, M. Austern, A. Bik, J. Dehnert, I. Horn, N. Leiser, and G. Czajkowski, “Pregel: A system for largescale graph processing,” in ACM SIGMOD International Conference on Management of Data, 2010.
 [7] “Apache Giraph,” http://giraph.apache.org/, 2013.
 [8] J. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin, “PowerGraph: Distributed graphparallel computation on natural graphs,” in USENIX Conference on Operating Systems Design and Implementation, 2012.
 [9] Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J. Hellerstein, “Distributed GraphLab: A framework for machine learning and data mining in the cloud,” VLDB Endowment, vol. 5, no. 8, pp. 716–727, 2012.
 [10] F. Kschischang, B. Frey, and H. Loeliger, “Factor graphs and the sumproduct algorithm,” IEEE Transactions on Information Theory, vol. 47, no. 2, 2001.
 [11] A. Kimmig, S. Bach, M. Broecheler, B. Huang, and L. Getoor, “A short introduction to probabilistic soft logic,” in NIPS Workshop on Probabilistic Programming, 2012.
 [12] S. Bach, B. Huang, B. London, and L. Getoor, “Hingeloss Markov random fields: Convex inference for structured prediction,” in Uncertainty in Artificial Intell., 2013.
 [13] Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J. Hellerstein, “GraphLab: A new framework for parallel machine learning,” in Uncertainty in Artificial Intell., 2010.
 [14] G. Karypis and V. Kumar, “Multilevel kway hypergraph partitioning,” in ACM/IEEE Design Automation Conf., 1999.
 [15] “hMetis v2.0pre1,” http://glaros.dtc.umn.edu/gkhome/fetch/sw/hmetis/hmetis2.0pre1.tar.gz, May 2007.
 [16] “Python NetworkX package 1.7,” http://networkx.github.io/.
 [17] M. Faloutsos, P. Faloutsos, and C. Faloutsos, “On powerlaw relationships of the internet topology,” in ACM SIGCOMM Comp. Comm. Rev., vol. 29, no. 4. ACM, 1999.
 [18] “GraphLab 2 v2.1.14245,” https://graphlabapi.googlecode.com/files/graphlabapi_v2.1.4245.tar.gz, August 2012.
 [19] M. Broecheler, P. Shakarian, and V. Subrahmanian, “A scalable framework for modeling competitive diffusion in social networks,” in IEEE Int. Conf. on Social Comp., 2010.
 [20] F. Niu, C. Zhang, C. Re, and J. Shavlik, “Scaling inference for Markov logic via dual decomposition,” in IEEE International Conference on Data Mining (ICDM), 2012.
 [21] D. Batra, “Dual decomposition implementation in GraphLab,” http://code.google.com/p/graphlabapi/source/browse/toolkits/graphical_models/dd_main.hpp, 2012.
 [22] S. Boyd, C. Cortes, C. Jiang, M. Mohri, A. Radovanovi, and J. Skaf, “Largescale distributed optimization for improving accuracy at the top,” in NIPS Workshop on Optimization for Machine Learning, 2012.
Comments
There are no comments yet.