Dynamic Averaging Load Balancing on Cycles

03/20/2020 ∙ by Dan Alistarh, et al. ∙ 0

We consider the following dynamic load-balancing process: given an underlying graph G with n nodes, in each step t≥ 0, one unit of load is created, and placed at a randomly chosen graph node. In the same step, the chosen node picks a random neighbor, and the two nodes balance their loads by averaging them. We are interested in the expected gap between the minimum and maximum loads at nodes as the process progresses, and its dependence on n and on the graph structure. Similar variants of the above graphical balanced allocation process have been studied by Peres, Talwar, and Wieder, and by Sauerwald and Sun for regular graphs. These authors left as open the question of characterizing the gap in the case of cycle graphs in the dynamic case, where weights are created during the algorithm's execution. For this case, the only known upper bound is of O( n log n ), following from a majorization argument due to Peres, Talwar, and Wieder, which analyzes a related graphical allocation process. In this paper, we provide an upper bound of O ( √(n)log n ) on the expected gap of the above process for cycles of length n. We introduce a new potential analysis technique, which enables us to bound the difference in load between k-hop neighbors on the cycle, for any k ≤ n / 2. We complement this with a "gap covering" argument, which bounds the maximum value of the gap by bounding its value across all possible subsets of a certain structure, and recursively bounding the gaps within each subset. We provide analytical and experimental evidence that our upper bound on the gap is tight up to a logarithmic factor.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

We consider balls-into-bins processes where a sequence of weights are placed into bins via some randomized procedure, with the goal of minimizing the load imbalance between the most loaded and the least loaded bin. This family of processes has been used to model several practical allocation problems, such as load-balancing [3, 7, 11], hashing [5], or even relaxed data structures [2, 1].

The classic formulation of this problem is known as the -choice process, in each step, a new weight is generated, and is placed in the least loaded of uniform random choices. If , then we have the classic uniform random choice scheme, whose properties are now fully understood. In particular, if we place unit weights into the bins, then it is known that the most loaded bin will have expected load, whereas if we have that the expected maximum load is . Seminal work by Azar, Broder, Karlin, and Upfal [3] showed that, if we place unit weights into bins by the -choice process with , then, surprisingly, the maximum load is reduced to . A technical tour-de-force by Berenbrink, Czumaj, Steger, and Vöcking  [4] extended this result to the “heavily-loaded” case where , showing that in this case the maximum load is

with failure probability at most

. An elegant alternative proof for a slightly weaker version of this result was later provided by Talwar and Wieder [13].

More recently, Peres, Talwar, and Wieder [10] analyzed the graphical version of this process, where the bins are the vertices of a graph, an edge is chosen at every step, and the weight is placed at the less loaded endpoint of the edge, breaking ties arbitrarily. (Notice that the classic -choice process corresponds to the case where the graph is a clique.) The authors focus on the evolution of the gap between the highest and lowest loaded bins, showing that, for -regular expander graphs, this gap is , with probability .

In the static case, where each node starts with an arbitrary initial load, and the endpoints average

their initial loads whenever the edge is chosen, the balancing process can be mapped to a Markov chain, and its convergence is well-understood in terms of the spectral gap of the underlying graph 

[12]. Sauerwald and Sun [12] considered this static case in the discrete setting, where the fixed initial load can only be divided to integer tokens upon each averaging step, for which they gave strong upper bounds for a wide range of graph families. By contrast, in this paper we consider the less complex continuous averaging case, where exact averaging of the weights is possible, but in the more challenging dynamic scenario, where weights arrive in each step rather than being initially allocated.

One question left open by the line of previous work concerns the evolution of the gap in the dynamic case on graphs of low expansion, such as cycles. In particular, for cycles, the only known upper bound on the expected gap in the dynamic case is of , following from [10], whereas the only lower bound is the immediate gap lower bound for the clique. Closing this gap for cycle graphs is known to be a challenging open problem [9]. As suggested in [10], to deal with the cycle case, there is a need for a new approach, which takes the structure of the load balancing graph into the account.


In this paper, we address this question for the case where averaging is performed on a cycle graph. We provide an upper bound on the gap of expected in the dynamic, heavily-loaded case, via a new potential argument. We complement this result with a lower bound of on the square of the gap, as well as additional experimental evidence suggesting that our upper bound is tight within a logarithmic factor. Our results extend to weighted input distributions.

Technical Argument

Our upper bound result is based on two main ideas. The first introduces a new parametrized hop-potential function, which measures the squared difference in load between any -hop neighbors on the graph, where is a fixed hop parameter. That is, if is our input graph, and is the load at node at time , then we define the -hop potential as:

The first technical step in the proof is to understand the expected (“steady-state”) value of the -hop potential. We show that, in expectation, the -hop potential has a regular recursive structure on regular graphs. While the expected values of -hop potentials cannot be computed precisely, we can isolate upper and lower bounds on their values for cycles. In particular, for the -hop potential on an -cycle, we prove the following bound:


In the second technical step, we shift gears, aiming to bound the maximum possible value of the gap between any two nodes, leveraging the fact that we understand the hop potential for any . We achieve this via a “gap covering” technique, which characterizes the maximum value of the gap across all possible subsets of a certain type.

More precisely, in the case of a cycle of length , for each node and hop count , we define the set family to be formed of nodes . (Since we are on a cycle, .) Then for any and , we will have


where is the maximal gap inside the set at time . Intuitively, this result allows us recursively characterize the gap value at various “resolutions” across the graph.

Finally, we notice that we can “cover” the gap across between any two nodes by carefully unwinding the recursion in the above inequality, considering all possible subsets of a well-chosen structure, and recursively bounding the gaps within each subset. (This step is particularly delicate in the case where is not a power of two, which we leave to the Appendix.) We obtain that


as claimed. The logarithmic slack is caused by the second term on the right-hand-side of (2). We note that this technique extends to the case where inserted items are weighted

, where the weights are coming from some distribution of bounded second moment.

Lower Bound

It is interesting to ask whether this upper bound is tight. To examine this question, we revisit the recursive structure of the -hop potential, which we used to obtain the lower bound in Equation 1. We can leverage this structure to obtain a lower bound on the expected -hop potential as well. Starting from this lower bound, we can turn the upper bound argument “inside out,” to obtain a linear lower bound on the expected squared gap:


This second moment bound strongly suggests that our above analysis is tight within logarithmic factors. We conjecture that the bound is also tight with regards to the expected gap, and examine this claim empirically in Section 5.

Extensions and Overview

The analysis template we described above is general, and could be extended to other graph families, such as regular expanders. In particular, we note that the recursive structure of the -hop potentials is preserved for such graphs. The main technical steps in analyzing a new graph family are to (1) identify the right upper bound on the -hop potential (the analogue of (1)); and (2) identify the right set family for the gap covering argument, and its recursive structure (the analogue of (2)). Obtaining tight bounds for these quantities is not straightforward, since they do not seem to be immediately linked to well-studied graph properties. Here, we focus on obtaining tight bounds on the gap for cycles, which is technically non-trivial, and leave the extensions for other graph families as future work. To substantiate our generality claim, we exhibit an application of our analysis technique to Harary graphs [6] in the Appendix.

We discuss the relation between our results and bounds for the graphical power-of-two process on a cycle [10] in Section 6.

Related Work

As we have already discussed broad background, we will mainly focus on the technical differences from previous work. As stated, we are the first to specifically consider the dynamic case for continuous averaging on cycles. In the static case with discrete averaging, the problem has been considered by Sauerwald and Sun [12]. However, their techniques would not apply in our case, since we consider that weights would be introduced dynamically, during the processes’ execution.

To our knowledge, the only non-trivial upper bound on the gap of the process we consider which would follow from previous work is of , by the potential analysis of [10]: they consider 2-choice load balancing, and one can re-do their potential analysis for (continuous) averaging load balancing, yielding the same bounds. However, as our bounds show, the resulting analysis is quite loose in the case of cycles, yielding an gap. This is a consequence of the majorization technique used, which links dynamic averaging on the cycle and a very weak form of averaging on the clique.

Our potential analysis is substantially different from that of [10], as they track a sum of exponential potentials across the entire graph. By contrast, our analysis tracks the squared load differences between -hop neighbors, establishing recurrences between these potentials. We notice that this is also different from the usual square potentials used for analyzing averaging load balancing, e.g. [8], which usually compare against the global mean, as opposed to pairwise potential differences. Our approach is also different from the classic analyses of e.g. [3], which perform probabilistic induction on the number of bins at a given load, assuming a clique.

Generally, our technique can be seen as performing the induction needed to bound the gap not on the bin loads, as is common in previous work, e.g. [3], but over the topology of the graph. This approach is in some sense natural, since we wish to obtain tight, topology-specific bounds, but we believe we are the first to propose and analyze it.

2 Averaging on the Cycle: Upper Bounding the Gap


We consider a cycle graph with nodes, such that each node is connected to its left and right neighbors, and . We consider a stochastic process following real time , in which, in each step, a weight is generated from a same distribution . We associate a real-valued load value with each node . In step , an edge is chosen uniformly at random, and the two endpoints nodes update their weights as follows:

We will assume that the second moment of the distribution is bounded. Formally, for each there exists such that . For simplicity, we will assume that weights are normalized by . This gives us that for every : .


be the vector of the bin weights after

balls have been thrown. First, we define the following potential functions:

Above, we assume that , for all . Notice that for every , we have that . We want to analyze what is the value of these functions in expectation after an additional ball is thrown, for a given load vector .

We start with :

Now, we proceed with calculating the expected value of , for :

Note that in the above calculations for and , for the terms which contain as linear multiplicative term disappear because we can assume that loads are normalized (this will not change our potentials) and we have:


If we remove conditioning on and express these equations for , we get:

Using the above equations we can prove the following:

Lemma 2.1.

For every and , we have that


Let be the vector of values of our potentials at time step and let , be the vector containing our desired upper bounds for each potential. That is: for each , we have that . An interesting and easily checkable thing about the vector is that


Next, consider the vector . Our goal is to show that for every step and coordinate , . we have that

and for , we have that

Hence we get the following equations(recall that ):


Next, using induction on , we show that for every


The base case holds trivially since . For the induction step, assume that . First, we have that

Additionally, we have that:

For , we have that

Next, observe that by our assumption:
. Finally, by using this observation we get that

This completes the proof of the theorem. ∎

3 Upper Bound on the Gap for

In this section we upper bound a gap in expectation for case. The proof for the general case is quite technical but not necessarily more interesting, and is provided in the section A in the Appendix.

We begin with some definitions. For a set , let

Also, let be (Notice that ). Our proof works as follows: for each and , we look at the vertices given by the sets and and try to characterise the gap after we merge those sets (Note that this will give us the gap for the set ). Using this result, we are able to show that is upper bounded by plus times maximum load difference between vertices at hop distance . Next, we use hop distance potential to upper bound maximum load between the vertices at hop distance . Using induction on , we are able to upper bound in terms of and . Notice that by our definitions, for each , ( contains only vertex ) and ( contains all vertices). Hence, what is left is to use the upper bounds for the hop distance potentials, which we derived in the previous section.

We start by proving the following useful lemma.

Lemma 3.1.

For any and , we have that


Fix vertex . Note that . Let and let . We consider several cases on the membership of nodes and , and bound the gap in each one:

Case 1. and . Then and we have that

Where we used the fact that both and belong to . This gives us that


Case 2. and . Then we have that:


Where we used and . Hence, we again get that


Case 3. and , is similar to Case 1.

Case 4. and , is similar to Case 2. ∎

Next, we upper bound the quantity

Lemma 3.2.

Notice that for any and , we have that , hence and this means that

Where we used a fact that sets are disjoint. ∎

Finally, using the two Lemmas above and Theorem 2.1 we can upper bound the expected gap at step :

Theorem 3.3.

For every , we have that


From Lemma 3.1 we have that

After dividing the above inequality by 2 and applying Lemma 3.2 we get that:

Notice that and we also have that

Hence, we get that

Next, we apply Jensen and Theorem 2.1: