Estimating the escape behavior of random walks has been an important performance indicator in fields such as biology [22, 20], epidemiology [14, 9], cosmology , computer science , and more [21, 27, 3, 2]. Maximizing the escape time plays a crucial role in containing the spread of diseases or computer viruses [14, 18], where the probability of an epidemic outbreak is closely related to properties of the contact network . In this work we identify a phenomenon related to the exact escape time of a heterogeneous random walk on the finite line. Specifically, we show that the escape time is always maximized by a unique structure of transition probabilities, also known as the “Pendulum Arrangement”.
The characteristics of escape times of random walks have been extensively studied under the names of first passage time, escape times, and hitting times. While analytical formulations of the escape time have been established [17, 1, 6], their analysis has been mostly based on mean-field theory, asymptotic characteristics, and approximations [17, 4, 1, 7, 25, 10, 13, 8]. Also related to our work, are studies on the speed of random walks in random environments [23, 24, 16, 19]. Specifically,  show that the speed is minimized asymptotically by equally spaced drifts on the line. In contrast, our work takes an exact, combinatorial view of the problem, revealing an intrinsic feature of the maximum escape time in the general setting of an arbitrary heterogeneous random walk.
We consider a heterogeneous random walk on a finite line 
. Given a vectorof transition probabilities, the process, as depicted in Fig. 1, starts at position , moves backward with probability (reflecting at ), forward with probability , and ends once it reaches position . Our goal is to rearrange the elements of the vector (corresponding to rearranging the transition probabilities of moving backward on the line), so as to maximize the expected escape time of the random walk, namely, the time to reach position for the first time. Conceptually, we wish to form a potential barrier under a fixed budget, but are unsure where to place the barrier on the line.
It is not clear a-priori whether the structure of this barrier has a closed form solution as it may depend on delicate relationships between the given probabilities. Intuitively, one might choose to arrange the transition probabilities in decreasing or increasing order. Here, an increasing order of the probabilities corresponds to forming a potential barrier toward the end of the line, reinforcing nodes in the vicinity of the termination node, whereas, a decreasing order corresponds to forming a barrier at the beginning of the line. Perhaps surprisingly, neither arrangement would maximize the escape time.
To obtain some intuition, consider an ascending order, where the highest probability is placed last. Notice that position is reached only after visiting position , i.e., the second to last position is always visited more than the last position. It is thus unreasonable to place the highest probability last, as this would only decrease the expected escape time because it will be used less often. A similar argument can be made for a descending order, by switching between the first two probabilities. We will make this intuition precise in our complete derivation.
Our main result shows that there is a unique optimal order of the transition probabilities that does not depend on their absolute, but rather their relative value, i.e., their sorted order. This also implies that changing the probabilities in a way that does not change their sorted order does not change the optimal arrangement. More specifically, we prove that the optimal order of the probabilities is such that they form a special pyramid-like shape, sometimes referred to as the pendulum arrangement (see Fig. 2), where the highest probability is placed in the middle.
Finally, we formulate a continuous optimization variant of the problem, where the transition probabilities are optimized under limited budget constraints. We show that our main result can greatly diminish the complexity of finding an optimal solution. We also provide numerical experiments that illustrate the potential gains of using the pendulum arrangement, and discuss possible alternative statistics, including the minimum escape time.
2 Problem Statement
A vector of transition probabilities defines a heterogeneous random walk on a finite line of states, as depicted in Fig. 1
. Formally, this process is defined by the following Markov chain. Letwhere is a random process that satisfies
We define the escape time as the arrival time of to the termination state given that it started at state , i.e.,
Our goal is to find the arrangement of the elements of that maximizes the expected escape time starting at state . Formally, let be the set of permutations on , i.e., is a bijective mapping of onto itself. A vector is a permutation of the elements of defined as . Our goal is to find a permutation such that
In what follows we will show that admits a unique solution that maps large transition values to the center, and small values to the edges of the line.
3 Main Result
This section states our main result, showing the optimal solution to Problem (P1) satisfies a unique symmetric arrangement, known as the pendulum arrangement, or its mirror. To that end, we define the mirror permutation , which reverses the vector it operates on.
Definition 1 (Mirror Permutation).
The mirror permutation is defined by .
Next, we define the pendulum arrangement.
Definition 2 (Pendulum Arrangement).
We say satisfies the pendulum arrangement if
We say is a pendulum arrangement of if such that and is a pendulum arrangement.
The pendulum arrangement has a special pyramid-like shape, as depicted in Fig. 2. Notice that traversing over its elements in descending order creates a pendulum-like motion hence explaining the name. Intuitively, the pendulum arrangement of a vector can be constructed by first sorting in decreasing order, and then placing the elements of the sorted array such that the largest element is in the middle, the next element to its left, the following element to its right, repeating this process until all elements have been placed in a pendulum-like ordering. This observation is made formal by the following lemma, which relates the pendulum arrangement to the sorted arrangement.
For let be the result of sorting the elements of in ascending order. Define
then is uniquely defined and satisfies .
The proof of the lemma is technical and deferred to Appendix B. We are now ready to state our main result.
Theorem 4 (Main Result).
maximizes the expected escape time, i.e., solves Problem (P1), if and only if it is ordered according to the pendulum arrangement, , or its mirror .
In other words, solving Problem (P1) reduces to finding a pendulum arrangement of the elements of , which is immediately obtained from their sorted order. Moreover, this solution is unique up to its mirror.
4 Proof of Main Result
The proof of Theorem 4 consists of two parts, as seen in Fig. 3. In this section we focus on the right part of Fig. 3, showing a closed form expression for the expected escape time , which reduces the problem to maximizing a sum over products. We then prove that the pendulum arrangement maximizes this sum of products, thus concluding the proof. This second part, which is used here as a tool, is the heart of the problem and we discuss and explain its main ideas in Section 5.
The following proposition derives a closed form expression for . Its proof uses a direct inductive claim and is provided in Appendix A.
Proposition 5 (Closed Form Expression).
We have that
One can immediately notice a symmetric property of in Eq. 1. Specifically, it is invariant to the mirror permutation, i.e.,
This in turn implies that the pendulum arrangement and its mirror both achieve the same value, and thus proving that one of them is optimal will suffice to conclude Theorem 4 (Main Result). Also note that this implies that ascending and descending orderings of the elements of achieve identical (yet sub-optimal) values. This fact is indicative of a symmetric characteristic of that foreshadows the underlying pendulum arrangement.
Focusing on the sum over products term in Eq. 1, we have the following theorem, which states that the pendulum arrangement is its unique maximizer.
Theorem 6 (Optimal Sum of Products).
For any we have that
As we will show next, combining Theorem 6 with Lemma 3 and Proposition 5 yields a straightforward proof for Theorem 4 (Main Result). The proof of Theorem 6 is the crux of this work and is outlined in the following section. Before diving into its details, we show how it can be used to prove Theorem 4 (Main Result).
Proof (of Theorem 4 (Main Result)).
Consider the expression for in Eq. 1. Denoting , where the equality is element-wise, we have that
Next, notice that and so using Theorem 6 we have that
Finally, notice that the function is strictly increasing in and thus the sorted order of and are the same. Since the pendulum arrangement only depends on this order (see Lemma 3), we get that and combining these arguments concludes the proof.
5 Sum Over Products
In this section we will focus on proving Theorem 6. We do so by considering each of the inner summations in Theorem 6, reducing the problem further to individually maximizing each of the inner sums of products (Theorem 8). We then move to define the improving permutation (Definition 9), a uniquely designed permutation that: (1) always improves the sum over products (Lemma 10); and (2) converges after at most applications to the pendulum (optimal) arrangement (Lemma 12). These results will finalize the proof of Theorem 6, thereby concluding the proof of Theorem 4 (Main Result). To that end, we begin by focusing on the following construct.
Definition 7 (Sum Over Products Value).
For any and define the value function of for window size as
The function is a sum over all products of adjacent tuples of length . For example, for and it can be explicitly written as Notice that the expression in Theorem 6 is in fact a summation of the sum over products value, , for various window sizes . Theorem 6 is thus an immediate corollary of the following, more general result.
Theorem 8 (General Sum Over Products).
For any we have that
5.1 Improving Permutation
The main tool for proving Theorem 8 is the following permutation.
Definition 9 (Improving Permutation).
For define the improving permutation of a vector by
We note that the vector , with respect to which is defined, is always the vector it permutes. While it is not denoted explicitly in , its identity will always be clear from context. The improving permutation, , compares elements across the symmetry axis , and switches their positions such that the larger element is to the right of the symmetry axis (see Fig. 4). Notice that this may result in up to exchanges. While this may seem overly complicated, it is easy to give counter examples where any exchange of two elements will decrease the outcome (see Remark 11). As its name suggests, applying to a vector increases its sum over products value, as shown by the following lemma. An exhaustive proof is provided Appendix C.
Lemma 10 (Improving Permutation).
For all and we have that
Moreover, if then there exists such that the inequality is strict.
Proof (sketch of Lemma 10).
We begin by denoting the product over a “window” of size starting at by , i.e., . With some algebra, we then show that
In Eq. 3 and were chosen such that if participates in then participates in . Since these are the only kind of switches makes, we conclude that Since puts the larger element in , i.e., in , we also have that with strict inequality if only some but not all of the elements were switched. Combining the last two claims, it is immediate to conclude that
This is equivalent to saying that elongating the longer side of a rectangle while maintaining its area fixed (by shortening the other side) increases its circumference. The above holds for the relevant indices and thus showing that concludes the proof. This is straightforward since essentially increases each of its terms individually.
When , it is always possible to find a so called “improving” permutation that only exchanges two elements; however, this is not the case for . To see this, take for example, the case of . Exhaustive search shows that this is the second to best ordering and thus any improvement must lead to one of the optimal orderings or its mirror . Notice that any such permutation must indeed exchange more than two elements. In other words, there exists an initialization vector for which no “simple” permutation (i.e., one which exchanges only two elements) could iteratively converge to the optimal ordering. This motivates the use of more elaborate improving permutations as proposed in Definition 9.
5.2 Pendulum Sort
Having established that () are always improving, we show that applying them consecutively converges to a pendulum arrangement. More specifically, the following lemma uses and (see Definitions 9 and 1) to construct such a sequence. An exhaustive proof of the lemma is provided Appendix D.
Lemma 12 (Pendulum Sort).
For all , we have that .
Proof (sketch of Lemma 12).
Recall that from Lemma 3 satisfies . We define as follows
and a simple telescoping argument yields that
We then show that for any , for all . Recalling Lemma 3 and choosing concludes the proof. To show that we first find explicit expressions for . These expressions are sorting procedures on the odd and even odd pairs of respectively. This means that applying them consecutively performs a sort of parallel bubble sort, which is depicted in Fig. 6. A simple analysis shows that this converges in steps and a more careful analysis gives the desired steps.
Proof (of Theorem 8).
First, recall that and so using Lemma 10 (Improving Permutation) recursively we get that
Taking and using Lemma 12 (Pendulum Sort) we then get that and since this holds for any permutation of , the first part of the proof is concluded. The uniqueness claim follows from the strict inequality condition of Lemma 10 (Improving Permutation). More concretely, let
and assume in contradiction that . However, from Lemma 12 (Pendulum Sort) we know that and thus one of the terms composing must change its input to something other than its mirror. The strict inequality condition of Lemma 10 (Improving Permutation) then implies that there exists such that contradicting the optimality of .
6 Discussion and Future Work
In this section we demonstrate a continuous extension to our main result, conduct numerical experiments on random environments that illustrate the significance of our findings, and discuss alternate statistics of the escape time.
6.1 Continuous Weight Optimization
We consider the following continuous optimization variant of the combinatorial problem (P1):
where is a set of budget constraints on the transition probabilities. The difficulty of (P2) strongly depends on the structure of the set . Theorem 4 (Main Result) implies that for (P1) is efficiently solvable. The following proposition readily follows from Theorem 4 (Main Result), and extends it to a slightly more general class of constraints. For let denote the extreme points of the convex hull of , and (see Definition 2).
For , if then .
In other words, if the pendulum arrangement is always an element of the extreme points of then the optimal solution to Problem (P2) is an extreme point of which is ordered according to the pendulum arrangement. The proof of Proposition 13 is provided in Appendix E and uses the fact that is convex in . This implies that there exists and thus applying Theorem 4 (Main Result) with the assumed structure of concludes the proof. This result allows us to greatly reduce the search for an optimal solution. Particularly, it may reduce this search to a small constant number of possible candidates, as shown by the following example.
Assume a linear budget constraint of the form
where . Trivially, whenever the optimal solution is given by the uniform vector . Yet, when , by Proposition 13, the optimal solution will be given by a pendulum arrangement over . This results in values of (up to a remainder term) placed in the center of the line. Concretely, with
6.2 Random Environments
Theorem 4 (Main Result) shows that the pendulum arrangement yields the maximum expected escape time. In this section we perform several numerical experiments to give a more quantitative grasp of the behavior of the expected escape time under different arrangements: maximal (pendulum), minimal, sorted, and random. The minimal arrangement is the one that yields minimal expected escape time, and is found using exhaustive search. The sorted arrangement refers to sorting the weights (transition probabilities) in ascending order. The random arrangement refers to a random (uniform) arrangement of the given weights. For small values of this can be calculated exactly by averaging over all possible arrangements. When this becomes computationally infeasible, we use Monte-Carlo methods to estimate this quantity.
Our first experiment compares the maximum, random, and minimum arrangements. To do so, we consider a random walk in a random environment setting on a line with nodes. We initialize the environment weights using a uniform distribution on and perform a Monte-Carlo simulation (only on the initialization) to evaluate the expected escape time of each arrangement. The results are depicted in Fig. 7
. Our choice of distribution keeps the expected value of the weights fixed while varying their variance. Unsurprisingly, the arrangement of the weights becomes more significant for higher variance weight initialization. Notice that the graph displays the logarithm of the escape time, and thus the increasing gaps between the arrangements imply a highly super-linear dependence on the variance.
Our second experiment examines the behavior of the escape time as a function of for the maximal, sorted and random arrangements (see Fig. 8). We observe two types of behaviors depending on the properties of the random environment. The first behavior occurs when all weights are smaller than , and yields a walk that is, in a sense, “strongly” transient, making the escape time grow slowly (linearly) in
regardless of the arrangement. While there is a significant gain in using the maximal (pendulum) and sorted arrangements, which perform similarly here, the overall behavior of the escape time does not change compared to a random arrangement. The second case reveals an interesting phase transition. It considers a case where the random arrangement is transient but some proportion of the weights are greater than. In this case the random arrangement behaves as in the first environment (up to small factors). However, starting at some , the maximal and sorted arrangements grow exponentially, with a significant gap between them. We have tried various environment parameters and this behavior seems to persist with the only change being the critical value of where the change in behavior occurs. We leave the formal investigation of this phenomenon to future work.
6.3 Alternate Statistics
In this work we focused on the maximization of the expected escape time. While maximizing the expected value is a highly accepted notion, one could also consider other criteria that, for example, consider some notion of risk. For instance, one might wish to find a permutation for which is maximized. Some classical examples include the Sharpe Ratio , and Mean-Variance criterion .
An alternative notion that is of separate interest is minimizing the expected escape time. This problem was studied in a simplified setting where weights are constrained to one of two values, showing that the asymptotic optimal order requires equal spacing between the larger weights [19, 12]. In Fig. 9 we depict three instantiations of general weight assignments for a line of nodes. Contrary to the maximal expected escape time, the minimal optimal permutation is value dependent, suggesting that understanding the structure of the minimal permutation is more involved. Extensive simulations lead us to the conjecture that “large” values are indeed spaced more or less evenly, but it remains unclear how to characterize this notion formally. We leave the topic of alternate statistics as an open question for future work.
In this work we conducted exact analysis of a newly discovered phenomenon of heterogeneous random walks. We showed that the maximum escape time is established when the transition probabilities relating to the slowdown drift of the process are ordered in a unique arrangement, known as the pendulum arrangement (see Fig. 2). Our result follows careful inspection of a sum over products combinatorial optimization problem, which may be of broader interest in fields out of the scope of this paper.
Finally, our work lays the foundations for Markov chain Design, through careful design of the topology and weights of Markov chains. This may enable the construction of networks that are insusceptible to cyber-attacks, resilient to the spread of infectious diseases, and control the flow of perilous processes (e.g., harmful ideas) on social networks and the web.
-  (2009) Abrupt convergence and escape behavior for birth and death chains. Journal of Statistical Physics 137 (4), pp. 595–623. Cited by: Appendix A, §1, §1.
-  (2005) Optimal search strategies for hidden targets. Physical review letters 94 (19), pp. 198101. Cited by: §1.
-  (2002) Critical points and transitions in an electric power transmission model for cascading failure blackouts. Chaos: An interdisciplinary journal of nonlinear science 12 (4), pp. 985–994. Cited by: §1.
-  (2007) First-passage times in complex scale-invariant media. Nature 450 (7166), pp. 77–80. Cited by: §1.
-  (1995) A survey on separability and generalized convexity or generalized monotonicity. In Recent Developments in Optimization, pp. 117–124. Cited by: Proof.
-  (2018) First-passage time distribution for random walks on complex networks using inverse laplace transform and mean-field approximation. arXiv preprint arXiv:1812.05598. Cited by: §1.
-  (2009) Biased random walks in complex networks: the role of local navigation rules. Physical Review E 80 (1), pp. 016107. Cited by: §1.
-  (2016) First passage time distribution in heterogeneity controlled kinetics: going beyond the mean first passage time. Scientific reports 6, pp. 20349. Cited by: §1.
-  (2004) Forecast and control of epidemics in a globalized world. Proceedings of the National Academy of Sciences 101 (42), pp. 15124–15129. Cited by: §1.
-  (2012) First passage time for random walks in heterogeneous networks. Physical review letters 109 (8), pp. 088701. Cited by: §1.
-  (2012) Network cosmology. Scientific reports 2, pp. 793. Cited by: §1.
-  (2013) On maximizing the speed of a random walk in fixed environments. Electronic Communications in Probability 18. Cited by: §6.3.
-  (2014) Estimating mean first passage time of biased random walks with short relaxation time on complex networks. PloS one 9 (4). Cited by: §1.
-  (2001) How viruses spread among computers and people. Science 292 (5520), pp. 1316–1317. Cited by: §1.
-  (2017) Generalized gambler’s ruin problem: explicit formulas via siegmund duality. Methodology and Computing in Applied Probability 19 (2), pp. 603–613. Cited by: §1.
-  (2004) Limit theorems for one-dimensional transient random walks in markov environments. In Annales de l’Institut Henri Poincare (B) Probability and Statistics, Vol. 40, pp. 635–659. Cited by: §1.
-  (2004) Random walks on complex networks. Physical review letters 92 (11), pp. 118701. Cited by: §1.
-  (2001) Epidemic spreading in scale-free networks. Physical review letters 86 (14), pp. 3200. Cited by: §1.
-  (2012) The need for speed: maximizing the speed of random walk in fixed environments. Electronic Journal of Probability 17. Cited by: §1, §6.3.
-  (2013) Distance matters: the impact of gene proximity in bacterial gene regulation. Physical review letters 110 (19), pp. 198101. Cited by: §1.
-  (1985) Diffusion-limited reactions. Elsevier. Cited by: §1.
-  (1916) Drei vortrage uber diffusion, brownsche bewegung und koagulation von kolloidteilchen. Zeitschrift fur Physik 17, pp. 557–585. Cited by: §1.
-  (1975) Random walks in a random environment. The annals of probability, pp. 1–31. Cited by: §1.
-  (2001) More randomness of environment does not always slow down a random walk. Journal of Theoretical Probability 14 (3), pp. 699–715. Cited by: §1.
-  (2012) Random walks and first-passage properties: trajectory analysis and search optimization. Ph.D. Thesis. Cited by: §1.
-  (2020) Sequential vaccination for containing epidemics. medRxiv. Cited by: §1.
-  (1988) Introduction to theoretical neurobiology: volume 2, nonlinear and stochastic theories. Vol. 8, Cambridge University Press. Cited by: §1.
Appendix: Missing Proofs
Appendix A Proof of Proposition 5
The proof follows standard induction analysis, (see e.g., Proposition 2 of ), and is provided here for completeness. For , due to the Markov property,
Rearranging the above yields
Denoting we get
Solving this equation by iteration yields
Furthermore we have that
Then, combining the above we get that
Finally Lemma 14 below shows how the final expression can be technically derived from the above, using simple algebraic manipulations.
It holds that
Recalling that denotes the vector of probabilities , we have that
where in the last two steps we use the definition of and the fact that
and notice that
Then, we have that
where the last step is proven by induction on . Substituting for completes the proof.
We show that
by induction on .
Base case: . We have that
Induction step. Assume Eq. 4 holds for some . We will show it holds for as well. Indeed,
In we used the induction step, in we used the fact that , and in reorganization of the summands.
Appendix B Proof of Lemma 3