Formally Verified Convergence of Policy-Rich DBF Routing Protocols

06/02/2021 ∙ by Matthew L. Daggitt, et al. ∙ 0

In this paper we present new general convergence results about the behaviour of Distributed Bellman-Ford (DBF) family of routing protocols, which includes distance-vector protocols (e.g. RIP) and path-vector protocols (e.g. BGP). First, we propose a new algebraic model for abstract routing problems which has fewer primitives than previous models and can represent more expressive policy languages. The new model is also the first to allow concurrent reasoning about distance-vector and path-vector protocols. Second, we explicitly demonstrate how DBF routing protocols are instances of a larger class of asynchronous iterative algorithms, for which there already exist powerful results about convergence. These results allow us to build upon conditions previously shown by Sobrinho to be sufficient and necessary for the convergence of path-vector protocols and generalise and strengthen them in various ways: we show that, with a minor modification, they also apply to distance-vector protocols; we prove they guarantee that the final routing solution reached is unique, thereby eliminating the possibility of anomalies such as BGP wedgies; we relax the model of asynchronous communication, showing that the results still hold if routing messages can be lost, reordered, and duplicated. Thirdly, our model and our accompanying theoretical results have been fully formalised in the Agda theorem prover. The resulting library is a powerful tool for quickly prototyping and formally verifying new policy languages. As an example, we formally verify the correctness of a policy language with many of the features of BGP including communities, conditional policy, path-inflation and route filtering.



There are no comments yet.


page 1

page 13

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

I-a What is policy-rich routing?

This paper proves new, general results about the convergence of asynchronous routing protocols in the Distributed Bellman-Ford (DBF) family, which includes distance-vector (RIP-like) and path-vector (BGP-like) protocols. In particular, the results apply to what we call policy-rich protocols, and so we begin by informally explaining this terminology.

Suppose that a router participating in a DBF computation has a best route to a destination . If it then receives a route for from an immediate neighbour it will first apply some policy associated with that neighbour and produce a candidate route . It will then compare with to determine which is best. Let us denote the outcome of this selection process as . A very simple example is the shortest-paths problem where is just an integer representing distance, for some weight , and .

If we dig deeply into the classical theory underlying best-path algorithms — both synchronous [1, 2], and asynchronous [3] — we find that it always assumes the following equation, or something equivalent, must hold:


This property is referred to as distributivity. The left-hand side of the equation can be interpreted as a decision made by a router sending routes and while the right-hand side is a decision made by the neighbour receiving those routes. Assuming the equality holds, the classical theory proves that routing protocols arrive at globally optimal routes — where every pair of routers use the best path between them.

By a policy-rich language we mean one in which distributivity does not hold. A clear example of how distributivity violations might arise in routing can be seen in the use of route maps which are functions (scripts) that take routes as input and return routes as output. For example, if and are route maps, then we might define another route map as:


where is a predicate on routes (such as “does this route contain the BGP community 17?”). To see how easily distributivity can be violated, suppose that:

Then the left-hand side of Equation 1 is:

while the right-hand side becomes:

For Eq 1 to hold we need , which may not be the case (indeed, if were always true, then there would be no point in defining !). Perhaps the most common example of such conditional policies is route filtering, where is equal to the invalid route.

A specialist schooled in the classical theory of path finding might be tempted to forbid the use of such “broken" policies when using the Bellman-Ford or Dijkstra algorithms. Yet, once again, practice has outstripped theory. The Border Gateway Protocol (BGP) [4] — a key component of the Internet’s infrastructure — is a policy-rich routing protocol.

We argue that today’s BGP is in fact broken since it is possible to write policies that can result in anomalous behaviour such as non-convergence [5, 6] and multiple stable states [7]. The latter are problematic as the extra stable states are nearly always unintended and often violate the intent of policy writers. Leaving an unintended stable state requires manual intervention and, in the worst case, a high degree of coordination between competing networks. This phenomenon is colloquially referred to as a BGP wedgie.

Hence a pertinent question is how to tame the policy language of BGP-like protocols to ensure good behaviour? Of course we could mandate that all protocols must conform to Equation 1. However, we would then not be able to implement typical inter-domain policies that are based on commercial relationships [8, 9]. In addition, something as simple as shortest-path routing with route filtering would be banned.

Gao & Rexford [10] showed that the simple commercial relationships described in [8, 9], if universally followed, are enough to guarantee convergence. However their model is BGP-specific and gives us no guidance on how policy-rich protocols should be constrained in general. Furthermore their conditions impose constraints on the topology of the network and therefore they require continuous verification as the topology changes.

Instead an alternative middle ground for generic policy-rich protocols has been achieved for both Dijkstra’s algorithm [11] and the DBF family [12]. Rather than insisting on distributivity, we require that for all routes and policies we have:


In other words, applying policy to a route cannot produce a route that is more preferred. Although Equation 3 is sufficient for Dijkstra’s Algorithm [11], it must be strengthened for DBF algorithms [12] to:


That is, applying policy to a route cannot produce a route that is more preferred and it cannot leave a route unchanged. We call policy languages that obey Equation 3 increasing and those that obey Equation 4 strictly increasing. Note that if policies and are strictly increasing, then the conditional policy defined in Equation 2 is also strictly increasing. In other words, a strictly increasing policy language remains strictly increasing when route maps are introduced.

However, without distributivity we can no longer achieve globally optimal routes and so we must be content with locally optimal routes [11] – each router obtains the best routes possible given the best routes revealed by its neighbours.

It is natural to ask if the strictly increasing condition is too strong and prohibits desirable policies? Sobrinho [12] shows that it is the weakest possible property that guarantees a path-vector protocol always converges. This implies that if the policy language is not strictly increasing than there exists a network in which the protocol diverges.

Nonetheless, as we discuss in Section V, the policy language of BGP is not strictly increasing, and there is no known way of making it so without removing the ability of AS’s to hide commercially sensitive information from each other. Therefore, the next natural question to ask is: given a non-strictly increasing policy language, which networks does the protocol converge over? Sobrinho shows that a network being free with respect to the policy language is a sufficient and necessary condition for a DBF protocol to converge over that network.

Subsequent to these foundational results, the algebraic approach has been extended to model other more complex methods of routing such as multi-path routing [13, 14], hierarchical routing [15], and quality of service routing [16].

I-B Our contributions

I-B1 Answers to big-picture questions

Given these prior results, the reader might be tempted to assume that the convergence of standard DBF routing protocols is a solved problem. However, this is far from the case. In this paper, we address the following open questions:

  1. [label=Q0)]

  2. Sobrinho’s proofs only guarantee that the protocols converge. Do they always converge to the same stable state?

    Answer: We prove that the answer is yes. From a theoretical perspective, this implies that the set of DBF protocols that converge and the set of DBF protocols that converge deterministically are the same. From a practical perspective, this implies that as long as a DBF protocol is guaranteed to converge then you don’t have to worry about problems like BGP wedgies.

  3. The proofs of Sobrinho and Gao-Rexford both assume TCP-like in-order, reliable communication between routers. Is this a fundamental requirement for convergence?

    Answer: We prove that the answer is no, by showing that the results still hold for a weaker, UDP-like model of communication in which route advertisements may be delayed, lost, reordered and duplicated. From a theoretical perspective, this provides strong evidence that the difficulty of solving a routing problem in a distributed environment is entirely independent of the communication model used. This is not the case in other distributed protocols, e.g. consensus protocols [17].

  4. Sobrinho reasons about path-vector protocols. Do there exist corresponding results for distance-vector protocols?

    Answer: we show that they do. In particular, that distance-vector protocols converge over strictly increasing algebras/free networks if the number of possible path-weights assigned to routes is finite. From a practical perspective this implies the feasibility of policy-rich distance-vector protocols, a relatively under-explored design space.

I-B2 Improvements to the model

In addition to the new high-level results above, we also make several important contributions to the underlying mathematical model for DBF protocols. A more technical discussion of these contributions can be found in Section II-F.

  1. [label=M0)]

  2. We propose a new algebraic structure for modelling policy languages which is both simpler and more-expressive than that of Sobrinho. In particular, it contains only 5 algebraic primitives instead of 8, and is capable of naturally modelling path-dependent policies such as route filtering and path-inflation.

  3. We identify that a DBF protocols are a single instance of a much broader class of asynchronous iterative algorithms, for which there exist prior models and convergence results. From a theoretical point of view, this allows us to cleanly abstract the underlying routing problem being solved from the distributed environment in which it is being solved. Practically, it greatly simplifies the proofs and allows us to prove asynchronous convergence purely by reasoning about the behaviour of the protocol in an entirely synchronous environment. In contrast, previous work [10, 12] has typically created their own custom model of message passing in an asynchronous environment and therefore have to directly reasoning about message timings and have avoided reasoning about unreliable communication.

  4. Thanks to this existing theory, we construct a mathematical model of an abstract DBF routing protocol that is the first to precisely describe the entire evolution of the protocol, including exact message timings, routers and links failing, routers and links being added, and routers being reset. As far as we aware, this is therefore the first fully provably correct executable model of an abstract DBF protocol. In contrast, previous proofs have focused on proving results about a snapshot of a single network configuration in which all routers, links and policies remain static.

I-B3 Formal verification

In the past decade there has been a strong trend towards the verification of infrastructure-related software in areas as diverse as operating systems [18], compilers [19, 20] and networking [21, 22].

In line with this trend we have fully formalised all the results in this paper in the Agda theorem prover [23]. As Agda is capable of expressing both proofs and programs, and as our model is executable, this constitutes the first ever fully formalised, executable model of a general DBF protocol as far as we are aware. The advantages of this formalisation effort are as follows:

  1. [label=V0)]

  2. It provides a much stronger guarantee of the correctness of our proofs. Reasoning about algorithms in distributed environments is notoriously tricky, and as Sobrinho himself says in [12], his proof of convergence is only “semi-formal”. Having a computer check our proofs is a valuable and powerful assurance as to their correctness.

  3. The generality of our verified proofs means that they also function as a library that can be used by others to rapidly prototype, explore the behaviour of and formally verify the correctness of novel policy languages. This is a significant step towards reusable and modular formal verification of routing protocols. The library is freely available [24], and we hope that it will prove useful to the community.

  4. To demonstrate the utility of our library, we construct a model of a path-vector routing protocol that contains many of the features of BGP including local preferences, conditional policy, path inflation and route filtering. We formally verify in Agda that the policy language is strictly increasing, and therefore no matter what policies are used the protocol will always converge deterministically to a single stable state given a sufficient period of stability. Such a protocol is therefore provably safe-by-design.

I-C Road map

In Section II, we use an algebraic approach inspired by that of Sobrinho [12] and existing literature on asynchronous iterative algorithms [25], to construct an executable model of an abstract, fully asynchronous, distributed DBF routing protocol. This approach provides a clear, implementation-independent specification of the family of protocols which we can reason about. In Section III we discuss how our novel use of an existing theory of asynchronous iterative algorithms does much of the heavy lifting, and finish by applying using existing results from this field to prove our new results about distance-vector and path-vector protocols. In Section IV we describe our Agda formalisation of these results, and in Section V we construct a safe-by-design algebra using the library.

I-D Previous versions

Previous versions of this work have appeared before in conference proceedings [26] and in the thesis of the first author [27]. The many improvements in this paper over these prior versions are discussed in Section VII.

Ii Model

In this section we construct a model of an arbitrary DBF routing protocol. For the reader’s convenience, a glossary of all the notation used in this and later sections can be found in Appendix A.

Ii-a Generality vs implementation details

In any mathematical model of a real-world system, there is a fundamental tension between capturing the most general form of the system and capturing every detail of the implementation. More implementation details can always be added to better model a particular system, at the cost of losing the ability to apply the work more generally.

In this paper we err on the side of generality. While we acknowledge that BGP is the dominant policy-rich path-vector protocol, the aim of our work is not only to inform how BGP policies may be constrained to provide better guarantees, but also to guide the design of future policy-rich protocols.

For example we do not explicitly model the hard-state nature of BGP. Instead we use a very general model of asynchronous computation which we believe can easily capture a hard-state protocol (this intuition is not fully formalised in this work). Had we explicitly modelled a hard-state protocol, it would prevent us from applying our work to soft-state protocols as well.

Ii-B Paths

Prior to anything else, it is important to discuss exactly what is meant by a path in the context of a distributed, asynchronous routing protocol. Traditionally paths are defined with respect to some graph . However because a network’s topology changes over time, the path along which an entry in a router’s routing table was generated may not correspond to a valid path in the current topology. Furthermore, and perhaps counter-intuitively, such a path may be formed from an amalgamation of paths from previous topologies, and therefore may not even have existed in any single past topology. Consequently, out of necessity, our notion of a path cannot be tied to the network’s topology.

Therefore, we begin by defining as the set of all valid router identifiers. An edge is an arbitrary pair of router identifiers, and therefore the set of edges is . A path is a defined to be a sequence of contiguous edges. A cycle is a path which finishes at the same router it started at. A simple path is a path that doesn’t contain any cycles. For our purposes, we also consider an additional simple path , which represents the absence of a path. We refer to the set of simple paths as .

Ii-C Routing algebras

In order to model all possible DBF protocols rather than one particular protocol, we first abstract over the routing problem being solved by representing it as a routing algebra.

Definition 1 (Routing algebra).

A routing algebra is a tuple where:

  • is the set of path-weights. A path-weight is the data that the routing protocol associates with each candidate path. For example in a shortest-path protocol the path-weights would be the lengths of the paths, e.g. .

  • is the choice operator, which given two path-weights returns the preferred path-weight. In the introduction, this was informally referred to as “best”. In a shortest-path protocol this would be the operator.

  • is a map from an edge to the set of policy functions that may be used on that edge. Each policy is a function , which when given a path-weight returns a new path-weight formed by extending the original path by the edge . For notational convenience we define the complete set of policies as .

  • is the path-weight of the trivial path from any router to itself. In a shortest-path protocol this is 0.

  • is the invalid path-weight, representing the absence of a path.

which we assume to always have the following minimal properties:

  1. [label=R0]

  2. is associative - the order doesn’t matter when making a sequence of choices between path-weights.

  3. is commutative - the order doesn’t matter when choosing between two path-weights.

  4. is selective - choosing between two path-weights always returns one of the two path-weights.

  5. is an annihilator for - the trivial path-weight is more desirable than any other path-weight.

  6. is an identity for - any path-weight is preferable to the invalid path-weight.

  7. is a fixed point for all policy functions - the extension of the invalid path-weight is also the invalid path-weight.

Table I contains examples of simple routing algebras representing a variety of common routing problems. A much more complex routing algebra that is based off of BGP is discussed in detail in Section V.

shortest paths
longest paths
widest paths
most reliable paths
TABLE I: Examples of some simple routing algebras where for arbitrary operator
e.g. .

Path-vector protocol are a subclass of DBF protocols which track the path along which a path-weight was generated. Routers then use this information to eliminate any path-weights that were generated via a cycle. It is well known that path-vector protocols have better convergence behaviour. To allow us to reason about such protocols we now define a subclass of routing algebras that can be used to model path-vector-specific behaviour.

Definition 2 (Path algebra).

A path algebra is a routing algebra equipped with an additional function that obeys the following properties:

  1. [label=P0]

  2. A path-weight has the invalid path iff it is the invalid path-weight.

  3. The path of the trivial path-weight is the empty path.

  4. Applying an edge’s policy function to a path-weight simply extends the path-weight’s path by that edge, unless it does not result in a non-simple path in which case it becomes the invalid path.

The function allows us to extract the path along which the path-weight was generated without revealing how the path is stored internally inside the path-weight. The advantages of this abstraction are discussed at the end of this section. Assumptions P1 – P3 ensure that the algebra’s implementation is tracking the paths correctly. In P3 the first case ensures that path-weights generated along paths with cycles in them are eliminated. The second case ensures that invalid routing announcements are also eliminated.

Note that assumption P3 is the primary reason that we require that each edge comes with its own set of policy functions in Definition 1. If instead we had only required the existence of a single set of policy functions (i.e. ) then we would be unable to express P3.

Ii-D Synchronous DBF protocol

Now that we can model what problem our abstract DBF protocol is solving, we move on to modelling how it solves it. We start by defining a synchronous version of the protocol where update messages between routers are exchanged instantaneously and in parallel. Let be the finite set of routers that will ever participate in the protocol and let . We represent an instantaneous snapshot of the network topology by an adjacency matrix  where is the current policy function on the link from router to router . Note that in practice the policy function may be the composition of the export policy of router and the import policy of router . The absence of a link from to is represented by the always-invalid policy  which is defined as .

The global state of the synchronous protocol can be entirely represented by the product of the state of each router’s routing table. We therefore represent an individual state by a matrix where row is router ’s routing table and so the element  is the path-weight of ’s best current route to router . We will use to denote the set of routing table states (i.e. the set of vectors over of length ), and to denote the set of global states (i.e. the set of matrices over ).

We define the sum of two matrices and as:

and the application of to as:

In one synchronous round of a distributed Bellman-Ford computation every router in the network propagates its routing table its neighbours who then update their own tables accordingly. We model this operation as :

where is the identity matrix:

The nature of the underlying computation becomes clearer when expanding out the definition of a single element of :


Router ’s new route to  is the best choice out of the extensions of the routes offered to it by each of its neighbours .

We can now model the evolution of the synchronous version of protocol, , as repeated applications of .

Definition 3 (Synchronous state function).

Given a network topology and an initial state , the state of the synchronous DBF protocol at time is defined as:

Ii-E Asynchronous DBF protocol

Although we have defined a synchronous model above, in reality DBF protocols are distributed and asynchronous. Update messages are dispatched at non-deterministic intervals and may be delayed, reordered, duplicated or even lost along the way. New routers and links may be added to the network, existing routers and links may fail, and routers may be rebooted and reset to their original state. Reasoning about the outcomes of such unpredictable events is known to be extremely challenging.

One advantage of first defining the synchronous model is that it highlights that the underlying algorithm is inherently iterative. Consequently, it is easy to recognise that a DBF protocol is simply a special instance of the family of dynamic asynchronous iterative algorithms. We can therefore use existing work [28, 25] that constructs mathematical model of such iterations. We now describe a specialisation of the model presented in [25] to DBF protocols.

The model assumes a discrete and linear notion of time, 

, denoting the times of events of interest in the network (a routing table updating, a router failing etc.). Layered on top of time is the notion of epochs

, which represent contiguous periods of time during which the network topology remains unchanged, i.e. no links or routers fail or are added and no policies are changed. Given this, we can now model a dynamic network, , as a function from epochs to adjacency matrices where is the topology of the network during epoch .
























Messages reordered

Message lost

Message duplicated
Fig. 1: Behaviour of the data flow function . Messages from router to router may be reordered, lost or even duplicated. The only constraint is that every message must arrive after it was sent. Reproduced from [25].

The non-deterministic evolution of the protocol is described by a schedule.

Definition 4 (Schedule).

(Definition 9 in [25]) A schedule is a tuple of functions where:

  • is the activation function, where is the set of routers which update their routing table at time .

  • is the data flow function, where is the time at which the route advertisements used by router at time was sent by router .

  • is the epoch function, where is the epoch at time .

  • is the participants function, where  is the subset of routers currently participating in the protocol during epoch e.

such that:

  1. [label=S0]

  2. information only travels forward in time

  3. epochs increase monotonically

This is a very weak model of asynchronous communication. As shown in Figure 1, nothing forbids the data flow function  from delaying, losing, reordering or duplicating messages. It also distinguishes between a router being a member of the network but not currently participating in the protocol. This allows us to model messages that continue to arrive from the previous epoch, e.g. router can receive outdated route advertisements from router even after router has ceased to participate.

However, it does mean that we need to distinguish between the topology of network and the topology as seen by the active participants. In particular we need to enforce that information only flows between two participating routers.

Definition 5 (Participating topology).

Given a network , an epoch and a set of participants , we define to be the participating topology:

Given a schedule, for notational convenience we also define , the set of participants at time , and , the current participating topology at time , as:

Using this machinery it is now possible to define the full asynchronous state function, , in terms of the original iteration function .

Definition 6 (Asynchronous state function).

(Definition 10 from [25] with , ) Given a dynamic network , an initial state , and a schedule , the state of the asynchronous DBF protocol at time is defined as:

where .

This models the evolution of the protocol as follows. If a router is not currently participating then it adopts its non-participating state . If a router has just begun to participate, either because or because it was not participating at the previous time step, it adopts the initial state . If the router is a continuing participant and is currently inactive then its state remains unchanged from that at the previous time step, . Otherwise, if it is currently active, it updates its state in accordance with router ’s local view, of the global state as currently advertised by its neighbours. More concretely,  is the last route advertisement for router  that router  received from its neighbour .

Note that this is a straightforward generalisation of the synchronous model as we can immediately recover from by setting , , and i.e. there is a single epoch in which at each time step, every router is both participating and active and all messages only take a single time step to propagate.

Ii-F Novel contributions

We now recap the main advantages of this new model over models proposed in previous work.

Simpler, more general algebraic structure

The algebraic structure we propose in Definition 1 is both more general and simpler than that proposed by Sobrinho [12], which in turn was inspired by the classical semiring approach [1, 29]. The main difference is in how we model path-weight extension. In semiring algebras it is modelled by a second binary operator . However this is incapable of modelling modern routing policies such as those described in Sections I & V. Sobrinho generalised this binary operator by introducing labels , signatures , an operator and a function mapping signatures to path-weights. However, by modelling extension policies as a set of functions, , and as demonstrated in Section V, we retain the ability to model modern routing polices while reducing the size of the algebra from 8 primitives to 5 primitives.

Paths as part of the algebra

The second way in which the algebra is more general is that, by assigning each edge its own set of extension functions, we model the tracking of paths in path-vector protocols inside of the algebra itself. In contrast, Sobrinho [12] models the paths as part of the algorithm, i.e. in the equivalent definition of . This means that the paths along which the path-weights were generated are not provided as inputs to his  and  operators. Therefore they cannot alter or make decisions based on those paths. Consequently, unlike ours, Sobrinho’s approach cannot model policies such as route filtering or path inflation in BGP.

Abstraction of paths

A third advantage of our new Definition 2 of a path algebra is that it only requires that the function can extract a simple path from a path-weight, but does not force the protocol to reveal how paths are actually stored internally. This abstraction results in proofs that are far more robust to variations in how paths are stored internally by the protocol. For example, in Section V we show that our algebras can model BGP-like path inflation by simply adjusting the function to strip out the inflated ASs, without having to adjust the proofs of convergence. In contrast, the model of path-vector protocols in Sobrinho [12] uses a concrete definition of paths, and therefore is incapable of modelling path inflation without altering all the proof of convergence.

Use of existing asynchronous theory

The additional complexity introduced by the more expressive dynamic asynchronous model , such as routers failing and rebooting, at first glance appears daunting. However, as we will discuss further in Section III, our use of the existing theory of dynamic iterative algorithms means that we have access to a range of powerful theorems that, impressively, are capable of totally abstracting away the asynchronicity of the system.

Executable model of the protocol

Sobrinho [12] only provides an abstract temporal model of individual events that occur during the operation of the protocol. In contrast, as far as we are aware, ours is the first fully formal and executable model of the full dynamic behaviour of an abstract DBF protocols in which routers and links can fail, join and reboot. This means that our model can be run and its behaviour observed, and that our proofs can be written in a style amenable to the formal verification discussed in Section IV.

Iii Convergence results

Iii-a Some useful concepts

Initially, we define some concepts that, although not required for constructing the model in Section II, are used in stating and reasoning about its convergence behaviour.

Iii-A1 Ordering over path weights

Given any routing algebra , the choice operator induces a preference order over path-weights as follows:

Definition 7 (Preference relation over path-weights ).

A path-weight  is preferred to path-weight , written , if is chosen over :

As is associative, commutative and selective, we have that is a total order and that for all path-weights we have .

Iii-A2 Assignments

In our model which router is using which path-weight is implicitly defined by the path-weight’s location in the routing state (i.e. given a routing state then the path-weight is being used by router ). Nonetheless a more explicit notion of which router is using which path-weight is crucial for stating sufficient and necessary conditions for the convergence of the model over a given network. We shall refer to such a concept as an assignment.

Definition 8 (Assignment).

An assignment is a pair of a router and a path-weight . The set of assignments .

Note that an assignment is a generalisation of the notion of a couplet defined by Sobrinho  [12]. Whereas a couplet stores the entire path along which the path-weight was generated, an assignment only stores the first router in the path. This generalisation is necessary as we seek to reason about distance-vector as well as path-vector protocols, and in the former case we do not have access to the full path.

Several binary relations may be defined over the set of assignments. The first is the lifting of the preference order over path-weights to assignments.

Definition 9 (Preference relation over assignments).

An assignment  is preferred to assignment , written , if they have the same router and the path-weight of is preferred to the path-weight of :

Note that unlike which is a total order over path-weights, is only a partial order over assignments, as it can only relate assignments to the same router. Intuitively, this makes sense as routers in a DBF protocol only compare paths that begin at themselves.

Definition 10 (Extension relation).

An assignment extends assignment in the current topology , written , if the extension of the non-invalid path-weight of across the relevant link is equal to the path-weight of :

In the context of a path vector protocol, the axioms of a path algebra are sufficient to guarantee that if then .

Definition 11 (Threatens relation).

An assignment threatens assignment in the current topology , written , if the extension of the non-invalid path-weight of across the relevant link is preferred to the path-weight of :

Intuitively means that after the next update router may switch away from and instead use a path-weight that is equal to or preferred to . It is easy to see that if then .

Iii-A3 Non-participating nodes

As seen in Definition 5, the constraint that non-participating routers do not advertise routes to or receive routes from other routers is enforced at the level of the adjacency matrix and therefore the iteration function . Consequently, after a single iteration the state of any non-participating router’s routing table will always be the corresponding entry of the identity matrix, i.e. the trivial route to itself and invalid routes to all other routers in the network. In [25] such states are said to be accordant with the current set of participants.

Definition 12 (Accordant states).

(Definition 15 from [25] with ) The set of routing states that are accordant with set of participants , , is defined as:

Iii-B Definition of convergence

Before proving convergence results about our model, we must first establish what does convergence even mean for a dynamic, asynchronous, iterative algorithm such as a DBF protocol? In this section, will build up a suitable definition by discussing in turn the complications introduced by each of these adjectives.


The definition of convergence for a synchronous iterative algorithm is relatively simple. After a finite number of iterations, the algorithm must reach a stable state such that further iterations result in no further change. In terms of the synchronous model defined in Section II-D, this would translate to:

The resulting stable state is a fixed point for , as:

From the perspective of a routing protocol, being a fixed point is equivalent to the statement that in state no router can improve its selected path-weights by unilaterally choosing to switch. Such a state can therefore be viewed as a locally optimal solution to the optimisation problem of finding the best consistent set of path choices in the current network topology. As will be discussed further in Section III-D, whether such a state is necessarily globally optimal depends on the properties of the routing algebra being used.


At first glance, it appears relatively easy to adapt the above definition to a basic asynchronous model with only a single epoch: simply require that for every schedule the asynchronous iteration reaches a stable state. Unfortunately due to the asynchronous and distributed nature of the computation, and as discussed in Section II-E, our definition of a schedule does not assume reliable communication between routers. Consequently, in an arbitrary schedule a router may never activate or a pair of routers may never succeed in communicating. In such a schedule, it is clearly impossible for the protocol to converge to the desired stable state. Therefore, we need to restrict ourselves to only requiring that the protocol converges over reasonable schedules.

As long established in the asynchronous iterations literature [28], the right definition of “reasonable" is that the schedule contains pseudocycles. A pseudocycle is a period of time within an epoch such that for every participating router: all messages in flight to that router at the start of the pseudocycle are either dropped or arrive and, after this has occurred, the router performs at least one update. A formal definition of this concept is provided in Definition 13 in [25]. Informally, a pseudocycle can be thought of the asynchronous counterpart of a single synchronous iteration. Lemma 8 in [25] makes this intuition formal by proving that during a pseudocycle, the asynchronous model makes at least as much progress towards the stable state as the synchronous model would with a single synchronous iteration.

Another aspect that we must consider when moving from the synchronous to the asynchronous model is that of non-determinism. As discussed in Section I-A, BGP suffers from non-deterministic convergence where the protocol may end up in multiple stable states, and which stable state is reached depends on the schedule and the initial state, rather than just the topology. This problem is known colloquially as a BGP wedgie [7]. Our definition of convergence should therefore require it to be deterministic, i.e. in each epoch only one stable state may be reached and that it is dependent only on the topology and the set of participants.


Finally, we now consider the full dynamic model with multiple epochs. In this scenario the network topology may change so frequently (i.e. the epochs may be so short), that the protocol never gets sufficient time to converge to a solution. Therefore we would like our definition of convergence to only require that the protocol will converge if given a sufficient period of stability. It is suggested in [25] that dynamic iterations should therefore be referred to as convergent rather than saying that they converge.

The final issue is that the protocol may only be convergent over some, rather than all, epochs and sets of participants. As shown by Sobrinho [12] and discussed further in Section III-D, DBF protocols only converge over network topologies that are free. Therefore the definition of convergent must be granular enough to express that the protocol is only convergent over a subset of epochs and sets of participants. A pair containing an epoch and a set of participants is referred to as a configuration. The set of all configurations is therefore defined as:

Taking all the above points into account, we therefore arrive at the following definition of what it means for our protocol to be convergent:

Definition 13 (Convergent).

(Definition 14 from [25]) The DBF protocol is convergent over a set of configurations iff:

  1. for every configuration there exists a fixed point for and a number of iterations .

  2. for every initial state , schedule and time  then if and the time period  contains pseudocycles then for every time such that and then

The essence of this definition is that for every valid configuration there exists a fixed point and a number of pseudocycles such that if, at an arbitrary point in time the configuration of the current epoch is in the set of valid configurations and the schedule contains that number of pseudocycles, then after those pseudocycles the protocol will have reach the fixed point and will remain in that fixed point until the end of the current epoch.

Iii-C An existing convergence theorem

One of the most important advantages to using an established model of asynchronous iterations in Section II-E is that there already exist theorems describing sufficient conditions of such an iteration being convergent. The most powerful (and perhaps initially surprising) feature of these results is that the sufficient conditions only involve properties of the functions being iterated, , rather than the full asynchronous iteration, . Consequently one can use them to prove that is convergent without ever having to directly reason about the unreliable, asynchronous and dynamic nature of the underlying network, e.g. messages being lost or reordered, routers and links failing or being added.

In this paper, we will use the AMCO conditions from [25]. For a broader survey of the range of alternative sufficient conditions available see [28].

Definition 14 (AMCO, Definition 23 in [25]).

The functions are a set of asychronously metrically contracting operators (AMCO) over a set of configurations  if for every epoch  and set of participants  such that then for all there exists a distance function such that:

  1. [label=D0]

  2. is indiscernable

  3. is bounded

and if then:

  1. [resume, label=D0]

  2. is strictly contracting on orbits over

  3. is strictly contracting on fixed points over :

  4. enforces accordancy:

The properties required of an AMCO may seem complicated at first glance but each have clear interpretations. Firstly for every configuration in there must exist a function which measures the distance between two routing table states (although it need not satisfy the formal definition of a metric). D1 requires that the distance between two states is zero iff the states are equal, and D2 that there exists a maximum distance. Distances over routing tables, , are then naturally lifted to distances over routing states by calculating the maximum distance between the participating routing tables. D3 requires that the distance between the states after consecutive iterations strictly decreases, and D4 that if there exists a stable state then the distance to it strictly decreases after each iteration. Finally, D5 requires that applying the operator to an accordant state must always result in another accordant state.

Theorem 1.

If is a dynamic AMCO over a set of configurations then is convergent over .


See Theorem 5 in [25] and the Agda proof [24]. ∎

Informally a synchronous version of the proof runs as follows: D3 require that each synchronous iteration moves the current state an increasingly smaller distance. As the distances are all bounded natural numbers by D2 then the distance must eventually reach zero, which by D1 means that one has reached a stable state. D4 then guarantees that this stable state is unique. With some non-trivial work, it is possible to lift this synchronous argument to an asynchronous argument because the distance function over routing states is decomposable into individual distance functions over routing tables.

The key point to take away is that by using this theorem in our later proofs, we reduce the problem of proving our model of the DBF protocol, , is convergent to merely constructing a suitable distance function which interacts with in the right way.

Iii-D Types of routing algebras

In this section, we adapt the hierarchy of necessary and sufficient conditions for convergence established by Sobrinho [12] to our new simpler algebraic model111Our terminology differs slightly from that of Sobrinho as we use the traditional algebraic/order theoretic names for these conditions. In particular we use distributive instead of isotonic and increasing instead of monotonic. This is because the order theoretic notion of monotonicity, , is much more closely related to the notion of distributivity, , than increasingness, .. The key insight of Sobrinho is that each of the routing algebra axioms in Definition 1 only reference either choice or extension, and that is the relationship between them that determines the convergence behaviour of the protocol.

As discussed in Section I-A, the gold-standard are distributive algebras:

Definition 15 (Distributive algebra).

A routing algebra is distributive if:

As was proven many times in the classic algebraic routing literature and, again by Sobrinho, then if the algebra is distributive then any stable state will necessarily be a globally optimal state in which every pair of routers is assigned the optimal path between them.

The next most desirable property for the algebra to have is to be increasing/strictly increasing:

Definition 16 (Increasing algebra).

A routing algebra is increasing if:

Definition 17 (Strictly increasing algebra).

A routing algebra is strictly increasing if:

Sobrinho proved that the algebra being strictly increasing is a necessary and sufficient condition for a path-vector protocol to converge to some stable state in a given epoch. However, if the algebra is not distributive then that stable state will only be locally optimal rather than globally optimal, and some pairs of routers will end up using suboptimal paths.

Finally there is the freeness property over a particular network topology:

Definition 18 (Free network topology).

A network topology is free with respect to the routing algebra if there does not exist a cycle and a set of path-weights such that:

where is calculated mod .

Intuitively a network is free if there does not exist a cycle where each router’s current assignment can threaten the assignment of the previous router in the cycle. Sobrinho proved that the network topology being free with respect to the underlying algebra is a necessary and sufficient condition for a path-vector protocol to converge over that network, and that in the shortest-paths algebra this is equivalent to there being no negative weight cycles. Let be the set of configurations in which the participating topology is free, i.e.

We now prove a couple of small lemmas showing some of the relationships between these properties.

Lemma 1.

If a routing algebra is strictly increasing then every network topology is free.


See Theorem 2 & subsequent discussion in [12] or our Agda proof [24]. ∎

Lemma 2.

If a path algebra is increasing then it is also strictly increasing.


Consider arbitrary and and such that . We already have that as the algebra is increasing so it remains to show that .

Suppose that . We then know that as . Likewise we know that as otherwise by Assumption 1 we would have . Therefore, by Assumption 3, we have that which is a contradiction. Therefore and we have the required result. ∎

Iii-E DBF convergence results

We now proceed to state our main convergence results, the proofs of which may be found in the appendices. We start by considering distance-vector protocols.

Theorem 2.

If is a routing algebra and is finite then is convergent over .


See Appendix B. ∎

Theorem 2 says that for any distance-vector protocol with a finite set of path-weights then for any epoch and set of participating routers such that the network topology is free and given a sufficient period of stability in which the participating routers continue to activate and communicate, the protocol will always converge to the same stable state no matter the exact timing of messages and even in the presence of unreliable communication.

A more useful result for strictly increasing, finite algebras is then immediately obtainable which shows that such a distance-vector protocol is always convergent.

Theorem 3.

If is a strictly increasing routing algebra and is finite then is always convergent.


As the algebra is strictly increasing then by Lemma 1 every configuration is free, and therefore we have the required result immediately by Theorem 2. ∎

In practice the finiteness condition is restrictive and excludes many routing algebras of interest. For example even the shortest-path algebra uses the infinite set as path weights. However recall that Theorems 2 & 3 guarantee that, given a sufficient period of stability, the protocol will reconverge even in the presence of messages from previous epochs. On the other hand shortest-path distance-vector protocols experience count-to-infinity problems when the state at the start of the epoch contains path-weights generated along paths that do not exist in the current topology. Finiteness is a sufficient condition to reduce count-to-infinity problems to merely count-to-convergence in distance-vector protocols. In the real world this is reflected in the design of RIP which artificially imposes a maximum hop count to ensure that is finite.

A common approach to avoid the finiteness requirement (and count-to-convergence issues) is that of path-vector protocols which track the paths along which the path-weights are generated. Path-weights are then removed if their path contains a cycle.

Theorem 4.

Given a path algebra then is convergent over .


See Appendix C. ∎

Again a more useful version of this proof for strictly increasing path algebras is immediately obtainable.

Theorem 5.

Given an increasing path algebra then is always convergent.


As the path algebra is increasing then by Lemma 2 it is also strictly increasing, and therefore by Lemma 1 every configuration is free. Hence we have the required result immediately by Theorem 4. ∎

Iii-F Novel contributions

We now recap the main contributions of these proofs over that of previous work.

Deterministic convergence

Sobrinho’s proofs only guarantee that the protocol will converge, and nothing in his proof forbids the stable state reached depending on the state at the start of the epoch or the ordering of messages between routers during the epoch. In contrast our proofs guarantee that the protocol always converges to the same stable state, no matter the initial state or what the ordering of messages is. This therefore eliminates the possibility of phenomena such as BGP wedgies.

Unreliable communication

The definition of the data-flow function, , in Definition 4, Section II-E means that we only assume unreliable communication between routers. This is in contrast to previous proofs of convergence, for both general DBF protocols [12] and BGP in particular [10], that assume reliable communication. This means that our model and resulting proofs in later sections in theory apply to protocols built on-top of unreliable transport protocols such as UDP [30].

Extension to distance vector protocols

Theorems 2 & 3 apply to distance vector protocols, whereas Sobrinho’s proofs only applied to path-vector protocols. In particular, this shows that convergence could still be guaranteed if complex conditional policies were added to distance-vector protocols like RIP.

Iv Formalisation in Agda

We have formalised every mathematical result in this paper, down to the most trivial lemma, in the Agda theorem prover. This includes the results about asynchronous iterative algorithms from [25]. As our proofs have been checked by a computer, we are far more confident in their correctness than usual. Even where we omit details or use standard informal mathematical reasoning to improve readability (e.g. Lemma 13 in Appendix C), the proofs are backed by mathematical arguments which are guaranteed to be fully rigorous.

Furthermore the act of formalisation itself was invaluable in creating and shaping these proofs. For example the unstructured way in which we initially laid out the pen-and-paper proof of Lemma 12 in Appendix C led us to overlook Case 2.3. Only when formalising the result, did we notice that this case remained unproven. This in turn led us to presenting the proof in the much cleaner structure displayed in this paper.

The library of proofs is freely available [24] and is highly modular. To prototype a new policy language users need only define it as an algebra and prove that the algebra obeys the strictly increasing conditions in order to guarantee the resulting protocol is always convergent. Furthermore the protocol is then executable so that its dynamic behaviour can be observed for a given network and schedule. In Section V we give an outline of how to define such an algebra in Agda. We hope that the library’s extensible nature means that it will be of use to the community when designing new policy languages.

V A safe-by-design algebra

We now present an example of how one could use our Agda library to develop a safe-by-design routing protocol. In particular, we explore the construction of a path-vector algebra that contains many of the features of BGP such as local preferences, community values [31] and conditional policies. These policies can perform operations such as path-filtering, path-inflation, modifying local preferences and communities. The conditions themselves are implemented using a simple language of predicates that includes the ability to inspect communities. The algebra is a generalisation of the Stratified Shortest Paths algebra [32].

It differs from the algebra underlying today’s BGP in two crucial ways. Firstly, the implementation of the MED attribute violates the assumption that is associative [33]. Secondly, BGP allows ASs to hide their local preferences and set them to arbitrary values upon importing routes from other ASs. This violates the assumption that the algebra is increasing.

Our algebra avoids these two issues by a) ignoring MED and b) having policies that only allow local preference to increase. For convenience, and to mirror the increasing terminology,

Note that we present our algebra only to give a practical example of how our theory can be used, and to show that “most" of the features of BGP are inherently safe. We are not presenting it as a practical solution to these two problems in real-world BGP. We discuss the open question of whether hidden information is compatible with increasing algebras in Section VI-B.

Path-weights in our protocol are defined as follows: [0]dataPathWeight:Setwhere
i.e. there exists an invalid path-weight, and all other path-weights have a local preference, a set of communities and a path. The trivial path-weight, , is defined as: [0]0#:PathWeight
where is the highest possible local preference, is the empty set of communities and is the empty path. The invalid path-weight, , is defined as: [0]∞#:PathWeight
The choice operator, , is defined as: [0]_⊕_:Op₂PathWeight
which operates as follows:

  1. If either or is invalid return the other.

  2. otherwise if the local pref of either or is strictly greater than the other return that path-weight.

  3. otherwise if the length of the path of either or is strictly less than the other return that path-weight.

  4. finally break ties by a lexicographic comparison of paths.

We will now construct the set of edge weights . We start by defining a simple yet expressive language for conditions that can be used by our policy language to make decisions. [0]dataCondition:Setwhere
The semantics for these conditionals is defined as follows: [0]eval:Condition→PathWeight→Bool
We next define the policy language as follows: [0]dataPolicy:Set₁where
The semantics of each type of policy are defined by the function that applies policies to path-weights: [0]apply:Policy→PathWeight→PathWeight
[10] then(applypx)elsex
where the inflate function prepends copies of the path’s source on the front of the path. Note that since the policy language provides no way of increasing a path-weight’s local preference or decrease the length of its path, it is not possible to define a non-increasing policy.

We define the set of edge weight functions for each edge as: [0]dataExtension{n}(ij:Noden):Set₁where
and the application of an extension function is defined as: [0]_▷_:Extensionij→PathWeight→PathWeight
where (i,j)⇿?p tests if the edge is a valid extension of path (i.e. if ), and i∉?p tests whether or not  already exists in (i.e. if the resulting path would loop).

The function from path-weights to simple paths required by a path algebra can be defined as: [0]path:PathWeight→Path
where deflate strips from the path any consecutive duplicate routers that might have been introduced by inflate. The reader might have noted at this point that all our proofs in this paper work over simple paths, yet this algebra internally stores non-simple paths. However this apparent contradiction is resolved by observing that, as was discussed in Section II-F, the definition of a path algebra only requires that we can extract a simple path from a path-weight, rather than requiring that the path-weight stores a simple path. This demonstrates the power of the path algebra abstraction.

Taken together, the above definitions ensure that the algebra

satisfies all the requirements of an increasing path algebra as defined in Section II-C. For formal proofs of these properties see the Agda formalisation [24].

Our implementation of Theorem 5 in Agda guarantees that a protocol based on this algebra converges from any state to a unique solution even in the presence of message loss, reordering and duplication, i.e. it is impossible to write policies that interfere with convergence.

There are many other features of BGP that are safely increasing but, for space reasons, are not included in this model. For example hierarchical AS paths are possible to add with minor tweaks to the function and the policy language.

Vi Open questions

Vi-a Rate of convergence of path-vector protocols

One aspect of convergence that this paper does not address is how long it takes for convergence to occur. Consider a network of routers. It has long been known that path-vector protocols with distributive, strictly increasing policies require time to converge in the worst-case [1, 2]. However, it was only recently proved that non-distributive, strictly increasing path-vector protocols require time to converge in the worst-case [34]. Meanwhile, a tight bound for the time required for the worst-case convergence of path-vector protocols in free networks is still not known however. Current results have it lying between and .

Another question of interest is that it appears that not all non-distributive, strictly increasing algebras require iterations (e.g. the shortest-widest-paths algebra). We suspect that a careful analysis of policy language features might be able to tease apart distinct classes with respect to worst-case convergence time.

Vi-B Hidden information

Unlike in (external) BGP, in the algebra described in Section V the local preference attribute is not deleted when exporting a route. This raises a more general issue for routing protocols that allow information to be hidden. It is an open question as to whether it is possible to have increasing algebras with hidden information without requiring global coordination.

Ensuring increasing policies in today’s BGP may require communicating lost information with some other mechanism such as community values. Of course only the relative ranking of local preference values assigned within an AS matter. For example one AS might use a local preference of 100 for its most preferred routes, while another could use 2000. In this context can we ensure increasing policies using only bilateral agreements between neighbouring networks or does it truly require global coordination? If the latter, then a political rather than a technical solution is required.

Vi-C Verification of data-center policies

BGP is widely used today to implement (private) connectivity within and between data-centers [35]. In such an environment the network architects have total control of the global topology and therefore hidden information is not an issue. Yet even here we have witnessed the use of conditional policies, combined with filtering and the manipulation of local preference on routes. Perhaps tools such as Propane [36] could be extended to either ensure that all policies are strictly increasing, or at the very least provide warnings when they are not?

Vii Previous versions of this work

A previous version of this work has appeared before in conference proceedings [26]. This version makes the following new contributions, listed in order of appearance:

  1. in this version, the set of policy functions in a routing algebra in Definition 1 are parameterised by edges, which allows us to define a path algebra in Definition 2 without having it depend on , the current topology of the network. Although an apparently minor change, this makes the path algebras significantly easier to work with.

  2. in this version we use the new dynamic model of asynchronous iterations from [25] rather than the static model from [28] which only captures the dynamics of a single epoch. This is not only a more accurate model as it can capture messages from non-participating nodes from previous epochs, but it also results in the model being fully executable.

  3. as a consequence of the previous point, in this version we therefore use the dynamic AMCO convergence results from [25] instead of the static ultra-metric convergence results from Gurney [37] used in the previous version.

  4. in the previous version we only proved results about strictly increasing algebras (Theorems 3 & 5), whereas in this version we include results about free networks (Theorems 2 & 4).

With the exception of Theorems 2 & 4, much of this work appears in Chapters 4 & 5 in the PhD thesis of the first author [27].

Viii Acknowledgements

Matthew L. Daggitt was supported by an EPSRC Doctoral Train- ing grant and the ESPRC AISEC grant.

Dr Matthew L. Daggitt received his PhD degree in computer science from the University of Cambridge in 2018 and is currently working as a post-doc at Heriot-Watt University, Scotland. His main interests lie in the formal analysis and verification of complex systems, such as networking protocols and machine learning algorithms.

Timothy G Griffin received his B.S. degree in mathematics from the University of Wisconsin, Madison, in 1979, and M.S. and his Ph.D. degrees in computer science from Cornell University, Ithaca, NY, in 1985 and 1988, respectively. Previous experience includes teaching at UNICAMP in Brazil and more than a dozen years in industry at Bell Laboratories, AT&T Research, and Intel Labs. He joined the Computer Lab at the University of Cambridge in 2005.


  • [1] M. Gondran and M. Minoux, Graphs, dioids and semirings: new models and algorithms.   Springer Science & Business Media, 2008, vol. 41.
  • [2] J. S. Baras and G. Theodorakopoulos, “Path problems in networks,” Synthesis Lectures on Communication Networks, vol. 3, no. 1, pp. 1–77, 2010.
  • [3] D. P. Bertsekas, R. G. Gallager, and P. Humblet, Data networks.   Prentice-Hall International New Jersey, 1992, vol. 2.
  • [4] Y. Rekhter, T. Li, and S. Hares, “A border gateway protocol 4 (BGP-4),” RFC 4271, 2006. [Online]. Available:
  • [5] D. McPherson, V. Gill, D. Walton, and A. Retana, “Border gateway protocol (BGP) persistent route oscillation condition,” RFC 3345, 2002. [Online]. Available:
  • [6] K. Varadhan, R. Govindan, and D. Estrin, “Persistent route oscillations in inter-domain routing,” Computer networks, vol. 32, no. 1, pp. 1–16, 2000.
  • [7] T. G. Griffin and G. Huston, “BGP wedgies,” RFC 4264, 2005. [Online]. Available:
  • [8] G. Huston, “Interconnection, peering and settlements: Part I,” Internet Protocol Journal (Cisco), vol. 2, no. 1, June 1999.
  • [9] ——, “Interconnection, peering and settlements: Part II,” Internet Protocol Journal (Cisco), vol. 2, no. 2, June 1999.
  • [10] L. Gao and J. Rexford, “Stable internet routing without global coordination,” IEEE/ACM Transactions on Networking (TON), vol. 9, no. 6, pp. 681–692, 2001.
  • [11] J. L. Sobrinho and T. G. Griffin, “Routing in equilibrium,” Mathematical Theory of Networks and System, 2010.
  • [12] J. L. Sobrinho, “An algebraic theory of dynamic network routing,” IEEE/ACM Transactions on Networking (TON), vol. 13, no. 5, pp. 1160–1173, 2005.
  • [13] C.-K. Chau, “Policy-based routing with non-strict preferences,” in Proceedings of the 2006 conference on Applications, technologies, architectures, and protocols for computer communications, 2006, pp. 387–398.
  • [14] J. L. Sobrinho and M. A. Ferreira, “Routing on multiple optimality criteria,” in Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, 2020, pp. 211–225.
  • [15] A. J. Gurney and T. G. Griffin, “Lexicographic products in metarouting,” in 2007 IEEE International Conference on Network Protocols.   IEEE, 2007, pp. 113–122.
  • [16] H. Geng, X. Shi, X. Yin, Z. Wang, and S. Yin, “Algebra and algorithms for multipath QoS routing in link state networks,” Journal of Communications and Networks, vol. 19, no. 2, pp. 189–200, 2017.
  • [17] M. J. Fischer, N. A. Lynch, and M. S. Paterson, “Impossibility of distributed consensus with one faulty process,” Journal of the ACM (JACM), vol. 32, no. 2, pp. 374–382, 1985.
  • [18] G. Klein, K. Elphinstone, G. Heiser, J. Andronick, D. Cock, P. Derrin, D. Elkaduwe, K. Engelhardt, R. Kolanski, M. Norrish, T. Sewell, H. Tuch, and S. Winwood, “seL4: Formal verification of an OS kernel,” in Proceedings of the ACM SIGOPS 22Nd Symposium on Operating Systems Principles (SOSP), 2009.
  • [19] R. Kumar, M. O. Myreen, M. Norrish, and S. Owens, “CakeML: A Verified Implementation of ML,” in POPL, 2014.
  • [20] X. Leroy, “Formal verification of a realistic compiler,” Commun. ACM, vol. 52, no. 7, pp. 107–115, 2009.
  • [21] A. Guha, M. Reitblatt, and N. Foster, “Machine-verified network controllers,” in PLDI, 2013.
  • [22] A. Zaostrovnykh, S. Pirelli, L. Pedrosa, K. Argyraki, and G. Candea, “A formally verified NAT,” in SIGCOMM, 2017.
  • [23] U. Norell, “Dependently typed programming in Agda,” in Proceedings of the 4th International Workshop on Types in Language Design and Implementation, 2009.
  • [24] M. L. Daggitt, R. Zmigrod, L. v. d. Stoep, and T. G. Griffin, “Agda routing library,” 2021. [Online]. Available:
  • [25] M. L. Daggitt and T. G. Griffin, “Dynamic asynchronous iterations,” 2020.
  • [26] M. L. Daggitt, A. J. Gurney, and T. G. Griffin, “Asynchronous convergence of policy-rich distributed bellman-ford routing protocols,” in Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, 2018, pp. 103–116.
  • [27] M. L. Daggitt, “An algebraic perspective on the convergence of vector-based routing protocols,” Ph.D. dissertation, University of Cambridge, 2019. [Online]. Available:
  • [28] A. Üresin and M. Dubois, “Parallel asynchronous algorithms for discrete data,” Journal of the ACM (JACM), vol. 37, no. 3, pp. 588–606, 1990.
  • [29] B. A. Carré, “An algebra for network routing problems,” IMA Journal of Applied Mathematics, vol. 7, no. 3, pp. 273–294, 1971.
  • [30] J. Postel, “User datagram protocol,” RFC 768, 1980.
  • [31] R. Chandra, P. Traina, and T. Li, “BGP communities attribute,” RFC 1997, 1996. [Online]. Available:
  • [32] T. G. Griffin, “Exploring the stratified shortest-paths problem,” Networking Science, vol. 1, no. 1, pp. 2–14, Mar 2012. [Online]. Available:
  • [33] T. G. Griffin and G. Wilfong, “Analysis of the MED oscillation problem in bgp,” in Network Protocols, 2002. Proceedings. 10th IEEE International Conference on.   IEEE, 2002, pp. 90–99.
  • [34] M. L. Daggitt and T. G. Griffin, “Rate of convergence of increasing path-vector routing protocols,” in 2018 IEEE 26th International Conference on Network Protocols (ICNP).   IEEE, 2018, pp. 335–345.
  • [35] P. Lahiri, G. Chen, P. Lapukhov, E. Nkposong, D. Maltz, R. Toomey, and L. Yuan, “Building scalable data centers: BGP is the better IGP!” presentation, 2012.
  • [36] R. Beckett, R. Mahajan, T. Millstein, J. Padhye, and D. Walker, “Don’t mind the gap: Bridging network-wide objectives and device-level configurations,” in Proceedings of the 2016 ACM SIGCOMM Conference, 2016.
  • [37] A. J. T. Gurney, “Asynchronous iterations in ultrametric spaces,” Tech. Rep., 2017. [Online]. Available:

Appendix A Notational glossary

Notation Meaning Defined in
Set of all node labels Section II-B
Set of all edges Section II-B
Set of all simple paths Section II-B
The invalid path Section II-B
Arbitrary origin node N/A
Arbitrary destination node N/A
Arbitrary neighbour node N/A
Arbitrary path N/A
Set of path-weights Definition 1
Choice operator over path-weights Definition 1
Set of policy functions for edge Definition 1
Set of all policy functions Definition 1
Trivial path-weight Definition 1
Invalid path-weight Definition 1
Function from path-weights to paths Definition 2
The constantly invalid policy Section II-D
Total order over path-weights Definition 7
Strict total order over path-weights Definition 7
Arbitrary path-weights N/A
Arbitrary policy function N/A
Set of labels of nodes that participate Section II-D
Set of routing tables ( vectors over ) Section II-D
Set of states ( matrices over ) Section II-D
Arbitrary adjacency matrix Section II-D
Identity matrix Section II-D
A single sync. iteration with topology Section II-D
Synchronous state function Definition 3
Arbitrary routing tables N/A
Arbitrary routing states N/A
Arbitrary stable state N/A
The set of times of events of interest Section II-E
The set of epochs Section II-E
An arbitrary network Section II-E
Schedule activation function Definition 4
Schedule data flow function Definition 4
Schedule epoch function Definition 4
Schedule participants function Definition 4
Participants function Section II-E
Participation adjacency matrix Definition 5
Asynchronous state function Definition 6
Set of accordant states Definition 12
Set of all configurations Section III-B
Set of free configurations Section III-D
Arbitrary time N/A
Arbitrary epoch N/A
Arbitrary set of participants N/A
Set of assignments Definition 8
Strict partial order over assignments Definition 9
Extended by relation over assignments Definition 10
Threatens relation over assignments Definition 11
Arbitrary assignments N/A
Dislodging order over assignments Appendix B
Height function over assignments Appendix B
Distance function over path-weights Appendix B
Distance function over routing table states Appendix B
Distance function over routing states Appendix B

Appendix B Proof of Theorem 2 - Distance vector protocols

This appendix contains the proof that distance-vector protocols over a finite routing algebra are convergent whenever the network topology is free.

Assume that we have a finite routing algebra and some network . We will prove that asynchronous model of the protocol is convergent whenever the network topology is free, by showing that the set of synchronous iterations is an AMCO over . The required result then follows by Theorem 1.

B-a Constructing the distance functions

Consider an arbitrary configuration consisting of an epoch  and set of participants  such that the participation topology  is free. To fulfil the AMCO conditions, for each router we must construct a suitable distance functions, , that can be used to compare the different states of its routing table. Intuitively, our notion of distance will be based upon the ability of the router ’s current entries of dislodging entries in other router’s routing tables.

We now describe the formal construction of such a distance function. Note that for notational convenience, whenever we refer to a concept that depends an adjacency matrix we will use as a shorthand for . For example we will write instead of and instead of .

B-A1 Dislodgement order over assignments

To begin with we construct a strict partial order over the set of assignments, which we will call the dislodgement order. The key idea for this is adapted from Sobrinho’s notion of a couplet digraph which we generalise to an assignment digraph. The vertices of this digraph are the set of assignments . There is an edge from assignment to assignment in the digraph if, in the current topology, either the latter is an extension of the former (i.e. ), or the former is strictly preferred to the latter (i.e. ). Then if there exists a path from assignment to assignment in this digraph.

Mathematically this relation is the transitive closure of the union of the and relations:

Intuitively, describes a chain of events triggered by a router choosing to use assignment that would eventually cause another router to remove assignment from its routing table, i.e. can eventually cause to be dislodged. In particular, each link represents an extension event and each link represents a choice event.

The crucial properties of this relation are as follows:

  1. [label=O0]

  2. is irreflexive - An assignment cannot dislodge itself.

  3. is transitive - If assignment can dislodge and can dislodge then can dislodge .

  4. respects - If assignment is extended by assignment then can dislodge .

  5. respects - If assignment is strictly preferred to assignment then can dislodge .

Properties O2, O3 and O4 are immediate from the definition of . In contrast O1, the proof of irreflexivity, is non-trivial and uses the assumption that is free. However, modulo the differences in the algebraic structures and the change from couplets to assignments, the proof is identical to that of Lemma 1 in the appendix of [12]. Readers interested in a proof using our algebraic model may consult our accompanying Agda code.

B-A2 Height of assignments

As both and are finite, the set of assignments is also finite. Therefore all upwards closed subsets under the relation must also be finite. Consequently the height of an assignment can be defined as the number of assignments it is capable of dislodging:

The crucial properties of are as follows:

  1. [label=H0]

  2. If is extended by then the height of is strictly less than the height of :

  3. If is strictly preferred to then the height of is strictly less than the height of :