Tiramisu: Fast and General Network Verification

06/05/2019 ∙ by Anubhavnidhi Abhashkumar, et al. ∙ 0

Today's distributed network control planes support multiple routing protocols, filtering mechanisms, and route selection policies. These protocols operate at different layers, e.g. BGP operates at the EGP layer, OSPF at the IGP layer, and VLANs at layer 2. The behavior of a network's control plane depends on how these protocols interact with each other. This makes network configurations highly complex and error-prone. State-of-the-art control plane verifiers are either too slow, or do not model certain features of the network. In this paper, we propose a new multilayer hedge graph abstraction, Tiramisu, that supports fast verification of the control plane. Tiramisu uses a combination of graph traversal algorithms and ILPs (Integer Linear Programs) to check different network policies. We use Tiramisu to verify policies of various real-world and synthetic configurations. Our experiments show that Tiramisu can verify any policy in < 0.08 s in small networks ( 35 devices) and < 0.12 s in large networks ( 160 devices), and it is 10-600X faster than state-of-the-art without losing generality.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction and Background

Many networks, including university, enterprise, ISP and data center networks, employ complex distributed control planes running atop rich underlying network structures. The control planes run a variety of routing protocols, such as RIP, OSPF, and BGP, which are configured in intricate ways to exchange routing information both within and across protocols. In some cases, the protocols are assisted by other protocols (e.g., iBGP assisting in distributing BGP information throughout a network). Networks also employ techniques to virtualize the control plane, such as virtual routing and forwarding (VRF), as well as techniques to virtualize multiple network links into different broadcast domains (VLANs).

Bugs can easily creep into such networks, e.g., through errors in the detailed configurations that many of the protocols need. Often, bugs may not be immediately apparent, and the network may be running “just fine” until a failure causes a latent bug to be triggered. When bugs do manifest, a variety of catastrophic outcomes can arise: the network may suffer from blackholes; services with restricted access may be rendered wide open; critical applications can no longer be isolated from other services/applications, and so on.

Verification tools and their trade-offs. A variety of tools attempt to verify if networks are violating, or could potentially violate, policies of the above kind. Data plane verifiers [13, 10, 11] analyze the current forwarding tables and check for blackholes, loops, or broken path isolation. Unfortunately, these tools don’t have the means to analyze if the network’s new data plane that materialized upon reacting to a failure, can satisfy relevant policies or not.

To overcome this issue, a variety of control plane analyzers were developed [5, 2, 16, 4, 6]. These proactively analyze the network against various environments, e.g., failures or external advertisements. While a significant step forward in network verification, control plane tools today make trade-offs between performance and generality.

On the one hand are “graph-based” tools such as ARC [6]. ARC encodes all paths that may manifest in a network under various failures into a series of weighted digraphs. This abstraction enables analyzing the network under many potential environments at once by running very fast polynomial time graph algorithms; e.g., checking if two hosts are always blocked amounts to checking if they are in different graph connected components. Unfortunately, ARC ignores many network design constructs, including modeling the intricacies of BGP and iBGP, and the existence of VLANs, and VRFs.

On the other hand are “SMT-based” tools such as Bagpipe [16] and Minesweeper [2]. These tools create a detailed model of the control plane by symbolically encoding routing information exchange, route selection logic, and the environment (e.g., failures) using logical constraints/variables. By asking for a satisfying assignment for a SMT formula that encodes the network and the property of interest, they can identify a concrete environment that leads to property violation. These tools offer much better coverage of control plane protocols than graph-based ones, but their verification performance is very poor, especially when exploring failures (§6), despite many internal SMT-specific optimizations.

Decoupling encoding and properties. We ask if it is possible to design a verification tool that marries the speed of graphs with the generality of an SMT-based encoding.

We start by noting that today the trade-off between performance and generality in tools, arises from a coupling between the control plane encoding used in the tools and how properties are verified. In graph-based tools, the weighted graph control plane model requires graph algorithms to verify properties. In SMT-based tools, the detailed constraint-based control plane encoding requires a general constraint solver to be used for all properties.

Our framework, Tiramisu, decouples the encoding from the property: it uses a sufficiently rich encoding for the network that models various control plane features and network design constructs. But then, it permits the use of custom algorithms that offer the best performance to verify a property of interest.

Richer graphs. Our framework starts with graphs as the basis for modeling networks, because graph-based control plane analysis has been shown to be fast [6]. We then embellish the graph model with richer graph constructs, such as hierarchical layering, the notion of hedges [7] (edges that fail together), and rich, multi-attribute edge and node labels. Our resulting graph can model all aspects we observe used in real world networks and configurations.

Given the graph structure, we then develop a suite of custom techniques that help verify various properties of interest.

Avoiding path enumeration. First, we note that some properties of interest do not require the computation of the actual path that the network would induce on a certain failure; they care mostly about whether paths exist or not. An example is whether two hosts are always blocked from each other. For such properties, we develop two techniques that avoid path enumeration altogether: a modified depth first search graph traversal algorithm, and a simple integer linear program (ILP) formulation that computes graph cuts. Importantly, graph traversal runs in polynomial time. And, for a given property, the simplified property-specific ILP can be solved much faster than a general SMT-encoding in SMT-based tools. This is because the former explores symbolically only the variables that are relevant to the property being verified, whereas the SMT solver for the general encoding searches through a much larger search space.

Domain-specific path computation.

For the remaining properties that require computation of paths, we run a modified path vector protocol atop our richer graph abstraction. Here, we leverage foundational work by Griffin

et. al which showed that various routing protocols can be modeled as instances of the stable paths problem [8] (this insight was used in Minesweeper), and that a “simple path vector protocol” (SPVP) [8]

emulates the computation of a solution to this problem. In our version of SPVP, each node consumes the multi-dimensional attributes of incoming edges and neighboring vertices, and uses simple arithmetic operations that encode existing protocols’ logic to select among multiple paths available at the moment. Importantly, in simple networks (e.g., those that use a single routing protocol) our protocol naturally devolves to being similar to distance vector protocol, which runs in polynomial time (being based on the Bellman-Ford algorithm). For more general networks, we empirically find that our protocol can quickly compute the paths that are relevant to verifying specific properties. The performance is faster than SMT-based tools, because our approach essentially uses a highly domain-specific approach to finding paths, compared to a SMT-based general search strategy; for instance, even for simple networks, a SMT-based strategy would invoke the solver to find a satisfying assignment.

Prototype and evaluation. We implemented Tiramisu in Java (7K LOC) and evaluated it with many real data center and university networks, as well as networks from the topology zoo [12]. We find that Tiramisu’s rich multi-layered graphs can be computed from configurations in a few s per traffic class. Using Tiramisu’s custom algorithms, various properties can be checked for complex networks in 3-80ms per traffic class. Compared to Minesweeper that uses a general encoding with an SMT solver, Tiramisu offers speed of up to 600X for reachability policy verification and 10-50X for bounded-length and path preference policies (both under failures). Tiramisu’s algorithmic approach renders it substantially faster even when verifying properties under no failures. Finally, Tiramisu scales well, providing verification results in 100ms per traffic class for networks with 160 routers.

2 Motivation

Given the significant performance benefits of graph abstractions [6], we use graphs as the basis to encode control plane computation in Tiramisu. However, the abstraction we use is significantly different from ARC [6]. In what follows, we provide an overview of ARC’s graph based approach for verification. We then identify its key drawbacks which motivate Tiramisu’s graph design.

2.1 Arc

ARC (Abstract representation for control plane [6]) is a state-of-the-art control plane verifier. It models a network’s control plane using a collection of directed graphs. There is one directed graph per traffic class which models the forwarding behavior of packets belonging to that traffic class. In ARC, nodes represent routing processes, and directed edges represent possible flow of traffic enabled by exchange of route advertisements between routing processes. Using a single attribute edge-weight, ARC can model OSPF costs and AS_path length. Finally, ARC verifies policies by checking some simple graph characteristics. Table 1 lists some of these policies and related graph properties. By leveraging graphs, ARC offers order-of-magnitude better performance [6] compared to state-of-the-art [5, 2].

However, ARC’s drawback is that it is limited in its network design coverage. It does not model layer 3 protocols such as iBGP and VRF. It does not model any layer 2 protocols including VLANs. It also does not model BGP protocol attributes such as local preference, communities etc. We use examples to show how these limitations can affect ARC’s correctness, and what we do in Tiramisu to overcome them.

Policy class Graph characteristics
P1: and are always block and are in separate components
P2: All paths from to traverses a waypoint after removing the waypoint, and are in separate components
P3: can reach when there are K link failures min-cut of graph is K
Table 1: Policies as graph characteristics
Figure 1: Multilayer dependency
Figure 2: Multiple network features

Cross layer dependency. Consider the network in Figure 1. The network consists of 5 routers (). Here, routers and are peers of router , and, router is an peer of router . All routers except belong to the same process, with each link except having a link cost of 1. Link has cost 5. In this scenario, learns a route to through its neighbor , with as its next hop. forwards all traffic destined to , to . peers forward traffic at the layer. Hence uses its process to find the path to reach . The path computed varies for different failure scenarios, which affects reachability of traffic. Depending on the path used under a given failure, the operator’s policy for traffic between C and E may be violated:
i) Under no failures, prefers path (OSPF cost 1), and then traffic flows from ;
ii) When the link fails, prefers a different path (cost 3). Crucially, traffic at gets directly rerouted to , because is the peer of ;
iii) When links and fail, prefers path (cost 6). Here traffic gets dropped at , because did not learn a route for , and is disconnected from . Note that, had router or redistributed it’s routes to , then would have learned a route to , avoiding the blackhole situation (iii) above.

Such dependencies between and cannot be modeled in ARC. ARC cannot analyze these scenarios and compute the actual paths used, and because of it, it cannot be used to verify any policies in this network.

ARC’s lack of modeling of cross-layer dependencies impacts its applicability in other network scenarios too. Consider the network in Figure 2. This network has 5 routers (), a switch , and three hosts . All routers run BGP, and routers and also run OSPF (with cost 1). Switch connects to router and on VLAN 1 and 2, respectively. adds community “c1” to its advertisements, removes “c1”, and blocks all advertisements with community “c1”. Finally, to prefer routes learned from over , assigns local preference values and to and , respectively.

Although routers and are connected to the same switch , they belong to different VLANs. Hence, traffic cannot flow through switch . By default, ARC assumes layer 2 connectivity. Hence according to ARC routers and are reachable and traffic can flow between them.

The overall theme is that protocols “depend” on each other. E.g. iBGP depends on OSPF, BGP and OSPF depend on VLANs etc. These protocols also operate at different “layers”. BGP operates at the EGP (Exterior Gateway Protocol) layer, OSPF at the IGP (Interior Gateway protocol) layer, and VLANs at Layer 2. A graph abstraction needs to encode layers and cross-layer dependencies. Thus, Tiramisu introduces a new multilayer graph abstraction, where traffic flow in the higher layers may depend on the traffic flow in the lower layers. Figure 3 and 4 show the multilayer graphs of the aforementioned networks, which we explain in detail in §3

Impact of BGP attributes. In Figure 2, the path taken from to depends on communities and local preference. There are two paths from to : i) , and, ii) . Because of local preference, path (i) is preferred over path (ii). However, blocks all advertisements with community “c1”. Since router adds this community, all advertisements through will have community “c1”. Hence the advertisement for path (i) is blocked. Although the advertisement for path (ii) also comes through , router removes community “c1” and router does not see that community. Therefore, there is only one path between and : . ARC, on the other hand, characterizes path (i) as valid.

Additionally, ARC cannot model other BGP attributes like local preference, MED etc. To support all BGP attributes as well as metrics of other protocols (OSPF cost, AD), Tiramisu uses multiple edge and node attributes. We elaborate in §3.

Physical link dependency. Consider the network in Figure 2 without link . Here, according to ARC, traffic from to can flow through 2 paths and . To evaluate reachability under failure, ARC calculates min-cut of this graph as 2, and concludes that it can withstand arbitrary 1 link failures. However, this is incorrect because and belong to the same underlying physical link whose failure causes disconnection.

In graphs, the notion of hedges [7] can be used to model such dependency among edges. A hedge is a set of edges that fail together. A graph can have multiple hedges, each with more than one edge in it, and a single edge can belong to multiple hedges. In Tiramisu, to support edges that fail together, we label sets of “related” edges as hedges. Overall, Tiramisu converts the control plane into a series of multilayer multi-attribute hedge graphs one per source-destination pair.

3 Tiramisu Graph Abstraction

Figure 3: Correlated Edges
Figure 4: Multilayer Hedge Graph for traffic class

We discuss the components of the Tiramisu graph abstraction.

Nodes. In Tiramisu, nodes are created for both switches and routers. For switches, nodes are created per device, per VLAN. E.g, in Figure 4, has two nodes and for VLANs and . For routers, nodes are created per device, per routing process. E.g, in Figure 4, router has two nodes and for it’s BGP and OSPF routing processes. If the router supports VRF (Virtual Routing and Forwarding), then the nodes are replicated per device, per virtual routing process (similar to VLAN). Note that by default, each routing process has a default_vrf, and nodes like represent the default_vrf of B’s BGP process. We identify routing processes, VLANs, and VRFs from device configurations. We also create, per router, a node representing the router’s forwarding information base (FIB). Finally, we create two special nodes representing the and of a traffic class.

Edges. Tiramisu categorizes it’s edges into multiple labels, depending on the edge’s end nodes. These labels include: FIB (f), eBGP (b), IGP (o), static (s), redistribute (r), iBGP (i), and physical (p). Some of these labels are for inter-device edges and some for intra-device edges. We show examples of these labels in Figure 4, and explain them below.

Inter device edges. An inter-device edge exists between different devices that have a physical link connecting them. There are three cases: (i) if both end nodes are switches and belong to the same VLAN, connect them with an edge of label “p”, (ii) if one node is a switch and other a router, connect them with edge of label “p”, e.g. . (iii) if both nodes are routers, they belong to the same routing process (IGP or eBGP) and the same VRF, then connect them with edge label “o” or “b”, respectively, e.g. .

Intra device edges. An intra-device edge exists between nodes that belong to the same device. There are three cases: (i) if the router redistributes routes from process to , add edge of label “r” from to , e.g. , (ii) if the router is an iBGP peer, add edge of label “i” from the router’s BGP process to the underlying IGP process, e.g. of Figure 3, or (iii) every node corresponding to a routing process on a given router is connected (both ways) to the router’s FIB node; e.g., is connected to (Figure 3).

Each traffic class has a and that originate the and IP addresses. Tiramisu connects the node to all routing processes of . Finally, it connects all routing processes of with node.

Edge costs. Tiramisu supports multiple routing protocols. Each protocol uses a different set of metrics to express link and path costs/preferences. E.g. OSPF uses link costs, and, BGP uses AS-path length, local preference (lp), Multi-Exit Descriptor (MED) etc. Additionally, an administrative distance (), allows operators to choose routes from different protocols. Hence a single edge weight cannot model all protocols. Tiramisu replaces these edge weights with a vector of metrics. The values of these metrics are inferred from device configurations. Depending on the edge label, certain metrics will be set as null, e.g. OSPF cost is null for “b” edges.

Hedges. All edges that represent the same physical link belong to the same hedge. E.g. in Figure 4, and belong to the same physical link , and thus the same hedge.

Traffic class specific constructs.: Some aspects of the network are specific to traffic classes, e.g. ACLs, filters, static routes, etc. Thus, Tiramisu first creates a single base graph representing the above features/protocols for the entire network. From that, it then creates traffic class-specific graphs.

ACLs prevent specific traffic classes from entering/leaving a router, and filters block advertising specific prefixes. Thus, for a given traffic class, Tiramisu logically removes an edge of the traffic class graph if there is (i) an ACL that blocks this traffic class on the interface associated with this edge, or, (ii) a filter that blocks this traffic class’s destination prefix on the routing process associated with the start node of this edge.

Static routes are also traffic class specific. Tiramisu adds an edges of label “s” from the node representing the source of the static route process to all nodes associated with the next hop router, for the relevant traffic class.

Node communities. Communities are also a traffic class specific construct. In Tiramisu, each node representing a process has three sets of communities: “ac” - communities added by the node, “rc” communities removed by the node, and “mc” communities matched/acted upon by the node. The action associated with each “mc” is to either block traffic or change edge metrics like local preference and MED.

3.1 Prohibited paths

All the algorithms we present to verify various properties rely on examining properties of source-destination paths in the above graph abstraction. In designing these algorithms care must be taken to avoid certain paths that cannot materialize in a real network under any failure. This mainly arises due to constraints on interactions between routing protocols and route redistributions/static routes, and due to communities. We show how we reason about which routes cannot be taken due to the former reason. We handle communities later.

For example, as shown in §2, traffic crossing from an EGP to the IGP layer gets dropped if the intermediate nodes at the IGP layer do not learn/have a route to .

Intuitively, a path can materialize in the network if there is relevant forwarding information available toward the destination in the RIB of some routing process running on each router on the path, and thus in the router’s FIB; this observation forms the basis for the seminal work on static reachability analysis of IP networks [17]. A path cannot materialize if it includes a router with no forwarding information for the destination. To keep track of such disallowed paths for graph traversal or when computing paths taken, Tiramisu uses a “tainting” strategy. In a preprocessing step, taints track how forwarding information may flow through a network’s RIBs and may populate FIBs. Taint on a node implies that a node may know of a route to a destination; lack of a taint implies the node will not know of any route.

For each traffic class, taints propagate in the corresponding Tiramisu graph starting at the destination ; the routing process connected to the is tainted, as are all other peer processes of the same routing protocol instance on other routers. Taints flow across redistribution and static route edges, and spread throughout the vertices corresponding to a single IGP. Taints also propagate from one iBGP peer to another (because iBGP peers learn of routes from each other).

Specifically, to identify nodes in the IGP layer that have a route to , Tiramisu first finds the node at the EGP layer that redistributes its route to IGP, and taints this node.111Note, that edges in Tiramisu represent the flow of traffic and not of advertisements. Hence, redistribution of EGP routes into IGP is represented by an “r” edge from the IGP node to the EGP node. Next, Tiramisu marks all nodes in the IGP process’s layer as tainted, implying that they may know of a route to in their RIB due to redistribution from the EGP (recall: IGPs flood learned routes, to routers in the same IGP process). On the other hand, if there is no redistribution from EGP into IGP, the corresponding IGP node is marked as untainted, because it will not not have a route to the destination in its RIB (but a route may exist in the router’s FIB because the EGP process computed a route); all other nodes in the same IGP process are also untainted. E.g. in Figure 3, all OSPF nodes are untainted.

The above processes is applied to all routing layers and EGP/IGP crossings, of which there may be many in a network. In the end, we have a tainted graph, with a subset of vertices carrying taints and others without taint; the latter vertices will not have a forwarding entry to the destination.

Given a tainted graph, we determine which paths are disallowed. Intuitively, an untainted node must reside on the same router as a tainted node for a path to the destination to be found in the router’s FIB. Thus, a potentially valid path has consecutive untainted nodes (e.g., the two IGP nodes). Any path with consecutive untainted nodes is an invalid or prohibited path. E.g. in Figure 3, is a subpath of prohibited paths.

Although our example considered iBGP, IGP and BGP interactions, taints can also be used to understand how static routes can shape the flow of routing information, and consequently impact paths.

Theorem 1

The above approach correctly identifies all prohibited path.

We prove this by contradiction as shown in Appendix B.1.

A potentially valid path may still not materialize in the network due communities. We handle communities specially in our algorithms in subsequent sections.

4 Avoiding Path Enumeration

For many properties of interest, verifying them does not actually require computation or enumeration of actual paths that may arise in the network; rather, we simply need to know of the existence of paths (possibly of a certain kind). Knowing of the existence of paths is far simpler than actually computing paths under a given failure.

Thus, we design two sets of algorithms, one that traverses a graph along all potential paths, and another that uses ILPs to reason about the number of paths and high-level path properties (maximum path length). We present these algorithms in the context of the properties they aid in the verification of. All algorithms use taints to take invalid paths out of consideration, and handle communities specially as mentioned above.

4.1 Tiramisu Depth First Search

(Depth First Search) is a graph traversal algorithm that identifies all nodes that are reachable from a given . A node remains unvisited after iff there exists no path from to that node. The DFS algorithm can be naturally used to verify certain reachability policies, such as, (“always blocked”): is true iff under every failure scenario, there exists no path between and node. However standard DFS does not avoid prohibited path constraints, nor does it account for how BGP communities may shape the existence of paths. E.g. in Figure 3, paths with subpath are prohibited. But identifies them as valid.

2:     is the graph
3:     is the root node for depth first search
5:     Set all nodes of as unvisited
6:     (, , 0)
8:     is current node being traversed
9:     tracks # consecutive untainted nodes
10:procedure (G, u, )
11:     Set node as visited
12:     for each  do
13:          v end node of edge e
14:          Ignore v, if already visited
15:          if  is untainted then
16:               increment by 1
17:               Ignore v, if is 3
18:          else
19:                0           
20:          (G, v, )      
Algorithm 1 Tiramisu Depth First Search

We first assume no communities are in use and propose a modified DFS algorithm (Algorithm 1) that avoids prohibited paths, (Tiramisu DFS). We then account for communities by invoking on carefully selected subgraphs.

Like DFS, explores all unvisited outbound neighbors of a node (line 6 to 8). However, it uses an additional variable to avoid traversing prohibited paths.

In this algorithm, represents the number of consecutive untainted nodes seen by ; simply avoids all paths with more than 2 consecutive untainted nodes (line 11). E.g., in Figure 4, when reaches node , becomes 3 and stops exploring this path. ’s complexity is the same as standard DFS.

algorithm does not handle communities. Nodes may add communities to path advertisements that others filter on. When such filtering happens, the corresponding paths cannot be taken by network traffic. To support communities, Tiramisu uses another simple algorithm, (Algorithm 2). makes a constant number of calls to verify . Thus, its overall complexity is still polynomial-time.

As shown in §2, nodes can add, remove or block on communities. In presence of such nodes, the order in which these nodes are traversed in a path decides if is reachable.

In the algorithm, Tiramisu first checks if and are unreachable according to (line 5, 6). If they are reachable, then checks if all paths that connect to has i) a node (say X) that blocks on a community in an advertisement for (line 8 to 10), ii) followed by a node (say Y) that adds that community to an advertisement for (line 11 to 13), iii) and no node between them X and Y that removes that community (line 14 to 16). (Recall here that advertisements flow in the opposite direction of data traffic.) If all these conditions are satisfied, then and are unreachable. If any of these conditions are violated, the nodes are reachable. In Figure 4, violates condition (iii).

2:     is the graph
3:     and are source and destination nodes
4:    It uses to answer multiple (un)reachability queries
5:procedure commTDFS
6:      nodes that add community
7:      nodes that remove community
8:      nodes that block on community
9:     if  and are unreachable by  then
10:          return true, since nodes are already unreachable
11:     else
12:          remove nodes
13:          if  and are reachable by  then
14:               return false, since there is path between and that is not blocked by community           
15:          add back all nodes, remove nodes
16:          if nodes and are reachable by  then
17:               return false, since blocking nodes can receive advertisement without community           
18:          add back all nodes, remove nodes
19:          if nodes and their respective nodes are unreachable by  then
20:               return false, since communities are always removed before reaching blocking nodes           
21:          Return true, if all above conditions fail      
Algorithm 2 Always blocked with communities

This algorithm can also verify (“always waypointing”). After removing the waypoint, if can reach , then there is a path that can reach without traversing the waypoint.

4.2 Tiramisu Hedge Min-cut

Another policy that does not require path enumeration is (reachable K failures): here, operators want to verify that the can reach as long as there are link failures. For this, ARC [6] computes the min-cut for the - graph, i.e., the minimum number of edges whose removal can disconnect the graph. If min-cut is , then is satisfied.

Tiramisu uses a multilayer hedge graph abstraction. Unfortunately, finding min-cuts in hedge graphs is a known NP-hard problem [7]. Thus, to verify , we propose a new Integer Linear Program (ILP) to find hedge graph min-cut, while also accounting for communities and prohibited paths.

For this property, Tiramisu uses integer variables, similar to Minesweeper [2]’s SMT encoding. However, Tiramisu’s property-specific ILP encoding is simpler than Minesweeper as it only adds integer and boolean variables relevant to hedge min-cut and avoids variables and constraints associated with route selection, which significantly reduces variables and constraints associated with route advertisements. Minesweeper’s detailed encoding exercises all of these variables in an attempt to compute the actual path, which renders it slow even for a small number of failures ().

Until now, the edges in our graphs represented the flow of traffic from to . For ease of understanding, in specifying the ILP, we reverse the edges to represent the flow of advertisement from to . For brevity, we explain the constraints at a high-level, leaving precise definitions to  Appendix C. Equation numbers below refer to equations in Appendix C.

The objective of the ILP is to minimize the number of physical link failures required to disconnect the from . Note that a single physical edge’s failure can cut multiple Tiramisu graph edges. All such edges belonging to hedge will share the same variable. Our objective then is:


Advertisement Constraints. We first discuss the constraints added to represent reachability of advertisements. The base constraints state that the originates the advertisement (Eqn 4). To disconnect from , the advertisement must not reach the (Eqn 5). For other nodes, the constraint is that an advertisement reaches a node if it gets propogated on any of its incoming edge (Eqn 6).

Next, we discuss constraints on propogation of advertisements. An advertisment can propogate through an edge , if it reaches the start node of the edge, the physical edge does not fail, the advertisement does not carry a community that is blocked on this node, and if edge carries atleast one advertisement that does not create a prohibited path. This is represented as shown in Eqn 7.

Communities. The base constraints state that each node that adds a community forwards that community and each node that removes that community does not forward it further (Eqn 8 and Eqn 9).

For other nodes, we add the constraint that node forwards a community iff any of its inbound neighbors forwards that community to node (Eqn 10).

Finally, we add the constraint that an edge carries a blocked community iff the start node of forwards a community that is blocked by the end node of edge (Eqn 11).

Prohibited Path Constraints. An edge , that only propogates advertisements that create prohibited paths, is an edge that satisfies the condition that start and end nodes of are untainted () (Eqn 12). Also, the start node receives advertisement only from untainted neighbors and not from any other neighbor. Such edges always create subpaths with three consecutive untainted nodes (§3.1) (Eqn 12).

4.3 Tiramisu Longest path

Similar to cut-based properties, there are others that don’t need path enumeration and can instead rely on custom-crafted ILPs. These properties compute high-level attributes of paths, such as bounds on length. An example is the always bounded length policy (). For a given , is true if under every failure scenario, traffic from to never traverses a path longer than hops. Enumerating all possible paths and finding their length is infeasible. However, this policy can be verified efficiently by viewing it as a variation of the longest path problem: for a given , is true if the longest path between and in the graph is .

ILP. Finding the longest path between two nodes in a hedge graph is also NP hard [9]. To verify , we propose another ILP whose objective is to maximize the number of inter device edges () traversed by an advertisement (). Note again that the path traversed by the advertisement is the opposite of traffic flow. But both have same path length. Notably, this ILP uses even fewer constraints and variables compared to §4.2, and thus can run even faster relative to Minesweeper.


Again, we elide detailed constraints to the Appendix D.

Single Path Constraints. We add constraints to ensure that only one path gets advertised, that the sends the advertisement, and that the receives it (Eqn 14 and Eqn 15).

For other nodes, we add the flow conservation property, i.e. sum of incoming flows is equal to outgoing flows (Eqn 16).

Advertisement Constraints. Finally, we add constraints on propogation of advertisements. Here, an advertisement can be blocked on edge if carries a blocked community or satisfies the prohibited path constraints (Eqn 17).

5 Path specific policies

As opposed to the properties in the previous section, the remaining properties that one may wish to verify require knowing the exact path taken in the network. Consider the policy that encodes a preference order among paths, say, . This states that should be taken by default; when path fails, path (if available) should be taken; and if and both fail, then , if available, should be taken. A path, say , can become unavailable for multiple reasons: (a) a router configuration along was updated to withdraw or filter the route to the ; (b) a link along failed. In all cases, we need to reason about what alternate paths materialize, and whether the materialized path is indeed the path that was meant to be taken. Simple graph traversal is thus insufficient to verify this property. We need the actual paths under various failure scenarios.

Griffin et al [8] observed that control plane protocols essentially attempt to solve the stable paths problem. Based on this, to compute paths, Minesweeper [2] models protocol interactions (advertisement generation/processing, best path selection, etc) as an instance of the stable paths problem. Furthermore, Griffin et al [8] proposed the Simple Path Vector Protocol (SPVP) for obtaining a set of paths that form a solution to the stable path problem.

Inspired by these two studies, we propose (Tiramisu PVP), a protocol that can compute actual network paths taken, while avoiding prohibited paths. extends SPVP with rich messages and per-node computation.

Whereas Minesweeper encodes the router’s path selection actions based on SMT constraints, models them using a protocol. Thus, Tiramisu emulates the execution of SPVP, whereas Minesweeper attempts to find the SPVP output.

To find the path traversed by a specific traffic class, Tiramisu runs on its traffic class-specific graph.

5.1 Tiramisu Path Vector Protocol

Advertisements. Similar to SPVP, for each node , and for each of its peer , uses variables to keep track of the most recent route advertisement received from . Each advertisement consists of (i) a path from the advertisement sender to the advertisement source ( of the traffic class), and (ii) the multi-metric path cost to reach . Also, represents the current path (best advertisement and best path cost) to reach from .

Route Selection. In Minesweeper, the values of were computed by applying import and export filters on these advertisements. In , these are replaced by an function. This function (i) rejects prohibited paths (§3.1), (ii) rejects paths blocked by communities (§4.1), and, (iii) computes the multi-atrribute path cost to reach through . Each node has a function () that provides a partial order over all paths. then updates its value to the best choice/most preferred .

also keeps track of the list of communities carried by the advertisements, and the function updates the community lists.

The and functions are closely modeled based on the route import, route export and route selection constraints of Minesweeper and SPVP. These functions are either inferred from configuration or based on standard conventions. E.g. could be based on BGP selection policy (prefer highest , shortest etc), and, specifies how to calculate the cost of as a function of and edge weight (add , set to etc).

Algorithm. Our final protocol is shown in Algorithm 3. In the initial state, sets the value of all nodes except to null (line 2). is set to , since it originates the advertisement (line 3). There are three main steps in each iteration. First, for each node , computes all its values based on the advertisement sent by its neighbors (line 6 to 8). If the current ’s value is different from the best advertisement, then updates and sends a message to all peers of node (line 9 to 11). This message contains the updated .

terminates when none of the nodes receive any messages about the new values (line 12, 13). This termination condition represents the converged state of the newtork. Similar to SPVP, if network reaches a converged state, then will find the best path to for all nodes.

Theorem 2

Assuming the network control plane converges, always finds the exact path taken in the network under any given failure scenario.

We leverage correctness of Minesweeper and SPVP to establish the correctness of as shown in Appendix B.2.

We return to the correctness of .

Theorem 3

traverse all real network paths that could materialize under some failure scenario, and does not traverse any path that cannot materialize on any failure scenario.

We leverage the correctness of Theorem 1 and Theorem 2 to prove correctness of Theorem 3 as shown in Appendix B.3.

2:     is the graph
3:     is the destination node
5:     set rib(i) values of all nodes except to null
6:     set rib(dst) to , since originates the advertisement
7:     while true do
8:          for each  do
9:               for each  do
10:                     =
11:                    compute using
12:                    if  is different from  then
13:                          =
14:                         send messages to all peers of node about change in                                              
15:          if no node receives any messages then
16:               break, since network has converged                
Algorithm 3 Tiramisu Path Vector Protocol

5.2 Tiramisu Yen’s algorithm

We return to determining how to verify the path preference property introduced earlier in this section. We first observe that there are similarities between analyzing path preference () and finding the shortest paths in a graph (the “ shortest paths problem” [18]). This is because, in shortest paths problem, the shortest path is taken only when paths have failed. Enumeration for all possible failures of all shorter paths is tedious. To handle that, Yen [18] introduced a new algorithm to find the shortest paths. The algorithm uses dynamic programming to avoid enumerating failures/link removals. Next, we explain Yen’s algorithm in detail. Then, we propose (Algorithm 4), a simple extension to Yen’s algorithm to verify . We show how in a simple fashion uses Yen’s algorithm to verify .

Yen’s algorithm. Yen uses two lists: (keeps track of the shortest path seen so far) and (keeps track of candidates for the next shortest path). At the start, Yen finds the first shortest path (line 2) from to using any shortest path algorithm (e.g. Dijkstra’s).

Next, it takes every node in path (line 11, 12), and finds the and of that node. The of is the subpath of from to node (line 13). The is the shortest path from node to (line 18) that satisfies the following two conditions: i) the path must not have any node from (line 17); ii) the path must not traverse any outgoing edge from that is part of any of the previous shortest paths having the same (line 15, 16). E.g. if is the shortest path and is at node , then is invalid.

Yen combines the and to form a new path (line 19). is added to if it doesn’t already exist in or (line 22). After traversing all nodes of path , Yen adds to (line 24). Next, Yen picks the shortest path from and reruns the previous steps with this path as the new . This continues till Yen finds paths.

2:     is the graph
3:    , are source and destination nodes
4:     is no. of path specified in path-preference policy
5:    , a map of preference level and path
7:      path from src to dst returned by
8:      1
9:      [], as best path requires no edge removal
10:      [], tracks paths already considered as
11:      [], tracks paths not yet considered
12:     do
13:           most preferred path in whose edges don’t overlap with
14:          if   then
15:               return false, since path preference is violated           
16:          for i 0 to P.length - 1 do
17:                node of P
18:                subpath of P from to
19:               for each   do paths in
20:                    if  has same at  then
21:                         remove outgoing edge of in , so that path is not considered also remove other edges that share the same hedge                                    
22:               remove all nodes and edges (hedge) of except to avoid loops
23:                path from to returned by
24:                +
26:                all edges removed in this iteration
27:               add nP to end of if np is valid and np [, ]
28:               add back all nodes and edges to the graph           
29:          add to
30:          P remove first path from
31:     while 
32:     return true, since loop didn’t find preference violation
Algorithm 4 Tiramisu Yen

Tiramisu Yen. The three failure modes highlighted at the beginning of this section boil down to some specific edges being removed from the Tiramisu graph, causing the path in question to cease to exist. Given this, we make a few modifications to Yen, resulting in an algorithm we call , to work atop our multi-graph abstraction and check for path preference; simply put, like Yen, we check if is the next path chosen after , and so on (line 8 to 10).

uses instead of Dijkstra to find the shortest path. In addition, associates each path with a variable, . This keeps track of edges that were removed to prefer this path (line 21). During each iteration of , identifies the most prefered path in that did not have an edge in (line 8). If this path varies from , then preference is violated (line 9, 10).

Revisiting: Reachability K with ACLs. The ILP in §4.2 for is not accurate when data plane ACLs are in use. ACLs do not influence route advertisements. Hence, routers can advertise routes for traffic that ends up being blocked by ACLs. Recall that during graph creation, Tiramisu removes edges that are blocked by ACLs §3. This leads to incorrect mincut computation as we show next:

Consider a - traffic class in a network. Say there are three network paths , and in increasing order of cost that learns of toward , and say has a data plane ACL on it. Suppose further than all three paths are edge disjoint. If a link failure removes , the control plane would select path to forward to , but all packets from are dropped at the ACL on . In this case, a single link failure (that took down ) is sufficient to disconnect and ; that is, the true mincut is 1. In contract, Tiramisu would remove the offending ACL edge from the graph abstraction, and since this preserves paths and , Tiramisu would conclude that the mincut is 2, which is incorrect.

We address this issue as follows. Nodes can become unreachable when failures either disconnect the graph or lead to a path with an ACL. Thus, we compute two quantities: (1) : How many minimum failures cause the control plane to pick a path that first encounters a blocking ACL? (2) : How many minimum failures cause disconnection in the Tiramisu graph with the ACL edge removed (as originally proposed)? The true min-cut value is ?

Computing in (2) is straightforward. For computing in (1), we first construct a graph without removing edges for ACLs. Then, we run until we find the first path with a dropping ACL on it. Say this was the shortest. Then, we use an ILP to compute the minimum number of edge failures that will cause the previous shortest paths to fail. If then is satisfied.

So far, we covered five properties to verify. But, using the algorithms in this and the previous section as the basis, Tiramisu can verify a variety of other properties which Minesweeper can also verify. We list these, along with how Tiramisu verifies them, in Appendix A.

6 Evaluation

% of Networks
Protocols/Modifiers University Datacenter Topology Zoo
eBGP 100% 100% 100%
iBGP 100% 0% 100%
OSPF 100% 97% 100%
Static routes 100% 100% 0%
ACLs 100% 100% 0%
Route Filters 100% 97% 100%
Local Prefs 50% 0% 100%
MPLS+VRF 100% 0% 0%
VLAN 100% 0% 0%
Community 100% 100% 100%
Table 2: Configuration constructs used in networks
Figure 5: Size of multilayer graphs of all networks
Figure 6: Size of multilayer graphs of all networks
Figure 7: Graph generation time (all networks)
Figure 8: Verify policies on university configs
Figure 9: Performance under all failures: Tiramisu vs Minesweeper (datacenter networks)
Figure 10: Speedup under all failures: Tiramisu vs Minesweeper (datacenter networks)
Figure 11: Speedup under no failures: Tiramisu vs Minesweeper (datacenter networks)
Figure 12: Variable reduction ratio - Tiramisu vs Minesweeper
Figure 13: Performance on scale (topology zoo networks)

Our implementation of Tiramisu is written in Java. We use Batfish [5] to parse router configurations. From these, we generate our multilayer graphs (§3). We implemented all our verification algorithms (§4 and §5) in Java. Tiramisu uses Gurobi [1] to solve our ILPs. In all, this amounted to K lines of code. We evaluate Tiramisu on a variety of issues:

  • [leftmargin=1em]

  • How quickly can Tiramisu verify different policies?

  • How does Tiramisu perform relative to Minesweeper [2]?

  • How does Tiramisu’s performance scale with network size?

All our experiments were performed on machines with 40 core 2.2 GHz Intel Xeon Silver Processors and 192 GB RAM.

6.1 Network Characteristics

In our evaluation, we use configurations from (a) 4 university networks, (b) 34 real datacenter networks operated by a large OSP, and, (c) 7 networks from topology zoo dataset [12]. The university networks have 9 to 35 devices. The university networks are the richest in terms of configurations constructs. They support eBGP, iBGP, OSPF, Static routes, ACLs/Filters, community, local preference, MPLS+VRFs and VLANs. The datacenter networks have between 2 and 24 devices. They do not employ local preference, MPLS or VLANs. Finally, the topology zoo networks have between 33 and 158 devices. The configs for these networks were synthetically generated using NetComplete [3]. These generated configs do not have Static routes, ACLs, MPLS or VLANs as Netcomplete cannot model them. Table 2 shows what percentage of networks in these datasets support each network protocol/modifier.

Figure 6 characterizes the size of all the multilayer graphs generated by Tiramisu for these networks. It first shows the number of nodes and edges used to represent the b

ase multigraph of these networks. We observe two outliers in both Figure 

6a and Figure 6b. These occur for networks (24 devices) and (26 devices), from the university dataset. These networks have multiple VRFs and VLANs, and Tiramisu creates nodes (and edges between these nodes) per VRF/VLAN per routing process. Note also that for the other networks, the number of routing processes per device varies. Hence, the number of nodes and edges do not monotonically increase with network size.

Policies. We consider five types of polices: () always unreachable, () always waypointing, () always reachable with failures, () always bounded length, and () path preference. Recall from other sections: and use ; uses and algorithms; uses ILP and .; And uses another ILP. Using these policies, we evaluate the performance of all our algorithms.

6.2 Verification Efficiency

We examine how efficiently Tiramisu can construct and verify these multilayer graphs. First, we evaluate the time required to generate these graphs. We use configurations from all the networks. Figure 7 shows the time taken to generate the base graphs and per traffic class-specific graph for all networks. Tiramisu can generate these graphs, even for large networks, in 30 s. The time to generate the traffic-class graph from the base graph is atmost 3 s on average per traffic-class.

Next, we examine how efficiently Tiramisu can verify various policies. Since the university networks are the richest in terms of configuration constructs, we use them in this experiment. Figure 8 shows the time taken to verify policies , , , and . In this and all the remaining experiments, the values shown are the median taken over 100 runs for 100 different traffic classes. Error bars represent the std. deviation.

We observe that can be verified in less than 3 ms. Since it uses a simple polynomial-time graph traversal algorithm (), it is the fastest to verify among all policies. The time taken to verify is higher than , because and algorithms are more complex, as they run our path vector protocol to convergence to find paths (and in ’s case the protocol is invoked many times). Finally, and , both use an ILP and, as expected, are slowest to verify. However, they can still be verified in 80 ms per traffic class.

Although and have fewer devices than , they have more nodes and edges in their Tiramisu graphs (§6.1). Hence it takes longer to verify policies on them.

6.3 Comparison with Minesweeper

Next, to put our performance results in perspective, we compare Tiramisu with Minesweeper. In this experiment we use the real datacenter networks. We consider policies , , , and . In Minesweeper, we have to specify the number of failures ; Minesweeper then verifies if the policy holds as long as there are failures. To verify a property under all failure scenarios, we set the value of to number of physical links in the network - 1. Figure 9 (a, b, c, and d) shows the time taken by Tiramisu and Minesweeper to verify these policies. Figure 10 (a, b, c, and d) shows the speedup provided by Tiramisu for each of these policies.

For policies that use ( and ), Tiramisu’s speedup is as high as -. For , the speedup is as high as .

is the only policy where speedup does not increases with network size. This is because larger networks have longer path lengths and more possible candidate paths. Both of these affect the complexity of the algorithm. The number of times invokes increases significantly with network size. Hence the speedup for is relatively less, especially at larger network sizes.

Note that since the number of nodes and edges did not monotonically increase with network size (§6.1), the time to verify these policies does not monotonically increase either.

Next, we compare the performance of Tiramisu and Mines- weeper for the same policies but without failures, e.g. “currently reachable” instead of “always reachable”. Tiramisu verifies these policies by generating the actual path using . Figure 11 (a, b, c, and d) shows the speedup provided by Tiramisu for each of these policies. Even for no failures, Tiramisu significantly outperforms Minesweeper across all policies. Minesweeper has to invoke the SMT solver to find a satisfying solution even in this simple case.

To shed further light on Tiramisu’s benefits w.r.t. Mineswe- eper, we compare the number of variables used by Mineswee- per’s SMT encoding and Tiramisu’s ILP encoding to verify and . In this experiment, we track reduction ratio, which is the number of variables in Minesweeper divided by number of variables in Tiramisu. This is shown in Figure 12. As expected, Tiramisu uses significantly fewer variables. Also, uses more variables than . Hence it has a lower reduction ratio. Tiramisu uses integer variables only for the aspects that matter towards the property in question, where Minesweeper uses binary and integer variables throughout its general encoding, irrespective of the property in question. This is one reason for Minesweeper’s poor performance.

6.4 Scalability

Until now, all our experiments were on small networks. In this section, we evaluate Tiramisu’s performance on large networks from the topology zoo. Figure 13 shows the time taken to verify policies , , , and, . Tiramisu can verify these policies in 0.12 s, even in these large networks.

As expected, verifying () is significantly faster than all other policies, and it is very low across all network sizes. However, for larger networks, time to verify () is as high as

. Again, this due to larger networks having longer and more candidate paths. Large networks also have high diversity in terms of path lengths. Hence, we see more variance in the time to verify

compared to other policies.

For large networks, the time to verify is significantly higher than other policies. This happens because ’s ILP formulation becomes more complex, in terms of number of variables, for such large networks.

7 Related Work

We surveyed various related studies in detail, in earlier sections. Here, we survey others that were not covered earlier.

Aside from Minesweeper and ARC, there are other control plane verification tools that attempt to verify policies against various environments (failures or advertisements). ERA [4] symbolically represents control plane advertisements which it propagates through a network and transforms it based on how routers are configured. ERA is useful to verify reachability against arbitrary external advertisements, but it does not have the full coverage of control plane constructs as Tiramisu or Minesweeper to analyze a range of policies. Bagpipe [16] is similar in spirit to Minesweeper and Tiramisu, but it only applies to a network that only runs BGP. FSR [15] focuses on encoding BGP path preferences.

Batfish [5] and C-BGP [14] are control plane simulators. They analyze the control plane’s path computation as a function of a given environment, e.g., a given failure or an incoming advertisement, by conducting low level message exchanges, emulating convergence, and creating a concrete data plane. Tiramisu also conducts simulations of the control plane; but, for certain policies, Tiramisu can explore multiple paths at once via graph traversal and avoid protocol simulation. For other policies, Tiramisu only simulates a simplified protocol (SPVP) running over a multi-layer multi-attributed graph, and the simulations are conducted in parallel per traffic class.

8 Limitations

One limitation of graph-based control plane models is that they cannot symbolically model advertisements, which is easier to do for SMT-based tools. Tiramisu shares this drawback. This means that Tiramisu cannot exhaustively explore if there exists an external advertisement that could potential lead to a property violation; Tiramisu can only exhaustively explore link failures. Tiramisu would have to be provided a concrete instantiation of an advertisement; in such a case, Tiramisu can analyze the network under the given advertisement and determine if any policies can be violated.

A related issue is that Tiramisu cannot be applied to verify control plane equivalence: Two control planes are equivalent, if the behavior of the control planes (paths computed) is the same under all advertisements and all failure scenarios.

In essence, while Tiramisu can replace Minesweeper for a vast number of policies, it is not a universal replacement. Minesweeper’s SMT-encoding is useful to explore advertisements.

9 Conclusion

While existing graph-based control plane abstractions are fast, they are not as general. SMT-based abstractions are general, but not fast. In this paper, we showed that graphs can be used as the basis for general and fast network verification. Our insight is that, rich, multi-layered graphs, coupled with algorithmic choices that are customized per policy can help achieve the best of both worlds. Our evaluation of a prototype (which we will release open source) shows that we can offer 10-600X better speed than state-of-the-art, scale gracefully with network size, and model all key features found in network configurations in the wild. This work does not raise any ethical issues.


  • [1] Gurobi. http://www.gurobi.com/, 2017.
  • [2] R. Beckett, A. Gupta, R. Mahajan, and D. Walker. A general approach to network configuration verification. In SIGCOMM, 2017.
  • [3] A. El-Hassany, P. Tsankov, L. Vanbever, and M. Vechev. Netcomplete: Practical network-wide configuration synthesis with autocompletion. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18). USENIX Association, 2018.
  • [4] S. K. Fayaz, T. Sharma, A. Fogel, R. Mahajan, T. D. Millstein, V. Sekar, and G. Varghese. Efficient network reachability analysis using a succinct control plane representation. In Symposium on Operating Systems Design and Implementation (OSDI), 2016.
  • [5] A. Fogel, S. Fung, L. Pedrosa, M. Walraed-Sullivan, R. Govindan, R. Mahajan, and T. Millstein. A general approach to network configuration analysis. In Symposium on Networked Systems Design and Implementation (NSDI), 2015.
  • [6] A. Gember-Jacobson, R. Viswanathan, A. Akella, and R. Mahajan. Fast control plane analysis using an abstract representation. In SIGCOMM, 2016.
  • [7] M. Ghaffari, D. R. Karger, and D. Panigrahi. Random contractions and sampling for hypergraph and hedge connectivity. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 1101–1114. SIAM, 2017.
  • [8] T. G. Griffin, F. B. Shepherd, and G. Wilfong. The stable paths problem and interdomain routing. IEEE/ACM Transactions on Networking (ToN), 10(2):232–243, 2002.
  • [9] D. Karger, R. Motwani, and G. D. Ramkumar. On approximating the longest path in a graph. Algorithmica, 18(1):82–98, 1997.
  • [10] P. Kazemian, G. Varghese, and N. McKeown. Header space analysis: Static checking for networks. In Symposium on Networked Systems Design and Implementation (NSDI), 2012.
  • [11] A. Khurshid, X. Zou, W. Zhou, M. Caesar, and P. B. Godfrey. VeriFlow: Verifying network-wide invariants in real time. In Symposium on Networked Systems Design and Implementation (NSDI), 2013.
  • [12] S. Knight, H. X. Nguyen, N. Falkner, R. Bowden, and M. Roughan. The internet topology zoo. IEEE Journal on Selected Areas in Communications, 29(9):1765–1775, 2011.
  • [13] H. Mai, A. Khurshid, R. Agarwal, M. Caesar, P. B. Godfrey, and S. T. King. Debugging the data plane with Anteater. In SIGCOMM, 2011.
  • [14] B. Quoitin and S. Uhlig. Modeling the routing of an autonomous system with c-bgp. IEEE network, 19(6):12–19, 2005.
  • [15] A. Wang, L. Jia, W. Zhou, Y. Ren, B. T. Loo, J. Rexford, V. Nigam, A. Scedrov, and C. Talcott. Fsr: Formal analysis and implementation toolkit for safe interdomain routing. IEEE/ACM Transactions on Networking (ToN), 20(6):1814–1827, 2012.
  • [16] K. Weitz, D. Woos, E. Torlak, M. D. Ernst, A. Krishnamurthy, and Z. Tatlock. Scalable verification of border gateway protocol configurations with an SMT solver. In ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2016.
  • [17] G. G. Xie, J. Zhan, D. A. Maltz, H. Zhang, A. Greenberg, G. Hjalmtysson, and J. Rexford. On static reachability analysis of ip networks. In INFOCOM 2005. 24th Annual Joint Conference of the IEEE Computer and Communications Societies. Proceedings IEEE, volume 3, pages 2170–2183. IEEE, 2005.
  • [18] J. Y. Yen. Finding the k shortest loopless paths in a network. management Science, 17(11):712–716, 1971.

Appendix A Other Policies

Some of the other policies that Tiramisu can verify are listed below:

Always Chain of Waypoints (). Similar to waypointing, we remove nodes associated with each waypoint, one at a time. Using , we check if nodes associated with one of the preceding waypoints in the chain can reach nodes associated with one of the following waypoints in the chain.

Equal Bound (). This policy checks that all paths from to are of the same length. The objective of the ILP in §4.3 can be changed to find the shortest path length. If the longest and shortest path length varies, then this policy is violated.

Always isolated (). Two traffic classes are always isolated if they never traverse the same link. ARC [6] verified this policy by checking if two traffic class-specific graphs have any edge in common or not. Similarly, in Tiramisu, we check if the two traffic class-specific graphs have any hedge in common.

Multipath Consistency (). Multipath consistency is violated when traffic is dropped along one path but blocked by an ACL on another. To support multipaths in , we change the variable to track multiple best advertisements. Using , Tiramisu can identify the number of best paths. We run on graphs with and without removing edges for ACLs. If the number of paths varies with and without ACLs, then the policy is violated.

Always no black holes (). Black holes occur when traffic gets forwarded to a router that does not have a valid forwarding entry. Blackholes are caused by i) ACLs: router advertises routes for traffic that are blocked by their ACLs; ii) static routes: the next-hop router of a static route cannot reach . Tiramisu uses to check these conditions. For (i) Tiramisu first creates the graph without removing edges for ACLs. Let be the router with a blocking ACL. If can reach router and can reach (using ), then traffic will reach router under some failure, and then get dropped because of the ACL. For (ii) if can reach the router with static route and the next-hop router cannot reach , then the traffic gets dropped.

Appendix B Proofs

b.1 Prohibited Paths

Any path with 2 consecutive untainted nodes is a prohibited path

Theorem 1

The approach specified in §3.1 correctly identifies all prohibited path.

Proof: We will prove this by contradiction.

In a router, each routing process will have its own Routing Information Base (RIB) that has routes learned by that routing process. The routes of all RIBs are also inserted in the Forwarding Information Base (FIB) of that router. Hence, the “f” edge exists between all routing process node and the fib node for each router.

The progression of taints is modelled based on graph of [17]. In , edge adjacency exists only for RIB adjacency. Similarly taints are spread only across RIB adjacent processes. Similar to [17], taints cross intra-device routing processes only with redistribution.

A prohibited path can exist only if you traverse a router that does not have a valid fib entry. Assume such a path exists and is not identified by Tiramisu’s tainting strategy. Such paths have to satisfy the following three criterias: (i) there has to be a router where path goes from tainted node to untainted node, i.e. from node representing routing process or FIB (e.g. , in Figure 3) that has forwarding entry, to node representing routing process that does not have forwarding entry (); (ii) node has to be followed by a node () representing another router with the same routing process; and (iii) node has to be succeeded by node which will either be a node representing node ’s router’s FIB () or another router with the same routing process as Y (

). Here both these nodes have to be untainted. However, this ends up with a path with three untainted nodes, which are classified by Tiramisu as prohibited. This contradicts the assumption that this path is not identified by Tiramisu.

b.2 Tpvp

Theorem 2

Assuming the network control plane converges, always finds the exact path taken in the network under any given failure scenario.

Proof: We leverage correctness of Minesweeper and SPVP to establish the correctness of . In SPVP, there is no restriction on order in which messages are processed by different routers. As long as there exists even one stable path, SPVP will find it irrespective of the order of message processing. In (line 5), we process messages in a fixed round-robin order. Since any ordering of messages in SPVP leads to a valid solution, a fixed ordering of messages should also lead to a valid solution.

The body of the loop code of is equivalent to SPVP. The updateCost function is a function that was not mentioned in the original SPVP algorithm. In the original SPVP algorithm, rib-in (u v) represented paths received from neighbor v. And the properties of those paths e.g. local-pref, ospf-weight etc were assigned based on import and export policies. Minesweeper also modelled rib-in (u v) as import and export policy constraints. Tiramisu’s updateCost function also updates path properties based on import and export policy constraints, similar to Minesweeper [2] and SPVP [8].

b.3 Tdfs

Theorem 3

TDFS traverses all real network paths that could materialize under some failure scenario, and does not traverse any path that cannot materialize under any failure scenario.

Proof: Let path be the path traversed by a particular src-dst pair in the actual network under some failure. Now assume this path is not traversed by because it does not exist in the graph. This contradicts Theorem 2, which showed that always finds the exact path used in the actual network, which implies the path must exist in the graph. Thus, will traverse all network paths that could materialize under some failure scenario.

Now assume there exists some path in the actual network that also exists in the graph. Also assume it is not traversed by . This means this path contains three or more consecutive tainted nodes. This contradicts Theorem 1 as it cannot be a real network path. Thus, does not traverse any path that cannot materialize under any failure scenario.

Appendix C Tiramisu Min-cut

Table 3 lists the boolean indicator variables and functions used in encoding of the ILP for the property in §4.2. We now provide the full ILP with detailed constraints; we repeat the description of the constraints to ensure ease of reading. Recall, edges in our graphs represented the flow of traffic from to . For ease of understanding, in specifying the ILP, we reverse the edges to represent flow of advertisement from to .

Name Description


set as 1 if edge fails
set as 1 if advertisement propagates on edge
set as 1 if advertisement reaches node
set as 1 if edge carries blocked community
set as 1 if edge only propagates advertisement that create prohibited paths
set as 1 if node forwards community


returns all nodes of graph
returns all edges of graph
returns all inter-device edges of graph
returns all physical edges of graphs
returns all edges of graph with untaint start and end nodes
returns all nodes except and
returns incoming edges of node
returns outgoing edges of node
returns incoming edges from untainted neighbors of node
returns start nodes of all incoming edges of node
returns communities added by node
returns communities removed by node
returns communities blocked on node
returns communities []
returns start node of edge
returns end node of edge
Table 3: Variables and Functions

ILP The objective of the ILP is to minimize the number of physical link failures required to disconnect the from . All edges belonging to hedge will share the same variable.


Advertisement Constraints We first discuss the constraints added to represent reachability of advertisements. The base constraints state that the originates the advertisement. To disconnect from , the advertisement must not reach the .


For other nodes, we add the constraint that advertisement reaches if it gets propagated () on any of its incoming edge .


Note that logical AND, OR operators can be represented as constraints in ILP.

Next, we discuss constraints on propagation of advertisements. An advertisement can propagate through an edge , if it reaches the start node () of the edge (), the physical edge does not fail (), the advertisement does not carry a community () that is blocked on this node, and if edge carries atleast one advertisement that does not create a prohibited path (). This is represented as


Community Constraints The base constraints state that each nodes that adds the community forwards that community and each nodes that removes that community does not forward it.


For other nodes, we add the constraint that node forwards a community iff any of its inbound neighbors () forwards that community to node .


Finally, we add the constraint that an edge carries a blocked community iff the start node of edge forwards any community that is blocked by end node of edge .


Prohibited Path Constraints An edge , that only propagates advertisements that create prohibited paths, is an edge that satisfies the condition that (Eqn 12) the start and end nodes of edge are untainted (). Furthermore the start node receives advertisement only on edges from untainted neighbors () and not on edges from any other neighbor (). Such edges always create subpaths with three consecutive untainted nodes.


Appendix D Tiramisu Longest Path

We now specify the ILP corresponding to §4.3 in detail. For ease of reading, we also provide the full description of the objective and the constraints.

Recall that to verify , we propose another ILP whose objective is to maximize the number of inter device edges () traversed by an advertisement (). Note again that, the path traversed by the advertisement is the opposite of traffic flow. Our objective is as follows:


Single Path Constraints To ensure that only one path gets advertised, and that the sends the advertisement and the receives the advertisement, we add the constraints in


For other nodes, we add the flow conservation property, i.e. sum of incoming flows is equal to outgoing flows


Advertisement Constraints Next, we add constraints on propagation of advertisements. An advertisement can be bloc- ked on edge if it satisfies the community () and path prohibition () constraints. These are similar to Eqn 11 and Eqn 12.