There are various concepts to secure traffic transmission against failure of path components such as links or nodes. The fastest is 1+1 protection. A sender duplicates traffic and forwards it over disjoint paths while the receiver forwards only the first copy received for every packet. In case of a failure, any packet loss can be avoided, which makes 1+1 protection attractive for highly reliable applications. 1+1 protection is implemented in optical networks to protect an entire trunk. It is also available for MPLS  and Ethernet , which are carrier technologies for IP and introduce signaling complexity. In this paper, we leverage the P4 programming language  to provide 1+1 protection for IP networks. We program P4 switches such that they feature IP forwarding, the sending and receiving node behaviour of 1+1 protection which includes IP encapsulation and decapsulation. We call this approach P4-Protect. Targets of our implementation are the software switch BMv2 and the hardware switch Tofino Edgecore Wedge 100BF-32X. A particular challenge is the selection of the fist copy of every duplicated packet at the receiver. We provide a controller that allows to set up 1+1 protection between P4 nodes implementing P4-Protect. Furthermore, protected flows can be added using a fine-granular description based on various header fields. We evaluate the performance of P4-Protect on the hardware switch. We show that P4-Protect can be used with only marginal throughput degradation and we illustrate that P4-Protect can significantly reduce jitter when both paths have similar delays.
The paper is structured as follows. Section II gives an overview of related work. Section III describes the 1+1 protection mechanism used for our implementation and extensions for its use on the general Internet. Section IV presents a P4-based implementation including specifics for the Tofino Edgecore Wedge 100BF-32X. We evaluate the performance of P4-Protect on the hardware switch in Section V and conclude the paper in Section VI.
Ii Related work
We review various resilience concepts for communication networks. Afterwards, we give examples for 1+1 protection.
Rerouting reorganizes the traffic forwarding to avoid failed components. This happens on a time scale of a second. Fast reroute (FRR) locally detects that a next hop is unreachable and deviates traffic to an alternative next hop . The detection may take a few 10s of milliseconds so that traffic loss cannot be avoided. Both rerouting and FRR do not utilize backup resources under failure-free conditions, but their reaction time suffers from failure detection delay. 1:1 protection leverages a primary/backup path concept. To switch over, the head-end node of the paths needs to be informed about a failure, which imposes additional delay. With restoration, recovery paths may be dynamically allocated so that even more time is needed to establish the restoration paths [5, p. 31]. 1+1 protection duplicates traffic and sends it over two disjoint paths whereby the receiving node needs to eliminate duplicates. That method is fastest, but it requires extra capacities also under failure-free conditions. Some services can afford short network downtimes, other services greatly benefit from 1+1 protection’s high reliability.
The surveys , , and  provide an overview of various protection and restoration schemes. The authors of  discuss survivability techniques for non-WDM networks like automatic protection switching (APS) and self healing rings (SHR) as well as dynamic restoration schemes in SONET. They further describe protection methods for optical WDM networks. A comprehensive overview of protection and restoration mechanisms for optical, SONET/SDH, IP, and MPLS networks can be found in .
SDN with inband signalling increases the need for fast and local protection against failures because the controller may no longer be reachable in case of a failure or highly loaded. In addition, with SDN new protection mechanisms can be implemented, e.g., to reduce state in the network. Examples are given in .
Ii-B 1+1 Protection
The ITU-T specification Y.1703  defines an 1+1 path protection scheme for MPLS. They add sequence numbers to packets and replicate them on disjoint paths. At the end of the paths, duplicate packets are identified by the attached sequence number and eliminated. P4-Protect works similarly. However, it does not require MPLS. It is compatible with IP, and works over the Internet.
The authors of  compare several implementation strategies of 1+1 protection, i.e, traditional 1+1 path protection, network redundancy 1+1 path protection (diversity coding) , and network-coded 1+1 path protection. Their analytical results show that diversity coding and network coding can be more cost-efficient, i.e., they require about 5-20% less reserved bandwidth. The delay impact of 1+1 path protection in MPLS networks has been investigated in . McGettrick et. al  consider 10 Gb/s symmetric LR-PON. They reveal switch-over times to a backup OLT of less than 4 ms. Multicast traffic has often realtime requirements. Mohandespour et. al extend the idea of unicast 1+1 protection to protect multicast connections 
. They formulate the problem of minimum cost multicast 1+1 protection as a 2-connectivity problem and propose heuristics. Braun et. al propose maximally redundant trees for 1+1 protection in BIER, a stateless multicast transport mechanism. It leverages the concept of multicast-only FRR .
Iii P4-Protect: Concept
We first give an overview of P4-Protect. We present its protection header, the protection connection context, and the operation of the protection tunnel ingress and egress nodes (PTIs/PTEs).
With P4-Protect, a protection connection is established between two P4 switches. Protected traffic is duplicated by a protection tunnel ingress (PTI) node and simultaneously carried through two protection tunnels to a protection tunnel egress (PTE) node. The PTE receives the duplicated traffic and forwards the first copy received for every packet.
Figure 1 illustrates the protocol stack used with P4-Protect. The PTI adds to each packet received for a protected flow a protection header (P) that contains a sequence number which is incremented for each protected packet. The packet is equipped with an additional IP header (IP-PTE) with the PTE’s IP address as destination. The PTI duplicates that packet and forwards the two copies over different paths. The paths may be different due to traffic engineering (TE) capabilities of the network or path diversity may be achieved through an additional intermediate hop. When the PTE receives a packet, it removes its outer IP header (IP-PTE). If the sequence number in the protection header is larger than the last sequence number received for this connection, it removes the protection header and forwards the packet; otherwise, the packet is dropped. The latter is needed as duplicate packets are also considered harmful.
Iii-B Protection Header
The protection header contains a 24 bit connection identifier (CID), a 32 bit sequence number (SN) field, and an 8 bit next protocol field. The CID is used to uniquely identify a protection connection at the PTE. The sequence number is used at the PTE to identify duplicates. The next protocol field facilitates the parsing of the next header. We reuse the IP protocol numbers for this purpose.
Iii-C Protection Connection Context
A protection connection is set up between a PTI and PTE. Their IP addresses are associated with this connection, including two interfaces over which duplicate packets are forwarded. For each connection, the PTI has a sequence number counter which is incremented for each packet forwarded over the respective protection connection. Likewise, the PTE has a variable which records the highest sequence number received for the respective protection connection. A CID is used to identify a connection at the PTE. A PTI may have several protection connections with the same CID but different PTEs (see Section IV-E1).
Iii-D PTI Operation
The PTI has a set of flow descriptors that are mapped to protection connections. If the PTI receives a packet which is matched by a specific flow descriptor, the PTI processes the packet using the corresponding protection connection. That is, it increments the , adds a protection header with CID, next protocol set to IPv4, and the set to . Then, an IP header is added using the PTI’s IP address as source and the IP address of the PTE associated with the protection connection as destination. The packet is duplicated and forwarded over the two paths associated with the protection connection.
Iii-E PTE Operation
During failure-free operation, the PTE receives duplicate packets via two protection tunnels. When the PTE receives a packet, it decapsulates the outer IP header. It uses the CID in the protection header to identify the protection connection and the corresponding . If the in the protection header is larger than , is updated by , the protection header is decapsulated, and the original packet is forwarded; otherwise, the packet is dropped.
The presented behavior works for unlimited sequence numbers. The limited size of the sequence number space makes the acceptance decision for a packet more complex. Then, a larger than may indicate a copy of a new packet, but it may also result from a very old packet. To solve this problem, we adopt the use of an acceptance window as proposed in . The window is sequence numbers large. Let be the maximum sequence number. If holds, a new sequence number is accepted if the following inequality holds:
If holds, a new sequence number is accepted if one of the two following inequalities holds:
This allows a packet copy to arrive sequence numbers later than the corresponding first packet copy without being recognized as new packets.
In this section we present the implementation of P4-Protect. We describe the supported header stacks, explain the control blocks, their organization in ingress and egress control flow, and we reveal implementation details about some control blocks. Finally, we sketch most relevant aspects of the P4-Protect controller.
Iv-a Supported Header Stacks
Incoming packets are parsed so that their header values can be accessed within the P4 pipeline. To that end, we define the following supported header stacks. Unprotected IP traffic has the structure IP/TP, i.e., IP header and some transport header (TCP/UDP), and protected IP traffic has the structure IP/P/IP/TP, i.e., the IP header with the PTE’s address, the protection header, the original IP header, and a transport header. IP traffic without transport header is parsed only up to the IP header.
Iv-B Control Blocks
We present three control blocks of our implementation of P4-Protect. They consider the packet processing by PTI and PTE.
Iv-B1 Control Block: Protect&Forward
When the PTI receives an IP packet, it is parsed and matched against the Match+action (MAT) table ProtectedFlows. In case of a match, the packet is equipped with an appropriate header stack, duplicated, and sent to appropriate egress ports. In case of a miss, the packet is processed by a standard IPv4 forwarding procedure.
Iv-B2 Control Block: Decaps-IP
When the PTE receives an IP packet with the PTE’s own IP address, the IP header is decapsulated. If the next protocol indicates a protection header, the packet is handed over to the Decaps-P control block; otherwise, the packet is processed by the Protect&Forward control block since the resulting packet may need to be protected and forwarded.
Iv-B3 Control Block: Decaps-P
In the Decaps-P control block, the PTE examines the protection header and decides whether to keep or drop the packet as it is a copy of an earlier received packet. To keep the packet, the protection header is decapsulated.
Iv-C Ingress and Egress Control Flow
The interdependencies between the control blocks suggest the following ingress control flow: Decaps-IP, Decaps-P, Protect&Forward. At a mere PTI, no action is performed by the Decaps-IP and Decap-P control block. The Protect&Forward takes care that protected traffic is duplicated and sent over two different paths and that unprotected traffic is forwarded by normal IPv4 operation. At a mere PTE, protected traffic is decapsulated and selected before being forwarded by normal IPv4 operation. Unprotected traffic is just forwarded by normal IPv4 operation.
Iv-D Control Block Implementations
In the following, we explain implementation details of the Protect&Forward control block and the Decaps-P control block. We omit the Decaps-IP control block as it is rather simple.
Iv-D1 Protect&Forward Control Block
The operation of the Protect&Forward control block is illustrated in Figure 2. It utilizes the MAT ProtectedFlows to process all packets. It effects that protected traffic is encapsulated at the PTI with a protection header and an IP header for tunneling.
The MAT ProtectedFlows uses a ternary match on the classic 5-tuple description of a flow: the source and destination IP address and port as well as the protocol field. In case of a match, the MAT maps a packet to a specifc protection connection and calls the protect action with the connection-specific parameters , , , , and . The protect action increments the register where is a connection-specific index to access a register containing the last sequence number. On the Tofino target, this is performed by a separate register action. The protect action further pushes a protection header on the packet including , i.e., the connection-specific CID, , and the next protocol set to IPv4. Then, it pushes an IPv4 header with the IP address of the PTI as source IP and the IP address of the PTE as destination IP. The protocol field of this outer IP header is set to P4-Protect. Finally, the multicast group of the packet is set to . It is a connection-specific multicast group. It effects that the packet is duplicated and sent to two egress ports in order to deliver it via two protection tunnels to the PTE. In case of a miss, the packet is unprotected and handled by a standard IPv4 forwarding procedure, which is not further explained in this paper.
Iv-D2 Decaps-P Control Block
The Decaps-P control block decides whether a packet is new and should be forwarded or dropped. It compares the sequence number of the packet’s protection header with the last sequence number of the corresponding protection connection. The latter can be accessed by the register where CID is given in the protection header. The acceptance is decided based on Equation (1) or Equation (3) depending on the value of and where is given as a constant.
As the check is rather complex, it requires careful implementation for the Tofino target 111Tofino is a high-performance chip which operates at 100 Gb/s so that only a limited set of operations can be performed for each packet, in particular in connection with register access.. It leverages the fact that we set . Furthermore, it requires a reformulation of Equation (1) and Equation (3).
If holds, the following two inequalities must be met:
Otherwise, if , it is sufficient that only one of the following two inequalities holds:
Both cases are implemented as separate register actions on the Tofino target. With 32 bit sequence numbers, a minimum packet size of 40 bytes and a transmission speed of Tb/s, a delay difference up to 1.6s can be compensated.
Iv-E Controller for P4-Protect
P4-Protect’s controller offers an interface for the management of protection connections and protected flows. It configures in particular the MAT ProtectedFlows but also other MATs needed for standard IPv4 forwarding or IP decapsulation. In the following, we explain the configuration of protection connections and protected flows.
Iv-E1 Configuration of Protection Connections
A protection connection is established by choosing registers on PTI and PTE to record the last sequence numbers and of a protection connection. The connection identifier is the PTE’s index to access . On the PTI, a different index may be chosen to access . Furthermore, the registers are initialized with zero. Moreover, the controller sets up a multicast group for each connection so that its traffic will be replicated in an efficient way to the two desired interfaces.
Iv-E2 Configuration of Protected Flows
A protected flow is established by adding a new flow rule in the MAT ProtectedFlows of the PTI. It contains an appropriate flow descriptor and the parameters to call the action protect. Those are the index associated with the corresponding protection connection, the CID needed at the PTE to identify the protection connection, the IP address of the PTI, the IP of the PTE, and the multicast group .
In this section we evaluate the performance of the implemented mechanism on the Tofino Edgecore Wedge 100BF-32X. First, we compare packet processing times with and without P4-Protect. Then, we demonstrate that very high data rates can be achieved with and without P4-Protect on a 100 Gb/s interface. Finally, we show that P4-Protect can provide a transmission service with reduced jitter compared to the jitter of both protection tunnels.
V-a Packet Processing Time
P4-Protect induces forwarding complexity. To evaluate its impact, we leverage P4 metadata to calculate the time a packet takes from the beginning of the ingress pipeline to the beginning of the egress pipeline. This is sufficient for a comparison as all work for P4-Protect is done in the ingress pipeline and all considered forwarding schemes utilizes the same egress pipeline. We compare three forwarding modes: a plain IP forwarding implementation (plain), P4-Protect for unprotected traffic (unprotected), and P4-Protect for protected traffic (protected).
Table I shows the ingress-to-egress packet processing time on both PTI and PTE for the three mentioned forwarding modes. The duration is given relative to the processing time for plain forwarding mode. We observe the lowest processing time at PTI and PTE for plain forwarding as it has the least complex pipeline. With P4-Protect, the processing time at both PTI and PTE is larger than with plain forwarding as the operations are more complex. At PTI, the processing time is even larger with protected forwarding (166%) than with unprotected forwarding (127%). At PTE, the processing times for protected and unprotected traffic are equal and 27% longer than with plain forwarding.
In our implementations, we have used only a minimal IPv4 stack for all three forwarding modes. With a more comprehensive IPv4 stack, the relative overhead through P4-Protect is likely to be smaller.
V-B TCP Goodput
We set up iperf3 connections between client/server pairs and measure their goodput. Each iperf3 connection consists of 15 parallel TCP flows. Two switches are bidirectionally connected via two 100 Gb/s interfaces. Four client/server pairs are connected to the switches via 100 Gb/s interfaces. Up to 4 clients download traffic from their servers via the trunk between the switches.
shows the overall goodput for a various number of client/server pairs, each transmitting traffic over a single TCP connection. The goodput is given for the forwarding modes plain, unprotected, and protected. We performed 20 runs per experiment and provide the 95% confidence interval.
A single, two, and, three TCP connections cannot generate sufficient traffic to fill the 100 Gb/s bottleneck link. However, with four TCP connections a goodput of around 90 Gb/s is achieved. This is less than 100% because of overhead due to Ethernet, IP, and TCP headers and due to the inability of TCP to efficiently utilize available capacity at high data rates. Most important is the observation that all three forwarding modes lead to almost identical goodput. The goodput for protected and unprotected forwarding is slightly lower than plain forwarding, which is apparently due to the operational overhead of P4-Protect.
V-C Impact on Jitter
We examine the effect of 1+1 path protection on jitter. Two hosts are connected to two Tofino Edgecore Wedges 100BF-32X. The switches are connected with each other via two paths with intermediate Linux servers. Their interfaces are bridged and cause an artificial, adjustable, uniformly distributed jitter. We leverage thetc tool for this purpose . All lines have a capacity of 100 Gb/s.
In our experiment, we send pings between the two hosts with and without P4-Protect. Figure 4 reports the average round trip time (RTT) deviation for the pings. Unprotected traffic suffers from all the jitter induced on a single path. Protected traffic suffers only from about half the jitter. This is because P4-Protect forwards the earliest received packet copy and minimizes packet delay occurred on both links.
In this paper we proposed P4-Protect for 1+1 path protection with P4. It may be utilized to protect traffic via two largely disjoint paths. We presented an implementation for the software switch BMv2 and the hardware switch Tofino Edgecore Wedge 100Bf-32X. The evaluation of P4-Protect on the hardware switch revealed that P4-Protect increases packet processing times only little, that high throughput can be achieved with P4-Protect, and that jitter is reduced by P4-Protect when traffic is carried over two path with similar delay but large jitter.
-  ITU, “ITU-T Recommendation G.7712/Y.1703 (2010), Internet protocol aspects – Operation, administration and maintenance,” ITU, Sep. 2010.
-  ——, “ITU-T Recommendation G.803/Y.1342 (2006), Ethernet Protection Switching ,” ITU, Jun. 2006.
-  P. Bosshart et al., “P4: Programming Protocol-Independent Packet Processors,” ACM CCR, vol. 44(3), 2014.
-  A. Atlas et al., “RFC5286: Basic Specification for IP Fast Reroute: Loop-Free Alternates ,” Sep. 2008.
-  J. P. Vasseur et al., Network Recovery. Morgan Kaufmann, 2004.
-  C. Metz, “IP protection and restoration,” IEEE Internet Computing, vol. 4(2), 2000.
-  D. Zhou et al., “Survivability in Optical Networks,” IEEE Network Magazine, vol. 14(6), 2000.
-  A. Fumagalli et al., “IP restoration vs. WDM protection: is there an optimal choice?” IEEE Network Magazine, vol. 14(6), 2000.
-  D. Merling et al., “Efficient Data Plane Protection for SDN.” IEEE (NetSoft), 2018.
-  H. Øverby et. al, “Cost comparison of 1+1 path protection schemes: A case for coding,” in IEEE ICC, 2012.
-  E. Ayanoglu et al., “Diversity coding for transparent self-healing and fault-tolerant communication networks,” IEEE ToC, vol. 41(11), 1993.
-  G. Niculescu et al., “The Packet Delay in a MPLS Network Using ”1+1 Protection,” in IEEE Advanced International Conference on Telecommunications, 2010.
-  S. McGettrick et al., “Ultra-fast 1+1 protection in 10 Gb/s symmetric Long Reach PON,” in IEEE ECOC), 2013.
-  M. Mohandespour et al., “Multicast 1+1 protection: The case for simple network coding,” in IEEE ICNC, 2015.
-  W. Braun et al., “Performance Comparison of Resilience Mechanisms for Stateless Multicast Using BIER,” in IFIP/IEEE, 2017.
-  A. Karan et al., “RFC7431: Multicast-Only Fast Reroute,” Aug. 2015.
-  L. Foundation, “Linux Traffic Control,” 2019.