Hybrid SDN Evolution: A Comprehensive Survey of the State-of-the-Art

03/30/2021 ∙ by Sajad Khorsandroo, et al. ∙ 0

Software-Defined Networking (SDN) is an evolutionary networking paradigm which has been adopted by large network and cloud providers, among which are Tech Giants. However, embracing a new and futuristic paradigm as an alternative to well-established and mature legacy networking paradigm requires a lot of time along with considerable financial resources and technical expertise. Consequently, many enterprises can not afford it. A compromise solution then is a hybrid networking environment (a.k.a. Hybrid SDN (hSDN)) in which SDN functionalities are leveraged while existing traditional network infrastructures are acknowledged. Recently, hSDN has been seen as a viable networking solution for a diverse range of businesses and organizations. Accordingly, the body of literature on hSDN research has improved remarkably. On this account, we present this paper as a comprehensive state-of-the-art survey which expands upon hSDN from many different perspectives.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

page 5

page 7

page 8

page 11

page 12

page 26

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Computer networks are typically built using different devices such as switches, routers, firewalls, and load balancers which communicate through various standard protocols. Network administrators are responsible for setting appropriate policies and managing all network devices in order to respond to a wide range of network events. Usually, these challenging tasks are performed manually with a rather limited number of tools available. Consequently, network management and configuration along with tuning network performance are quite tedious and potentially error-prone tasks. ”Internet ossification” [1] is another serious challenge network operators are faced with. Internet is considered as one of the critical infrastructures in today’s world and it has a huge and very complex deployment base. Therefore, it is extremely difficult, or sometimes impossible, for the Internet to be updated in terms of underlying protocols as well as physical infrastructures. Network programmability [2] is a proposed solution to tackle these challenges and help the Internet be updated based on the emerging applications and services. Particularly, Software-Defined Networking (SDN) [3] is an evolutionary networking paradigm in which control plane has been decoupled from data plane. This leads to considerably simplified network configuration and management along with enhanced flexibility and agility. The main idea behind SDN is to allow a logically centralized software-based controller (i.e. control plane) takes care of network intelligence and decision making, while data plane is responsible for traffic forwarding tasks. Such tasks can then be programmed through either an open standard interface/protocol (such as OpenFlow (OF) [4], or ForCES [5]) or a domain-specific language (such as Programming Protocol-Independent Packet Processors (P4) [6]). Consequently, major network industry parties have set up the Open Networking Foundation (ONF) [7] to promote SDN and to standardize the OF protocol. This has caused an intense adoption of SDN in almost every field of networking, from Data Center (DC) [8, 9, 10] and cloud networks [11, 12] to Wide Area Network (WAN) [13, 14, 15], wireless [16, 17] and recently 5G [18, 19, 20]. Hence, both industry and academia are spending considerable amount of time and money to embrace SDN as a prevailing networking paradigm. Accordingly, SDN market is expected to witness a substantial growth from USD 8.8 billion in 2018 to USD 28.9 billion by 2023, at a Compound Annual Growth Rate (CAGR) of 26.8% during this period [21].

Although SDN is a fast growing networking paradigm, the traditional IP networks (a.k.a legacy networks) are still widely in place for good reasons. Despite the fact that initial SDN models assumes that a logically centralized controller manages the network, larger and more sophisticated SDN deployments need several controllers collaborating in a distributed manner. However, control plane scalability, resiliency, and fault tolerance are still under active investigations. Besides, distributed SDN solutions such as controller locality [22], controller placement [23], and network state consistency [24] still largely rely on the solutions existed in the legacy networking domain.
SDN security is another issue which needs to be addressed. Although SDN market is flourishing and Tech Giants such as Google [13] embraced it as the early adopters, security is still a major challenge for small to medium-size enterprises. While SDN is getting mature and being widely deployed, it definitely becomes an attractive target too. In addition, SDN

introduces new attack vectors which did not exist in traditional networks

[25, 26].
Complexity is another concern. Although SDN is an absolutely competitive advantage for Big Techs [27, 28], it can impose unnecessary burden on technical staffs in terms of operational complexities. These complexities possibly arise in implementation, deployment, and even administration of the networks [29]. Top-tier providers, however, benefit from vast technical resources to tackle these difficulties while their smaller rivals simply cannot.
Strictly speaking, not all the possible challenges in adoption of SDN are technical. There are indeed business and financial risks as well. When existing legacy network infrastructures work smoothly, organizations are usually reluctant to pay for new equipment and retrain their technical staffs. Neither are there well-studied, production-level, uniform solutions present to support network migration process incrementally. The situation is exacerbated by potential risk of service interruptions and Service-Level Agreement (SLA) violations during the transition from legacy to SDN.
Despite the fact that transition from legacy network to SDN can be associated with some potential challenges (e.g. financial barriers, possible technical difficulties, and lack of standards for a seamless migration, to name a few), SDN still has obvious advantages which cannot be overlooked. Therefore, a hybrid deployment (a.k.a Hybrid SDN (hSDN)) [30, 31], where SDN and traditional network nodes coexist, can be a compromise solution. To get the most out of a hybrid networking solution where heterogeneous network equipment, protocols, and paradigms interact, seamless coordination between logically centralized SDN control plane and legacy distributed Routing Information Base (RIB) should be guaranteed [32].

Fig. 1: An overview of discussed hSDN topics

In this paper, we comprehensively survey state-of-the-art hSDN. Comprehensively reviewing a large body of related literature, we investigated hSDN from multiple perspectives as illustrated in Figure 1. More specifically, Section 2 investigates the existing hSDN models and organized them in control and data planes with great details. Security and privacy of hSDN are addressed in Section 3. We discuss potential vulnerabilities and possible threats in a hybrid network environment while we also study how they can be detected and mitigated. This section concludes the discussion with a detailed review of existing security frameworks and solutions in the field. Section 4 explores network management in terms of network data collection, network automation, and network updates. It also investigates existing literature in regards to hSDN reliability, resiliency, fault tolerance and load balancing. Traffic engineering with reference to traffic management and measurement along with Quality of Service (QoS) and QoS routing schemes in a hybrid environment are presented in Section 5. We elaborate on the implementation and deployment of hSDN solutions in the context of emergent networks environments in Section 6. We specifically focus on the fifth generation cellular networks (a.k.a 5G), cloud and data center networks, Internet of Things (IoT), Blockchain, Software Defined WAN (SD-WAN) and Software Defined Branch (SD-Branch) use cases. Section 7 provides a run-through of existing simulation tools and testbeds while Section 8 overviews existing hSDN solutions from business and standardization groups perspective. We explore hSDN open issues and potential future research directions in Section 9 while Section 10 concludes the paper.

2 Hybrid SDN Architecture

The benefits of SDN are clear: (i) network programmability, fostering network automation, (ii) network management reducing operational costs by simplifying the management tasks and (iii) network virtualization. These aspects are spurring network operators and enterprises to upgrade their networks with SDN-enabled switches and servers. However, introducing SDN technologies in a legacy network requires capital and operational investments and might raise security and reliability concerns. The latter aspects, discussed in this paper, are not only due to the SDN technology itself, but they are also correlated to the ability of the network architects and administrators to define the deployment strategy that best suits the network scenario, and to ensure a smooth coexistence between legacy and SDN devices.

As also discussed in other works [33, 34, 35, 36], there exists a variety of deployment strategies for hSDN networks. The choice on which strategy to adopt depends on many factors, such as type of the network (either enterprise, telecom operator, data center, etc.), type of network services offered to the users, required performance, capital and operational budgets. In this Section we briefly review the hSDN architecture and then we discuss a range of different strategies for hSDN deployment.

2.1 hSDN deployment strategies

Here we introduce the hSDN concept starting from the classic plane-oriented view of the SDN architecture as depicted in Figure 2. Bottom-up, the SDN data plane, also known as forwarding data plane, is built with SDN-enabled devices (orange boxes in the figure) either virtual or physical controlled by the SDN controller via an open vendor-agnostic South-Bound Interface (SBI). A popular common SBI is OpenFlow (OF) [37], with other well-known candidate protocols that will be mentioned in this paper. The control plane uses the SBI to program the forwarding behaviour of the data plane. A hSDN network comprises traditional networking, in which the traffic is forwarded based on the decisions of distributed routing/switching mechanisms, as represented in the figure with green elements.

The SDN control plane consists of a centralized software controller that translates the requirements of the applications down to the data plane nodes and gives relevant information up to the SDN applications. It supports the network control logic and provides the application layer with an abstracted view of the SDN-enabled network resources. The control plane is logically centralized and usually implemented as a physically distributed system for scalability and reliability purposes (see Section 2.3.9 for more details). A set of common controllers are reported in Figure 2.

The North-Bound Interface (NBI) of the SDN controller allows SDN applications to send configuration commands to the SDN devices and to retrieve information about network topology and data plane state. Unfortunately, the NBI

is not yet standardized and a variety of different interfaces are implemented across open-source and commercial

SDN controllers. This limits the portability of SDN applications from one controller to another. Some examples of such interfaces are shown in the NBI block of the figure.

Fig. 2: hSDN architecture

As represented in Figure 2, in the hSDN networking architecture, both legacy and SDN devices coexist. While in the latter the Forwarding Information Base (FIB) is populated by the SDN controller by translating the high-level routing policies issued by the SDN applications, in the former the task is accomplished by distributed routing protocols such as Open Shortest Path First (OSPF), Border Gateway Protocol (BGP) and Intermediate System to Intermediate System (IS-IS). In both cases, the FIB, also known as forwarding table, is used by the routing or bridging function to identify the proper output port of the device for each packet. The forwarding tables are implemented using Content-Addressable Memorys or, in the case of SDN-enabled switches, using more complex and expensive Ternary Content-Addressable Memorys. TCAMs can store three bit states (0, 1, and “don’t care“) and work very well in conjunction with OpenFlow, where the flow table entries foresee a wildcard bit, used to inform the switch to ignore the value of the specified header field. As SDN enables a much finer traffic control compared to traditional L2 and L3 forwarding, SDN forwarding rules requires more memory space (up to 773 bits with recent OF versions [37], compared to 60 bit of a L2 forwarding entry). This might limit the application space of SDN when hardware enhancements are not available or feasible due to the cost and power consumption of TCAM components.

The issue related to the capacity of SDN forwarding tables has been already investigated and tackled by leveraging the properties of the SDN paradigm [38, 39, 40]. In this regard, hSDN can be seen not only as a transitional/testing deployment strategy towards a full adoption of the SDN technologies, but also as an opportunity to maximise the potential of the SDN technologies, thus overcoming the memory limitations through a coordinated mixture of SDN and traditional networking. All these aspects are highlighted in previous works [34, 33, 35, 36] where a variety of hSDN deployment strategies are discussed:

  1. [label=()]

  2. Island-based. The network is topologically divided in areas, also called islands, consisting of either legacy devices or SDN nodes (see Figure 3(a)). This deployment model is often adopted to gradually introduce and test the SDN technology in confined regions, with limited impact on existing network operations. An example of the island-based model is B4 [13], a Software-Defined WAN (SD-WAN) implemented for connecting Google sites (see Section 6.4.2 for a detailed view). Like in other deployments, the backbone network is managed with SDN, where coarse-grained network policies are required. Whereas the traffic in the DCs is routed using legacy protocols and long fine-grained forwarding tables. Figure 3(a) shows an example of the island-base model composed of three islands. In the figure, the orange color is used to represent SDN elements (nodes, FIB, controller and networks), while the green color represents legacy elements (nodes, FIB and networks).

  3. Service-Based. This deployment model follows the requirements of the services to be provisioned onto the network. While advanced services are usually implemented using SDN technologies (e.g., Load Balancing (LB), Access Control Lists, etc.), others are typically managed through traditional techniques ( Virtual Private Network (VPN), Multi-protocol Label Switching (MPLS), IPSec, etc.). As depicted in Figure 3(b), the service-based strategy might use hybrid nodes, where both distributed and the SDN control planes are supported. This type of nodes are obtained by adding SDN capabilities to legacy routers or switches through executing software agents, upgrading the hardware components or installing a software SDN switch implementation (such as Open Virtual Switch (OvS) [41]). In hybrid nodes, the forwarding tables are managed by both legacy and SDN paradigms and can combine TCAM with standard memory (SRAM/DRAM).

  4. Class-based. This model is based on partitioning the network traffic into different classes, some controlled with SDN, others with traditional mechanisms. A common way to define the classes is using the Virtual LAN (VLAN) tag. Other options include separating the traffic based on the transport protocol (TCP or UDP), application protocol (HTTP, FTP, etc.) or service (e.g., VPN). Like in the service-based model, the network devices (at least a subset of them) are required to support both traditional and SDN networking (see Figure 3(c)). Hence, the traffic-class strategy can be adopted to gradually and safely transition from traditional networking to SDN, as one of the two paradigms might also serve as a backup solution.

  5. Controller-based. Advanced SDN controllers such as ONOS and OpenDaylight, implement a range of SBIs, some of which allow them to interact with legacy devices. This is very convenient in deployments where a centralized control plane is the main requirement. In this scenario, the SDN controller interacts with the control plane of the devices to manipulate the forwarding tables or to tune the routing protocol settings. The controller-based strategy, illustrated in Figure 3(d), often represents the first step of the migration towards a fully-fledged SDN network environment.

  6. HAL-based. The main objective of a Hardware Abstraction Layer (HAL) is to make a legacy network device SDN-compatible through a set of abstractions. Specifically, the HAL hides the technology and hardware-specific features of the network device, with the aim of presenting an abstracted SDN-compatible device to the controller. This approach can be also employed to extend the capabilities of SDN-enabled nodes, e.g. network virtualization support, compatibility with multiple versions of the OpenFlow protocol [42]. This invasive approach, requires the installation of hardware-specific software components onto the device. The HAL takes the full control of the FIB, as represented in Figure3(e).

  7. Agent-based. A software module, called agent, is executed on the network device to enable traffic control and forwarding through SDN. The opposite is also possible, though less common, where a software agent is installed on SDN-enabled devices to support the traditional routing/switching protocols (Figure3(f)). Unlike the HAL, the agent allows the coexistence of both legacy and the SDN control planes, hence sharing the forwarding tables of the device. This is also called dual-stack mode [43].

  8. Overlay-based. As the name says, this model is based on building an overlay SDN network on the top of a mixed legacy/SDN network (see Figure 3(g)). The overlay network consists of virtual SDN devices that are mapped onto the SDN portion of the physical network, and are interconnected through virtual links. A virtual link can be mapped onto one or more physical links in the physical substrate. The FIBs of the SDN devices includes traffic forwarding rules, plus a set of entries that are required to build the virtual view of the network for the SDN controller (e.g., the Big Virtual Switch [44]).

Fig. 3: hSDN models

The aforementioned models have been deeply discussed in previous surveys [33, 35]. We refer the reader to them for more details on the benefits and drawbacks of the different deployment strategies. Instead, in the rest of this chapter, we review the state-of-the-art in hSDN with particular focus on the strategies introduced above.

2.2 The hSDN Data Plane

The data plane executes traffic processing (e.g., CRC checking, QoS queueing, TTL update, etc.) and traffic forwarding based on the decisions of the control plane. In hSDN networks, the data plane combines traditional devices (typically Layer 2 (L2) Ethernet switches, Layer 3 (L3) routers, and other hardware appliances) with SDN-enabled devices, based on the adopted deployment strategy (cf. Section 2.1). The SDN portion of the data plane is composed of simple forwarding elements with no embedded intelligence. The decisions on how to forward or process the network traffic are taken by a remote and logically centralized entity called controller, which interacts with the SDN-enables devices through its SBI. Modern SDN controllers support multiple SBIs such as OpenFlow (OF) [45], ForCES [5] and some others [46]. Other SBIs such as Network Configuration Protocol (NETCONF) [47] and Simple Network Management Protocol (SNMP) can be also used to this purpose, even though they might be not suitable for addressing low-latency or high-bandwidth requirements [46]. It is also worth mentioning that there exist control protocols that natively support hybrid data planes. One is OF starting from version v.1.3, whose specification defines the OF-hybrid mode switch [48]. In hybrid mode, the ”NORMAL port” action programs the switch to forward packets based on the legacy protocols stack. Also, the interface to the Routing System I2RS [49] defines a SBI that allows the manipulation of the routing control on a modified node, using an agent to control legacy routing tables. SDN controls some specific flows and leaves legacy routing protocols the management of other flows.

In this section we review the scientific works available in the literature that have tackled the issues related to the integration of traditional and SDN data planes.

2.2.1 Hal-based data planes

As introduced in Section 2.1, HAL is a software layer that turns a legacy device into an SDN-compatible switch. In [42], a HAL has been designed to provide OpenFlow support to traditional devices. The proposed HAL is divided in two layers: the first is a common platform independent layer, called Cross-hardware platform layer, which is in charge of the node abstraction and configuration. The second layer is platform dependant and overlooks the translation and execution of commands in the node (see Figure 4). This HAL has been implemented for various Hardware (HW) platforms, such as Programmable network processors, light-path devices, Point to multi-point access networks e.g. Gigabit Ethernet Passive Optical Network (GEPON), Data Over Cable Service Interface Specification (DOCSIS)), Net Field Programmable Gate Array (FPGA).

Fig. 4: HAL-based model

Szalay et al. [50, 51] propose HARMLESS, a framework that relies on a server to implement an SDN control plane legacy L2 switches. Every port has to be configured with a unique VLAN identification on a legacy switch and this configuration can be repeated in the rest of the switches. Commodity switches are connected to the server through the trunk ports and received frames are sent to the server through these ports with its input port VLAN ID. Afterwards, when the server receives the frame with the input port VLAN, in a special OF translator switch, it converts the VLAN ID. into a virtual port number that connects this switch with a second standard SDN-enabled switch. Once the packet has being processed by the SDN-enabled switch, that packet is returned to the translator switch through a different virtual port, tagged with a different VLAN ID and finally sent back to a legacy switch.

Similar HAL-based approaches are presented in other recent surveys [35].

2.2.2 P4-based data plane

P4 is a domain-specific programming language for expressing how the network traffic is processed and forwarded in the data plane, including P4-enabled forwarding device such as switches, routers, network cards, etc.  [6]. The workflow of P4 programming model comprises three main elements: (i) the P4 architecture, which identifies the P4-programmable data plane blocks and interfaces, (ii) the P4 program, written in P4 language and compiled for the target hardware architecture or software device, and (iii) the target, either a software switch or a hardware element, including programmable network cards [52]. P4 enables programmers to define a variety of protocols and data plane behaviours to be implemented in nodes without depending on manufacturers to implement new features. However, despite its great potential, the adoption of P4 in hybrid environments is still very limited.

Feng et al. [53] propose Clé, a solution for improving the security of legacy networks with P4-based programmable data planes. Clé finds the minimum set of critical legacy switches that should be upgraded to maximise the security of the network through SDN functionalities. In the SDN switches, forwarding and security functions (basic firewalling and Intrusion Detection System (IDS)) are implemented using the P4 language [54].

Martinez et al. [55] propose ARP-P4, a P4-Runtime implementation of the data plane and a new L2 legacy protocol combined with SDN. ARP-P4 defines a data plane that forwards any ingress packet locally via ARP-Path or remotely via P4 Runtime installed rules through an SDN controller.

Recently, various solutions to translate P4 programs for traffic processing into Field Programmable Gate Arrays code have been proposed on the market and in the scientific literature. Among others, we mention Netcope P4 [56], P4FPGA [57], P4FPGA  [58] and P4toFPGA [59]. The main idea behind these initiatives is to combine the high-level P4 programming language with the performance of FPGA-based packet processing.

2.2.3 Segment Routing (Sr)

Segment Routing is a data plane technology that can be used with the hSDN paradigm, as it allows a source node to specify a path as an ordered list of segments (instructions that a node executes on the incoming packet) in order to guide a packet across the network. A segment scope can be local to a SR node or global within an SR domain. In SR, the header has sufficient information to route the packets from the ingress to the egress node; besides, it can be applied to a technology with the source routing capability such as MPLS and IPv6.
SDN and SR technologies can be deployed together to enable key benefits such as scalability (SR minimizes state changes by maintaining the state at the source node only), agility (rapid response to topology changing in the source node only), and centralized manageability (SDN provides a global view of the network topology across multiple domains) [60]. One of the latest proposals, Serement et al. [61], implements and compares two scenarios with SR combined with MPLS- Traffic Engineering (TE) in one scenario and SDN in the second. They conclude that SR jointly with SDN presents two outstanding characteristics that the MPLS-TE scenario does not: (i) Global view of the network, in MPLS-TE, when a tunnel is going to be opened, the source router calculates the best path that meets the resource requirements without taking into account other network nodes’ requirements. As a result, this could lead to an inefficient usage of network resources. Conversely, SDN controller has the global view of the network which results in optimal usage of network resources. (ii) Continuous link monitoring, in MPLS-TE, the nodes react by rerouting tunnels only when there is a link failure. The SDN controller can be programmed to monitor congestion with certain threshold. If the defined threshold is exceeded, the SDN controller can establish a new segment path and load balance traffic between the old and the new path.

Zhang et al. [62] have recently proposed a mechanism for Real-Time (RT) multimedia services based on multipath transport, SDN and SR. They take advantage of the global view of SDN, the path allocation algorithm in the controller, considers future traffic distribution and current links load in order to reduce network congestion. In addition, SR is used to eliminate the scalability problem of SDN with multipath transmission. Their proposal balances network traffic as well as it improves the quality of end-user’s service experience.

2.2.4 hSDN in mobile networks

Poularakis et al. [63] propose a hSDN architecture for mobile networks in order to alleviate the reliability limitations and risks of service interruption inherent in these kind of networks. The data plane nodes use a solution based on SR [60]. In order to deploy SR, the network is divided into segments and the SDN controller is in charge of the inter-SR and it periodically broadcasts the list of segment labels (endpoint nodes of the segments) to the nodes instead of flow rules. The routing inside a segment is made by a legacy routing protocol such as OLSR (Optimized Link State Routing Protocol). The control plane node forwards packets depending on the previously distributed SDN segment label. The packets are forwarded through the segment nodes domain to the endpoint node of the segment. Then, this last node, which is also connected to next segment, forwards the packet through the next segment. The process is repeated until the packets reach their destination.

2.2.5 Field Programmable Gate Arrays (FPGAs)

Kaljic et al. [64] present a hybrid FPGA/CPU node architecture with a dedicated central processing unit that aims at overcoming some OF limitations. The OF switch programmability is limited to the level of the flow table configuration. Thus, an SDN node can not implement new or legacy (e.g. ARP) protocols or advanced packet processing functionalities such as encryption, transcoding, Deep Packet Inspection (DPI), etc. The hybrid architecture is chosen because the FPGA technology achieves the SDN high-speed packet processing while the CPU allows the implementation of new protocols and advanced packet processing functionalities.

2.3 hSDN Control Plane

The control plane is responsible for instructing the data plane in accordance with the application plane requirements. Due to the fact that a hybrid control plane has to manage both legacy and SDN forwarding devices, there are many legacy protocols that can be part of the controller in addition to the OpenFlow (OF) protocol. For example, there are some legacy L2 protocols such as Spanning Tree Protocol (STP), VLAN, Address Resolution Protocol (ARP), Link Layer Discovery Protocol (LLDP), SNMP to control connected L2 data planes. Similarly, protocols like OSPF, BGP, IS-IS, control the L3 data plane.

One of the limitations of the SDN control plane in hybrid networks is that the de-facto protocol for discovery topology, the LLDP can not discovers legacy nodes. This can be addressed by a recent proposal, the Hybrid Domain Discovery Protocol [65], which discovers all nodes in a wired network.

The remainder of this section elaborates on the hSDN control plane. For the sake of coherence, we follow the classification of hSDN models presented in Section 2.1.

2.3.1 Island-based model proposals

In an island-based model, the network is topologically split into areas consisting of only legacy nodes, whereas the other areas of the topology have just SDN nodes. Shi et al. [66] propose an island-based approach to deploy a new high-speed connection service (100 times faster than usual) for authenticated users. This model is implemented at the Kentucky University campus, represented in Figure 5, where a portion of the buildings adopt SDN-based networking, while other use Cisco legacy devices. To route the traffic from legacy networks to the SDN network, the authors developed a new controller module designed to communicate with legacy switches using Policy Based Routing (PBR).

Fig. 5: Island-based model at the Kentucky University campus

Another example of the island-based model is the application of SDN in the BGP inter-domains communications to ease and improve its control. Lin et al. [67] propose a practical transition to SDN, called BTSDN. The key idea in this approach is to integrate the border BGP routers, which have high price and performance, with SDN. It is noteworthy that in BTSDN, BGP still works in the same way as it does in the Internet. BGP routers in inter-domain routing run External Border Gateway Protocol (eBGP) while they run Internal Border Gateway Protocol (iBGP) in intra-domain routing tasks. Inside the BTSDN domain there is a controller which also runs the iBGP protocol and acts as an iBGP router to learn the Internet routing table. The controller is also responsible for configuring intra-domain and inter-domain flow for the hosts communications.

2.3.2 Agent-based model proposals

Feng and Bi [68] propose an architecture called “OpenRouteFlow”. OpenRouteFlow provides a centralized control over legacy nodes by upgrading the legacy node software with an agent called “OpenRouteFlow”. This agent communicates the distributed routing information to the controller. Moreover, the OpenRouteFlow agent receives, in parallel, the application oriented control instructions from the controller, via OF messages, which are then map into ACL and QoS operations of the router.

The complementary case is also possible, where an SDN node is upgraded with a legacy agent. Tilmans and Vissicchio [69] propose an “IGP-as-a-Backup SDN” architecture. On one hand, the controller configures the route using the OF protocol and takes advantage of the optimal network operation in the long term. On the other hand, the legacy agent builds backup paths using the OSPF protocol. Every time a network failure occurs, the Inter Gateway Protocol (IGP) agent quickly restores the connectivity using the local routing information. In this way, network failures are repaired faster than in a pure SDN network.

Alvarez-Horcajo et al. [70] propose a hybrid switch with partial delegation of basic bridging and new cooperative mechanisms between controller and these switches. The flow processing mode by default is done using a L2 legacy protocol. The SDN controller can also installs forwarding rules on the switches. Finally, there is a ”cooperative” flow processing mode based on the fact that the controller has knowledge of the operation of the switches, it could help then with some specific tasks because of its global knowledge of the underlying network, for instance to compute restoration paths or to help in recovery mechanisms.

2.3.3 Class-based model proposals

Open Source Hybrid IP/SDN (OSHI) nodes [71] combine SDN with IP Linux kernel routing to enable research in hybrid IP/SDN carrier network scenarios. The traffic traversing OSHI nodes is divided into classes, some of them are forwarded to the next hop based on the decisions of an external SDN controller, others using the Linux routing tables (Figure 6). The SDN-enabled forwarding plane of a OSHI node is implemented with OvS [41], a software framework that implements a fast forwarding path in the Linux kernel. The legacy IP routing is implemented with a combination of the Linux kernel IP networking and Quagga [72], which acts as a routing daemon.

Fig. 6: OSHI Hybrid IP/SDN node architecture

2.3.4 Controller-based model proposals

Kuliesius et al. present [73], a practical deployment of hSDN, moving two classic functions of the SDN control plane, namely network discovery and control, to the management plane. They develop each function on a module that interacts with the control plane via Representational State Transfer (REST) APIs so that they can fit any controller that supports REST interfaces. Topology discovery for SDN nodes is based on the Broadcast Domain Discovery Protocol (BDDP). This protocol has the same structure as the LLDP/ OpenFlow Discovery Protocol (OFDP) except that the destination multicast address is changed to a broadcast address and the protocol field is also changed. Furthermore, to reveal information about legacy devices, the topology discovery module generates SNMP requests to register devices and/or gets SNMP trap messages. To register switches, they must be pre-configured with an SNMP IP address and authentication data. The module also uses the port traffic load to detect devices. If the port statistics show activity but the Broadcast Domain Discovery Protocol (BDDP) discovery does not find the neighbor on this active port, the module should turn the port down. The SDN controller then converts OF rules into SNMP in order to to communicate and configure the legacy device.

Caesar et al. [74] develop an example of how to apply the controller-based model to manage external traffic in a BGP domain. They propose a similar device to a central controller called “routing control platform” (RCP). It establishes an iBGP session with all the routers in the domain. RCP collects information about external destinations and internal topology and select the iBGP routes for each router in the domain and sends them using iBGP. RCP reaction is fast to the link failures in the network and it also provides scalability.

Vissicchio et al. [36] propose a controller to manage a legacy network through fake nodes. In their next work, Vissicchio et al. [75] propose a central controller called “Fibbing”. It manipulates the input of traditional routing protocol by introducing fake nodes in the network through the injection of fake Link State Advertisementss. Based on the path requirements, it injects fake LSAs in the network to introduce fake nodes in the network topology. Fibbing provides LB, traffic steering, and providing a backup path as well.

2.3.5 Service-based model proposals

MPLS-TE is efficient in achieving a control plane based on both legacy and SDN, protocols. Legacy MPLS-TE control allows the manipulation of the routing and bandwidth allocation of each tunnel to avoid congestion and meet the bandwidth and latency requirements [76]. When a tunnel is heavily congested, an efficient way to alleviate it is to open new tunnels and split traffic among them. If congestion decreases, tunnel merging is performed by concentrating the traffic back to a single tunnel and removing the other tunnels. Legacy MPLS-TE does not have built-in splitting and merging capability and the SDN control plane takes over these limitations.

2.3.6 Overlay-based model proposals

In this model an SDN is built as an overlay on the top of the legacy network. The logical overlay is composed of the SDN nodes and of the logical links that connect them. These logical links are implemented over one or more legacy nodes and their links. Levin et at. [77] propose an implementation of the overlay-model called ”Panopticon” where SDN nodes are connected with legacy nodes. Pure SDN applications can run over the Panopticon network. The key idea of this implementation is the application of traffic at the reference point which forces every packet, going from source to destination, to traverse an SDN switch. Upon reaching the SDN switch, the SDN controller handles the packet. In this way each packet is processed as it would be in an SDN network.

Caria et al. [78] propose to divide an OSPF network into separate sub-domains that are connected with SDN nodes, see Figure 7. Changes in the network trigger broadcasting of LSA and re-computation of the routing table in legacy routers. These LSAs are also received by SDN nodes and passed to the SDN controller, where the hybrid network manager processes them. The manager optimizes the sub-domain routing configurations for LB by computing the OSPF link costs based on the network division. Then, through the SDN nodes in the corresponding boundary of sub-domains, these configurations are applied to the network via LSAs.

Fig. 7: Overlay-based model

2.3.7 Combined proposals

Poularakis et al. [79] propose three methods (region-based, overlay-based and agent-based) to combine distributed and SDN control planes in mobile networks:

  1. Dynamic migration of SDN control protocol: Part of the nodes in the SDN network are enabled to dynamically swap to an IP routing protocol (e.g. Optimized Link State Routing Protocol (OLSR)) and to ignore the SDN forwarding rules. As a legacy protocol can adapt to network changes faster than a remote SDN controller, dynamic migration can be adopted in regions of the network where topology changes are frequent.

  2. Cluster-based hierarchical control: This method divides the mobile network into several zones and runs an IP routing protocol for each zone independently from the rest. The centralized SDN controller will be responsible for the traffic routing in the overlay network formed by the zones.

  3. The distribution of backup SDN rules: In this third method, SDN proactively distributes backup SDN rules in legacy nodes that can be taken to repair network failures. As a proof-of-concept for this third method, they implement a prototype with a local agent on a smartphone. They also install OvS so the smartphone becomes an SDN switch. The agent periodically sends ”heartbeat” messages to neighbor nodes to decide whether the link is up or down. If it is down the agent takes the backup SDN rule.

In Table I, we provide a summary of the most relevant above-mentioned deployment methods and their specifications, proposed by the research community, over hSDN networks for the data and control planes.

Proposal Model Plane Node Use Case Key idea Protocols Evaluation
Parniewicz et al. [42] HAL DP L TE in an European network Install HAL in legacy nodes OF Commercial implementation
Szalay et al. [50], [51] HAL DP L ACL, LB, Access gateway Traffic tags with VLAN VLAN, OF Testbed
Feng et al. [53] Service DP L Avoid security attacks Attract traffic to SDN nodes ARP, OF -
Kaljic et al. [64] Service DP L & SDN Implementation of new protocols FPGA run SDN, CPU run new protocolos OF Small testbed
Poularakis et al. [63] Service DP L & SDN hSDN in mobile network SDN Distributes SR OF, OLSR Small testbed
Shi et al. [66] Island CP L High speed connections Control legacy nodes with commands & SNMP OF& SNMP Campus network
Lin et al. [67] Island CP L & SDN Integration with BGP domains Integrate BGP routers with SDN OF & BGP Quagga Testbed
Feng and Bi [68] Agent CP L Efficent routing Integrate an agent in legacy node OF & IGP Click Router testbed
Poularakis et al. [79] Class CP L & SDN Avoiding service interruption An agent follows backup SDN routes OF Proof-of-concept
Salsano et al. [71] Class DP L TE, Backup Quagga & OvS integration IP, OSPF, BGP, OF Simulation & Implementation
Kuliesius et at. [73] Controller CP L & SDN Legacy and SDN integration Control legacy nodes with SNMP OF, SNMP, NETCONF Testbed
Levin et at. [77] Overlay CP L & SDN Deploy of SDN in operating networks Waypoint enforcement of traffic OF, VLAN Hardware testbed & Mininet
Caria et at. [78] Overlay CP SDN Traffic Engineering Sub-domain controlled by SDN OF, OSPF Numerical analysis
Plane: DP (Data Plane) or CP (Control Plane)
Modified node: L (Legacy) or SDN)
TABLE I: Comparison of hSDN Main Proposals

2.3.8 Controller placement

Implementing an SDN controller in a single server leads to scalability and robustness issues. A physically distributed but logically centralized SDN controller architecture resolves these issues. Nonetheless, the distributed SDN controller implies the problem of the location and number of controller instances and the node-to-controller mapping. This problem is referred to as the Controller Placement Problem (CPP) [80]. A good controller placement policy is vital to guarantee a reliable and efficient network management. Nevertheless, previous surveys dedicated to hSDN networks [30, 31, 81] do not cover the problem of the controller placement in depth. In this regard, we analyze recent CPP surveys in pure SDN networks.

Prakasa et at.[82]

review the existing literature on the controller placement. They propose a complete taxonomy of controller placement that is divided based on : (i) Network type (wired or wireless), (ii) Traffic characteristics (static or dynamic), (iii) Controller characteristics (if the capacity of controllers is limited or not), (iv) Solution approach (optimization formulation, heuristic approaches, network partitioning, game theory), (v) Reliability of network elements (controller, links, nodes) and (vi) optimization objective(s) (latency, connectivity, cost, etc.).

Another survey [83]

presents a deep study on different solutions, which are classified based on the following metrics: latency, flow setup time, availability, capacity, energy, cost and some combinations of them. Lu et al.

[80] survey CPP from the perspective of optimization as the main objective. They classify CPP into four aspects: latency, reliability, cost and multi-objective. In the rest of this section, we discuss some of the most relevant works and solutions about CPPs in pure SDNs.

The algorithm proposed by Mohanty et al. [84] aims to reduce the routing cost of the network considering the demands of nodes and controllers capacity. Schutz and Martins [85] propose a mathematical formalization of the CPP, which takes into consideration constrains of propagation latency and controller capacity, and determines the minimum number of controllers, their location and the mapping of nodes to each, while keeping a balanced load distribution among controllers. Simulations on 16 real network scenarios show that the proposed approach produces balanced and resilient solutions, which are proven to be optimal for most of the evaluated networks.

Two interesting works have been recently published on the CPP in hSDN. Guo et al. [86] study the joint optimization of upgrading the numbers of SDN switches and controllers in hSDN networks under a budget constraint. Deploying multiple controllers should take the following two factors into account: First, controllers should be able to process a number of flow requests in a timely manner, otherwise it will impact in the network performance. Second, the distance is a significant parameter in WANs because the propagation delay is part of the total flow request delay. They show that the problem is NP-hard and propose heuristic for small and large networks.
Yuan et al. [87] focus on controller placement in hSDN WANs. Having only one controller may not be convenient in large WANs for reasons such as latency (caused by large distances), load balancing, scalability and reliability. The authors argue that the latency of the control messages might be affected by the legacy routers they traverse. In this regard, they propose a dynamic controller assignment approach that considers the impact of legacy nodes on the latency on the control channel.

2.3.9 The control plane scalability

One of the main problems with CPP is the control plane scalability due to different types of existing SDN network architectures, as depicted in Figure 8 [81].

Fig. 8: SDN network architectures
  1. A single controller based SDN architecture consists of only one controller to maintain the global network state and control of the entire data plane of the network (Figure 8 (a)). In this scenario, a single controller can become a bottleneck for the network. To deal with scalability issues in single-controller deployments, a range of different approaches are envisioned in [88]

    . They involves multi-threading strategies, optimization of the flow tables in the switches and machine learning techniques for traffic classification and prediction.

  2. In the flat topology, the network is divided into multiple domains, each of these domains is managed by a controller (Figure 8 (b)). The controllers exchange information about their domains between each other.

  3. In the hierarchical topology, local controllers run SDN applications which do not require a global view of the network. A global controller coordinates the local ones (8 (c)).

  4. In the hybrid distributed controllers architecture, flat and hierarchical architectures are used to organize the controllers in the network (Figure 8 (d)). In this scenario, the local controllers are connected with root controllers, and root controllers are directly connected with each other. Multiple domain controllers, i.e. root controllers in the network, maintain global network information by coordinating among themselves at regular intervals. Each root controller is responsible for handling multiple local controllers. Network discovery, routing, and other SDN services are run both at the local and the root controllers.

3 Security and privacy in Hybrid SDN

The adoption of SDN technologies in legacy networks has fostered the design of novel security solutions, enabling the implementation of security services tailored to the specific needs of the users and their applications, as well as allowing a faster and coordinated reaction to zero-day network threats.

Despite these benefits, SDN also might bring an increase in the attack surface. The SDN controller, the SDN control channel and the size of the TCAM of SDN switches are specific features of the SDN architecture that may represent additional targets for network intrusions or other malicious cyber-attacks. In a hybrid environment, they can be exploited by cyber-criminals to undermine the whole network, legacy devices included. In this section, we analyse the security risks in hybrid SDN environments and the solutions proposed by the research community.

3.1 Vulnerabilities and Threats

As analysed in recent surveys [35, 33, 31], the coexistence of different control planes can be source of security issues in hybrid SDN networks. For example, updating a hybrid network might lead to forwarding inconsistencies [35]. Moreover, security loopholes can be caused by a poorly secured interaction between legacy and SDN devices, like in the scenario where a legacy switch reports statistics to the SDN controller over an unsecured connection [33]. It is also worth mentioning the vulnerability to ARP spoofing attacks that might be determined by autoconfiguration mechanisms enabled in hSDN networks to adapt the traffic routes to topology changes [31].

On the other hand, hybrid SDN is also open to a wide range of security issues inherited from traditional and software-defined networking. In this regard, the remainder of this section focuses on classifying and discussing the specific vulnerabilities of legacy and SDN-enabled networks with the aim to provide a comprehensive overview of the attack surface of hybrid deployments.

3.1.1 Unauthorized access

Both traditional and SDN network devices expose monitoring, management and control interfaces. An attacker can either use the default credentials, or obtain weak credentials, to log into the devices through such interfaces. This scenario opens the door to a number of malicious activities ranging from unauthorized re-configuration of network devices to the manipulation of traffic routing policies [89, 90].

Specific to SDN, the control channel interfaces the controller with the SDN-enabled switches. As per SDN functional specification, multiple controllers can access the data plane for load balancing and redundancy. This opens the opportunity for an attacker to impersonate an SDN controller, hence to control how the traffic is routed in the network by injecting custom rules in the flow tables of the switches, potentially disrupting the normal operations through configuring black-holes, loops in the networks, or simply for re-directing part of the traffic towards a remote host for inspection [91]. As modern SDN controllers allow the dynamic activation/deactivation of the SDN applications, with direct access to the SDN controller an attacker could disable a firewall or other security-specific SDN applications that are protecting the network [92, 93].

3.1.2 Vulnerabilities from applications

Besides the vulnerabilities and threats introduced in the network by malicious software (known as malware) executed on end hosts, network security can be also compromised by malicious/malfunctioning SDN applications. Indeed, as SDN applications can operate network control through the northbound interface of the controller, they can potentially harm the normal network operations and management. In the case of in-house applications, the risks for the operator/network administrator are mainly related to software bugs. On the other hand, SDN applications coming from third-party repositories (as envisaged, for instance, by authors of [94]) might embed malware code intentionally designed to damage the network.

In addition, advanced SDN controllers (e.g., ONOS [95] and OpenDaylight [96]) allow multiple applications to be executed in parallel to perform different tasks on the same network traffic (e.g., monitoring, load balancing, packet filtering, network address translation, etc.). If not properly coordinated, such applications might introduce inconsistencies in the forwarding rules, which can cause unexpected traffic routing or security breaches. In [97], the authors propose a so-called Network Engine to mitigate these issues.

3.1.3 Man-in-the-Middle attacks

The vulnerability of user data to threats such as eavesdropping and tampering is a general concern in network security. In this regard, the objective of Man in the Middle (MITM) attacks is to steal sensitive information by monitoring the communication between endpoints, or by impersonating one of them.

A MITM attack can target the user traffic in the data plane as well as the SDN control channel between the SDN controller and the data plane. In a hybrid scenario, an attacker can build a MITM attack to impersonate the SDN controller, hence to inject custom rules in the network, or to mimic the behaviour of the SDN switches with the aim to send crafted messages to the controller (e.g., false topology updates). This vulnerability can be avoided by implementing a strong authentication/encryption mechanism between switches and controllers. For instance, the OpenFlow specification defines Transport Layer Security (TLS) as the mechanism (although optional) for mutual authentication between the OpenFlow controller and the switches. However, due to its non-trivial configuration procedure [98], TLS is not always enabled by operators and network administrators, making the control channel vulnerable to MITM attacks.

In virtualized SDN environments, network hypervisors (FlowVisor [99], OpenVirteX [100] etc.) logically operate between control and forwarding planes to assign “slices” of the network traffic to multiple controllers. Unauthorized access to the hypervisors allows a hacker to build a MITM attack on the control channels of all the virtual networks. This issue can be partially mitigated by implementing the virtualization mechanism directly in the data plane [101]. A similar level of vulnerability to MITM attacks is enabled by composition frameworks, such as CoVisor [102], NetIDE [97], which can be placed between the SDN controller and switches to coordinate the execution of SDN applications written for different controllers.

3.1.4 Misconfigurations

Communication networks are complex environments, whose management procedures are often based on manual (hence error-prone) intervention of expert personnel. In recent years, network administrators have started adopting SDN technologies to implement more advanced network automation methodologies. Nevertheless, the deployment of the SDN components and their correct configuration pose several technical challenges. On the one hand, as discussed in Section 3.1.3, the configuration of the TLS mechanism between SDN controller and switches is a complex task. A misconfigured TLS channel, or the adoption of weak authentication mechanisms and certificates can expose the network to several security threats [103]. In addition, as multiple SDN applications can access the forwarding plane through the northbound API of the controller and multiple controllers can control the same network, a lack of coordination between the different forwarding logics might lead to inconsistencies in the network. These aspects have been tackled by a number of scientific works [104, 105, 102, 97], where the authors propose methods for the composition of forwarding rules and the resolution of conflicts between different SDN applications.

3.1.5 Denial of Service (DoS) attacks

DoS attacks are common and harmful types network intrusions in which the cyber-criminals generate malicious traffic from compromised devices with the aim of disrupting the normal functions of the target networks or computing systems. In hybrid environments, the SDN controller can bring clear advantages in the mitigation of this type of the attacks. Indeed, thanks to its privileged position in the network, the controller can programmatically insert traffic filtering rules in strategic SDN-enabled devices to block DoS attacks efficiently [106, 107, 108]. However, both SDN controller and switches can be subject to a variety of DoS attacks:

  1. Volumetric attacks, which try to overwhelm the resources of the target system by flooding the victim with large volumes of traffic. In SDN-enabled networks, an attacker may send a high rate of unique packets to force a continuous communication between the switches and the controller, saturating the SDN controller with control messages. Depending on the control logic implemented in the controller, this might also lead to a saturation of the (limited) capacity of the TCAM memory of the switches [109].

  2. Protocol attacks, which exploit the weaknesses or a specific behaviour of protocols. The objective of these attacks is to consume the computational resources of end hosts and network devices such as firewalls. One example in this category is represented by Syn Flood attacks, which exploit the TCP three-way handshake to consume the resources of the victim server and making it unavailable to the legitimate users. The SDN controller can be vulnerable to this type of DoS attacks.

  3. Application layer attacks, where the attacker exploits the specific logic of an SDN application or service to crash the controller, making it inaccessible to the SDN-enabled switches. These attacks usually produce lower volumes of traffic compared to volumetric and protocol attacks. This aspect makes them harder to detect.

Of course, also the legacy portion of the hSDN network is vulnerable to DoS attacks. Legacy network devices can be attacked through the management interface [110] or can enable hackers to build DoS attacks against the network infrastructure [111].

3.2 Threat detection and mitigation

The hSDN paradigm can be exploited to implement advanced network security solutions by combining state-of-the-art technologies available for traditional and softwarized networks. Nevertheless, the integration of traditional hardware appliances and software-defined functions is not trivial and can be influenced by multiple factors, such as: (i) the hSDN deployment model (cf. Section 2), which might determine how the traffic is routed within the network and where the security functions can be executed [112, 113], (ii) the specific security policies and best-practices of the operator/enterprise, which define how each class of network traffic should be processed [114, 115], and (iii) the user requirements in terms of QoS (usually maximum end-to-end latency and minimum bandwidth), as busy links and devices might create bottlenecks in the network.

In terms of QoS, specialized hardware appliances, also called middleboxes, are widely used in traditional networks to achieve line-rate traffic monitoring, shaping, filtering and other network functions. However, they often run proprietary functions and operating systems, and are fixed in their position in the network. hSDN can partially overcome this limitation by complementing traditional middleboxes with advanced security functions implemented in software and executed either in the control plane, the data plane, or both. This includes SDN controller and switches and Network Function Virtualization (NFV)-enabled servers/end-hosts. In this regard, network programmability enabled by SDN allows the implementation of efficient network monitoring solutions entirely executed in the data plane [116, 117], as well as the design of advanced threat detection methods that can exploit the global view of the network state from the logically centralized control plane [118, 119]. In addition, SDN and NFV enable the provisioning of complex network security services in the form of sequences (often called chains) of Virtual Security Network Functions, i.e., security-specific Virtual Network Functions such as firewalls and Intrusion Prevention Systems, running on physical or virtual machines [114, 115, 120]. This technique is called Service Function Chaining (SFC[121]. The dynamic network management enabled by SFC allows the implementation of highly-customisable security services and a faster adaptation of the system to unknown threats. On the other hand, compared to specialized hardware appliances, VSNFs may have a negative impact on the performance of the network and on the QoS level experienced by the users. The virtualization overhead, the utilization level of the servers and the techniques adopted to implement the VSNFs are the most significant contributors to the performance degradation. To mitigate the problem, recent approaches propose to either adopt kernel bypass technologies such as the Data Plane Development Kit (DPDK) [122], Netmap [123] or Vectorized Packet Processing (VPP) [124], or to move the software packet processing in kernel space using eBPF/XDP [125].

An overview of existing security solutions for hSDN deployments is presented in the next section.

3.3 Frameworks or Existing Security Modules

There is small body of work in the scientific literature proposing methods, algorithms or architectures for the security of hSDN networks. The most relevant papers are categorized in this section.

Article Category Technique
Heng et al. [126] Firewall
Adaptation of traffic filtering rules to different types
of switches
Gamayunov et al. [127] Firewall
Translation of firewall ACLs
for legacy networks into flow rules for SDN devices
Fiessler et al. [128] Firewall
Combination of SDN techniques with traditional
firewalls
Feng et al. [53]
Partial deployment
Framework for efficient upgrade of legacy devices
Ding et al. [129]
Partial deployment
Greedy algorithm for incremental deployment of
SDN devices
Cheng et al. [130]
Threat prevention
and mitigation
A solution for SDN controllers to protect the BGP
peering in hybrid networks
Ubaid et al. [131]
Threat prevention
and mitigation
Mechanism for the localization of the attackers in
ARP spoofing attacks
Sebbar et al. [132]
Threat prevention
and mitigation
Algorithm for the mitigation of a variety of security
threats
Zkik et al. [133]
Distributed hybrid
SDN networks
Security layer for hybrid distributed SDN networks
designed to detect and mitigate network intrusions
Zkik et al. [134]
Distributed hybrid
SDN networks
Modular security plane composed of firewall and
anomaly detection modules
Amin et al. [34, 135]
Network monitoring
Automated re-configuration of ACL policies upon
changes in the topology
Lorenz et al. [136]
NFV-based solutions
Patterns for the integration of SDN/NFV-based
security solutions into traditional enterprise networks
Doriguzzi-Corin et al. [115]
NFV-based solutions
Algorithms for the progressive and application-aware
placement of security functions in telecom networks
H. Jmila et al. [120]
NFV-based solutions
Methods for the provisioning of security-aware
service requests
Shameli et al. [114]
NFV-based solutions
Algorithms for the placement of security functions
in data centers
TABLE II: Security frameworks for hSDN

3.3.1 Firewall for hSDN networks

FlowAdapter [126] is a firewall designed to work in environments with mixed SDN and legacy devices. FlowAdapter translates the filtering rules from the controller flow table pipeline to switch hardware flow table pipeline, so that the same rules can be fitted into different types of hardware. In the opposite direction, Gamayunov et al. [127] propose a solution for translating the firewall access lists for legacy networks into flow rules for SDN devices.

The authors of [128] propose a hybrid firewall solution called FireFlow, which combines the standard controller-switch SDN architecture with a software-based firewall in the data plane. FireFlow offloads the complex decisions to the firewall, such as those based on string matching rules.

3.3.2 Partial deployment of Sdn-based security functions

Authors of [53] argue that upgrading only a few legacy switches is sufficient to achieve the desired level of security for all the network infrastructure. In this regard, the authors propose Clé, a system composed of three main modules: the Device Upgrader determines which legacy devices must be upgraded to SDN, the Unified Controller re-directs the traffic towards the SDN switches present in the network for threat analysis, and Clé Data plane encompasses legacy and P4 switches. Security functions such as firewall and IPS are implemented in P4 language and installed onto the P4 switches.

In [129], the authors propose a greedy algorithm for the incremental deployment of SDN programmable switches in legacy infrastructures that aims at maximising the number of monitored network flows.

3.3.3 Threat prevention and mitigation

In [130], the authors propose S-SDBGP (Secure SDBGP), a solution for SDN controllers to protect the BGP peering in hSDN networks. S-SDBGP includes three security-related software modules. A TCP authentication module that generates an authentication code for each BGP packet. The objective is to avoid unauthorized modifications of the BGP data. Another module is in charge of verifying the legitimacy of the prefixes announced by the Autonomous System (AS). Finally, the AS path validation module leverages BGPsec [137] to prevent attackers inserting malicious ASs in the AS path. Interestingly, it is not clarified in the paper the reason why BGPsec is not enough to provide the same level of security as the proposed framework.

In [131], the authors propose a graph-based traversal mechanism for the localization of the attackers in the case of ARP spoofing attacks. The solution is studied for hSDN networks, where ARP spoofing can affect the applications running on the controller. The approach is based on configuring both legacy and SDN switches to send the ARP packets to a server for analysis. The experimental results show that the proposed solution can detect and mitigate ARP spoofing attacks.

KPG-MT [132], is an algorithm for the mitigation of a variety of security threats such as MITM, DoS and malware. KPG-MT has been implemented at different levels of the SDN architecture. Evaluation results demonstrate the effectiveness of the proposed algorithm in different SDN-enabled environments, hSDN networks included.

3.3.4 Distributed hSDN networks

AMSecP [133] is a security layer for distributed hSDN networks designed to detect and mitigate network intrusions such as identity usurpation, MITM and DoS attacks. Despite the complex software architecture, the threat detection logic relies on a simple algorithm that monitors the flow rates and takes decisions based on pre-defined thresholds for benign and malicious traffic. Indeed, the evaluation is focused on the execution time of the software components rather than the detection accuracy.

In [134], the authors present an architecture for securing logical distributed hSDN networks. The main contribution is a centralized modular security plane for network threat detection and mitigation. The security plane includes an anomaly detection module, a network IPS and a stateful firewall.

3.3.5 Network monitoring

The authors of [34] argue that network topology changes might affect network policies (e.g., ACLs) configured at the interfaces of forwarding devices. They propose a solution based on a graph difference technique to auto-detect the affected interfaces and network policies and to re-configure the policies based on the new topology state. An evolution of this work is presented in [135], where the authors propose an approach to optimize the implementation of the ACL policies in hSDN networks. The proposed algorithm is based on constructing a K-partite graph to search for the possible placement of the ACL policies. The set of candidate nodes for ACL policy implementation is determined by computing the shortest path algorithm on the graph.

3.3.6 NFV-based solutions

Authors of [136] cover different architectural design patterns for the integration of SDN/NFV-based security solutions with traditional enterprise networking. By analysing a stateful firewall use-case, the authors conclude that the hSDN approach is the best choice due to good scalability with low latency, although it may be complex to implement and monitor. Other works [114, 115] propose strategies for provisioning privacy/security services in SDN/NFV-enabled networks by means of chains of software functions executed on SDN/NFV-enabled devices in telecom networks or DCs.

4 Network Management in Hybrid SDN

Network management is a challenging task, especially in large networks. Through efficient network management processes, it should be guaranteed that a network is highly available, scalable, and reliably performant. Besides, network management sometimes entails more sophisticated tasks such as network isolation across complex network boundaries and decoupling logical and physical infrastructure through network virtualization solutions. The diverse requirements of managing a hybrid heterogeneous network can be intrinsically challenging. Accordingly, existing literature tend to use a hybrid network controller which is responsible to map legacy networking state and commands to SDN and vice versa. Besides, the controller should be able to communicate with both legacy and SDN switches while it needs to expose well defined Application Programming Interfaces to communicate with existing SDN controllers.
For example, in a hybrid network management solution proposed by Arora et al. [138], a hybrid network controller controls a network consists of SDN switches and legacy Switches. It includes a user interface configured to display network information and to receive high-level network management commands and an SDN controller configured to communicate with the SDN Switches to update and manage the SDN switches. There is also a Remote Procedure Call (RPC) manager configured to communicate with the legacy switches to update configuration and rules in them along with a hybrid manager configured to a route between two communicating nodes in the network based on a network topology, to calculate legacy switches and SDN switches in the route to update, and to create a connection and network isolation using the RPC manager and SDN switch configuration.

Fig. 9: Hybrid-controller architecture in HybNET [139]

Another implementation is HybNET [139], a framework for automated network management of hybrid network infrastructures. As Figure 9 depicts, HybNET consists primarily of three components including a physical infrastructure, a path finder, and a controller. The physical infrastructure describes the basic mechanism that HybNET connects to the network infrastructure, and the path-finder provides a basic algorithms to find viable paths in the network for connectivity. The main component and workflow is a part of the controller which manages and orchestrates the entire management framework. Each network operator request is handled as an atomic transaction. HybNET checks if the rules have been updated successfully, and reports any errors while maintaining state consistency at all times. Additionally, HybNET supplies a common API for network operators to use to process transactions. For each transaction, configurations are prepared and sent to legacy switches or SDN controllers through remote communication mechanism, such as RPC and REST calls, and a persistent state view is kept in the mapping database.

4.1 Network Update

Today’s network deployments have to be highly available, scalable, manageable, and respond to different changes in an agile manner. Therefore, network-update is a frequent operation to tackle with possible network failures, or respond to the fast policy changes [140, 141]. Nonetheless, any network update should be safe meaning that making any alteration to the network device states needs to be done with minimum or no service disruption. Moreover, any network update should lead to an incremental transition from an initial correct network state to a final correct state with reasonable computation and network bandwidth overhead. In other words, a successful network update should guarantee strong consistency in the FIB of different network nodes [142]. In an hSDN environment where legacy and SDN network devices/protocols coexist, consistency models can be applied to various network properties such as forwarding rules (for both SDN and traditional portions of the network), network policy and even performance (mostly for the SDN portion of the hybrid environment). Besides, network-update consistency can also be enforced in per-packet and per-flow levels [140].

4.1.1 Network updates within a traditional network Domain

In legacy networks where network updates are mainly pertinent to forwarding paths, distributed routing protocols play an important role in network updates. Within a network with a uniform administration policy, link state IGPs are typically used to compute forwarding paths within a network. This can be adjusted by network administrators form a network logical view in terms of weighted graph which. Such a view is then being shared among all routers in a network domain [143]. When a router’s logical network view changes for any reason, the router tries to rebuilt a new logical network view while it starts disseminating messages to its neighbors. Each neighbor also follows the same procedure until all the nodes in the network build their own new and consistent logical network view. During this period of time, which is referred to as routing convergence time [144], there is no guarantee on when and in what order the routers receive messages about new network states. This introduces transitory network state inconsistencies which potentially leads to usually short-lived forwarding disruptions between some of the network nodes.
Accordingly, both industry and academia made effort to propose feasible solutions for such a problem. Defining IGP extensions has been discussed extensively in [145, 146, 147, 148, 149]. While such solutions are ideal in theory, they have some serious limitations in practice. For example, extending routing protocols does not support all network update scenarios. It sometimes can be difficult to implement such extensions and it also takes time for them to be accepted by the community. As a consequence, other solutions such as network update optimization and scheduling [150, 151, 152] and ”Ships in the Night” ( Ships in the Night (SITN)) strategy [153, 154] emerged.

4.1.2 Network updates between autonomous systems

Network update within an AS is associated with state distribution among network nodes sharing similar network administration policy. However, the core of network updates among ASs is based on information propagation in terms of route distribution. Therefore, any routing convergence problem, network state inconsistencies, or anomalies in the network behavior potentially originate from route distribution issues [155]. Since MPLS traffic between ASs is managed by routing protocols such as BGP, applying solutions discussed in section 4.1.1 may result in undesirable problems [156] (e.g. loop forwarding). Appropriately, authors in [146] proposed a solution by completing BGP alternate route distribution and forward traffic through them before the routing session gets expired. Enhancing virtual router migration [157], BGP configuration migration [158], and applying SITN to BGP [159] are other examples of proposed solutions in this context. While all the above-mentioned solutions are protocol-dependant, [160] allows routers to run several configurations in parallel by proposing a generalization in the SITN strategy.

4.1.3 Network updates in Sdn environments

In SDN deployments, data plane nodes are dummy devices which rely on the control plane for decision making on packet processing tasks. Due to the logically centralized control plane which maintains a holistic view of the network at any time, issues such as rerouting traffic in case of path failure, route convergence, and reacting to the network state changes can be addressed almost immediately in SDN [161]. In SDN environments, the controller frequently checks the network state while it also enforces security policies and tune network performance. Consequently, the consistent network update that the controller performs not only has an influence on network forwarding behavior, but also affects other network properties such as security and performance.
Network update operations in an SDN can be extended to network characteristics other than forwarding states. This is in contrast to the updates in legacy networks [162, 140]. Through a labeling technique, the authors enforce update consistency in packet granularity. Many research works extend what is proposed in [162, 140]. For instance, [163, 164, 165] propose more guarantees during network updates such as loop-free and lossless update processes with minimum congestion. Another examples are [166, 167, 168, 169] where a customized update algorithm has been devised. These algorithms can selectively relax some of the extra consistency requirements in SDN deployments.

4.2 Network Automation in hSDN

Network automation helps network administrators and network providers configure large network deployments with complex topologies in a fast systematic way. Network automation is based on collecting data from network devices, process them taking into account network administration policies and applying necessary changes to the network devices under control [170]. With the advent of Artificial Intelligence (AI) and Machine Learning (ML), network automation gradually gains more popularity in network control and management solutions [171].
In a hybrid environment, network device configuration files are created consistently through network configuration templating technique. These templates then reliably determine which part of the configuration file needs to be static or dynamic. Decoupling templates from configuration data and leveraging configuration management systems such as Ansible and Salt facilitate automated management of a hybrid network.

4.2.1 Data collection for management plane

Network data collection is another domain in which network automation plays an important role. Data can be pulled from the network nodes (i.e. pull model) or the network nodes can stream data to the data collection agents (i.e. streaming telemetry [172] or push model). Regardless of which data collection model is being used, the interface used to connect to the management plane of the network nodes is of essence. SNMP is a widely deployed protocol which can be used to collect data from network nodes based on the pull model. As Figure 10 illustrates, a network node plays the role of a Network Management Station (NMS) which is an SNMP server. The server collects exposed data from managed network nodes (aka SNMP agents) using Management Information Bases (MIBs). SNMP can fetch MIB information through Get Requests while it also supports updating MIB information using Set Requests. Although SNMP has been around for a few decades, it was not designed as a real-time API to network devices. Therefore, it is gradually being replaced with next generation network data collection tools and management protocols [173].

Fig. 10: Simple Network Management Protocol (SNMP)

Another network management protocol through which management plane can collect data is NETCONF. It is a connection-oriented protocol standardized by Internet Engineering Task Force (IETF). As Figure 11 shows, RFC6241 conceptually partitioned the NETCONF protocol into four layers namely the content layer, the operations layer, the messages layer, and the secure transport layer.NETCONF leverages Extensible Markup Language (XML) to encode data and send it securely between the NETCONF client and the NETCONF server. It uses XML to encode a request in the form of RPC which triggers the execution of prearranged operations in a transaction-based manner [174].

Fig. 11: Network Configuration protocol (NETCONF)

4.2.2 Network auto configuration

Upon successful data collection form different network nodes, various networking tasks can be performed by automation. This brings considerable flexibility and agility to network management solutions and let the solutions keep up with dynamic nature of hybrid network topologies.

Auto configuration of Access Control List (ACL) policies after network topology changes in hSDN are discussed in [175]. In a hybrid network, frequent topology changes may violate network policies in terms of ACL at the interfaces of forwarding devices. This may result in security vulnerabilities and network performance degradation. Consequently, the authors propose a new approach for hSDN that auto-detects the interfaces of forwarding devices in addition to the network policies that are affected due to changes in the network topology. Afterwards, a graph which represents the topology of the network is created and a graph difference technique is applied to detect changes in the topology. The network management system then uses the graph and its analysis results to verify policies for updated topology. If there is any violation in policy implementation, affected interfaces along with associated policies are detected and necessary revisions are being applied.

Auto configuration of SDN data plane in a hSDN deployment proposed by Katiyar et al. [174]. When a new SDN switch attaches to a hSDN network, automation of the switch initial configuration reduces the installation costs and maintains the consistency of the updated hybrid network. However, this is a challenging task due to the heterogeneity of a hybrid network. Katiyar et al. achieved this goal by detecting a new SDN switch in a hybrid network. It then provides the switch with appropriate configuration parameters and ensures seamless service among SDN and non-SDN network segments. This solution also introduced DHCP-SDN as an extension of Dynamic Host Configuration Protocol (DHCP) to support SDN compatible nodes. It enables a new SDN switch to automatically start operating even though the intermediate switches are not configured with the SDN-specific VLAN setup.

4.3 Reliability, Resiliency, Fault-tolerance, and Load balancing

A hSDN can be seen as a distributed system. Hence, various distributed systems issues including reliability, resilience, LB and fault-tolerance need to be managed for seemless operation of the network as a whole. In this section, we briefly explore existing research on these topics.

Chu et al. [176] propose an approach to guarantee traffic reachability in the presence of any single link failure. By redirecting traffic on the failed link to SDN switches through pre-configured IP tunnels, the proposed approach is able to react to the failures quickly. With the help of coordination among SDN switches, it is also able to explore multiple backup paths for failure recovery. This allows the proposed approach to avoid potential congestion in the recovered network by choosing proper backup paths. The proposed scheme requires a small number of SDN switches in the hSDN network to achieve fast recovery and guarantee 100% reachability from any single link failure. The proposed approach is able to better load-balance the post-recovery network compared to IP Fast Reroute and shortest path re-calculation.

Yasunaga et al. [177] propose a load balancing (LB) method for symmetrically routed hSDN networks that can handle existing distributed routing parameters, link cost and path selection. The proposed method optimizes the link cost and the path selection simultaneously under a symmetrically routed condition, while traditional methods optimize them individually. The LB performance of the proposed method is better and more stable than any of the traditional methods. Furthermore, the proposed method has high versatility and provides a good LB performance even in networks without distributed routing protocols.

Vissicchio et al. [142] study the problem of computing operational sequences to safely and quickly update arbitrary networks. They characterize cases, for which this computation is easy and propose a generic sequence-computation approach, based on two new algorithms that they combine to overcome limitations of prior proposals. The proposed approach always finds an operational sequence that guarantees strong consistency throughout the update with very limited overhead. Moreover, it can be applied to update networks running any combination of centralized and distributed control-planes, including different families of IGPs, OpenFlow or other SDN protocols, and hSDN networks. The proposed approach therefore supports a large set of use cases, ranging from traffic engineering in IGP-only or SDN-only networks to incremental SDN roll-out and advanced requirements in partial SDN deployments.

Shozi et al. [178] study multi-path LB in a hSDN data plane. The load balancer’s performance is tested using four different hSDN topologies, namely, Fat Tree, Ring, Mesh and Torus topology with the OpenDayLight controller. The main contribution of this work is balancing the load in congested data plane links (either before or after network failure) in hSDN environments. The obtained results display an improved overall network performance and reduced network delay.

Link fault protection becomes a key problem in hybrid networks with both SDN and non-SDN switches. Existing solutions need large computation time and high configuration overhead (e.g., tunnels and flow table entries for each switch). To address the above limitation, Jia et al. [179] propose a solution named Hybrid-Hie to achieve fast reroute and load balancing in hSDN networks. The proposed solution configures the split ratio on each backup path in advance by predicting the link utilization. It can achieve efficient fault recovery for inter-domain links, and achieve better load-balance and recovery path stretch.

Yang et al. [180] aims to minimize the repair path length to reduce the delay experienced by the rerouted traffic and saving network bandwidth. The key enabler is a tunneling mechanism for constructing multi-tunnel repair paths by leveraging multiple SDN switches. Multi-tunnel repair paths provide additional flexibility in rerouting and a new SDN candidate selection algorithm is then designed to take advantage of them. To further reduce the repair path length, they propose to replace the link-based tunneling adopted by fault-detection routers by destination-based tunneling. If the network traffic distribution is available, they further show that destination-based tunneling can be used to avoid network congestion. A brief summary of related issues including reliability, resilience, fault-tolerance, and load balancing is given in Table III.

Article Objective Technique
Chu et al. [176] Single Link Failure Recovery Pre-configured IP tunnels
Yasunaga et al. [177] Load Balancing Joint optimization of link cost and path selection
Vissicchio et al. [142] Safe Update Compute operational sequences for safe update
Shozi et al. [178] Load Balancing Multi-path load balancing
Jia et al. [179] Link Fault Protection Fast reroute and load balancing
Yang et al. [180] Link Failure Protection Minimize repair path length
TABLE III: Reliability, resilience, fault-tolerance, and load balancing in hSDN

5 Traffic Engineering

Traffic Engineering (TE) aims to measure and analyze RT network traffic, and design routing mechanisms to improve utilization of network resources. In TE, a wide range of optimization techniques is applied to achieve maximum network performance parameters. In traditional networks, most common traffic engineering mechanisms are MPLS [181] which uses short path labels and GMPLS[182] which extends MPLS to manage further class of interfaces and switching technologies. In SDN, a centralized controller communicates with the forwarding elements and maintains global network views, network topology, traffic demand and link state information to determine routing paths. Capability of fine-granularity network control over flows renders SDN a leading candidate for many TE solutions and several mechanisms have been proposed for SDN networks [183, 184]. In this section, we provide an overview of TE mechanisms proposed for hSDN networks.

5.1 Traffic Measurement

Traffic measurement includes monitoring, measuring and collecting network status information. Traffic measurement is challenging in hSDN since only a subset of routers are SDN capable and legacy routers do not have the flexibility in traffic statistics collection. In this section, we focus on traffic measurement schemes proposed for hSDN.

Measurement of all the flows in hSDN is too costly and not really necessary. Cheng et al. [185]

propose a method to collect load of a subset of significant links and estimate the rest with estimation error of 5%. Polverini et al. 

[186] provides definition and assessment of an effective criterion, based on the flow spread parameter, to identify the flows to be measured that minimise the estimation error. Experimental results show that a small percentage of flows are enough to drive the estimation error an order of magnitude lower than that obtained with the classical solution solely based on link load measurements. The proposed algorithm is also able to distribute measurement tasks fairly among network nodes, taking into account the available forwarding tables space.

Another challenge in hSDN is measurement of RT link-loads. The latency for collecting the global link-load information can be prohibitively long in a Wide Area Network (WAN) due to the long routing protocol convergence time. Cheng et al. [185] propose a comprehensive traffic monitoring method for collecting RT load information of all links. In this method, the controller only needs to collect the load of a small subset of significant links and then, it estimates the link load of the rest. Minimum number of SDN routers is placed to cover these links. Experiment results show that the proposed method can quickly estimate the global link load at an estimation error rate of 5% within sub-seconds. Compared with state-of-the-art methods, the proposed method adapts better to dynamic traffic changes and can reduce the maximal link usage by up to 40%.

Recently, new paradigms are investigated to select critical flows for the traffic matrix. Zhang et al. [187]

propose a reinforcement learning based scheme for this purpose. Reinforcement learning is dynamically learning by adjusting actions based on continuous feedback to maximize a reward. To mitigate the impact of network disturbance, one interesting

TE solution is to forward the majority of the traffic flows using Equal-Cost Multi-Path and to selectively reroute a few critical flows using SDN to balance link utilization of the network. However, critical flow rerouting is not trivial due to the vast solution space for critical flow selection. Moreover, it is impossible to design a heuristic algorithm for this problem based on fixed and simple rules, since rule-based heuristics are unable to adapt to the changes of the traffic matrix and network dynamics. Zhang et al. [187]

propose a reinforcement Learning-based scheme that learns a policy to select critical flows for each given traffic matrix automatically. The proposed scheme then reroutes these selected critical flows to balance link utilization of the network by formulating and solving a simple Linear Programming problem. Extensive evaluations show that the proposed method achieves near-optimal performance by rerouting only 10%-21.3% of the total traffic.

Monitor placement for network measurement is vital for effective measurement of link latencies. Tian et al. [188] investigate link latency measurement problem in scenarios where conventional routers can only support the shortest path routing protocol, and where conventional routers can support source routing protocol. For both scenarios they show how to deploy a minimum number of monitors and how to construct measurement paths between monitors to measure all the link latencies. Several algorithms are presented to solve these problems and the evaluations on different topologies show the performance benefits of the proposed methods.

A brief summary of traffic measurement schemes for hSDN is given in Table IV.

Article Objective Technique
Traffic measurement
Polverini et al. [186] Identify flows that reduce Flow spread based algorithm
estimation error most
Cheng et al. [185] Compressive traffic monitoring Collect load of small subset of links,
estimate the link load of the rest
Zhang et al. [187] Selection of critical flows Linear programming approach
for traffic matrix using ML
Tian et al. [188] Link latency measurement Minimize number of monitors
Traffic management
Casado et al. [189] Vendor-neutral network model Use MPLS for hSDN
Tu et al. [190] Mitigate congestion Tunnel splicing between MPLS and SDN
Sugam et al. [191] Improve network utilization Fully Polynomial time approximation schemes
He et al. [192] Optimize TE performance Provable approximation guarantees
Ren et al. [193] Distributed algorithm for TE Integer Linear Programming formulation
Guo et al. [194] Minimize maximum link utilization Change weights and flow-splitting ratio
Nakahado et al. [195] Decrease network congestion Edge nodes replaced with SDN switches
Sharma et. al. [196] Integrated network management Integration of dynamic control for flows
and control system
Galan-Jimenez et al. [197] Traffic matrix assessment Mixed measurement and estimation
Ren et al. [198] Minimize maximum link utilization Jointly determine appropriate switch
and splitting fractions
Guo et al. [199] Optimize migration sequence Genetic algoritm based heuristic
of legacy routers
Wang et al. [200] Forwarding graph construction Forwarding graphs with high throughput
Guo et al. [201] Optimize routing over multiple Problem NP-hard. Heuristic algorithm
traffic matrices
Davoli et al. [202] TE in IP carrier network Segment routing TE model
Seremet et al. [203] Investigate Benefits of Comparison of scenarios with/out SDN
Segment routing for TE
Wang et al. [204] Heterogeneity in forwarding SDN placement to enhance controllability
characteristics in hSDN
TABLE IV: Summary of Traffic Measurement and Traffic Management

5.2 Traffic Management

Traffic management investigates how to manage and schedule network traffic based on the network status information provided by traffic measurement. In this section, we focus on traffic management schemes proposed for hSDN.

Casado et al. [189] argue that an ideal network design would involve HW that is simple, vendor-neutral and future-proof and the ideal Software (SW) control plane should be flexible. They indicate that SDN falls short of these goals and propose Fabric which is a better form of SDN by retrospectively leveraging the insights underlying SDN.

Tu et al. [190] propose a tunnel splicing mechanism for heterogeneous networks with MPLS and OF routers. Two key mechanisms are suggested. The first abstracts the underlying network devices into uniformed nodes in order to shield the details of various equipments. The second strips the manipulation of flow table and MPLS label switch table from controller and fulfills it in an independent module. The proposed paradigm has been developed on a Linux system and tested in experiment networks. Emulation results are also provided to show its feasibility and efficiency.

A hybrid network model which encompasses both SDN and MPLS-based functionality is presented and experimentally evaluated with the help of a prototype implementation in [205]. This model gives the Proof of Concept (PoC) for the co-existence of MPLS and SDN-based network elements in the network. The switches can be configured using a centralized controller to monitor and push configurations. The model is able to accomplish conflict-free separation of centralized and decentralized controls. Also, a centralized controller is able to provision traffic flows better as compared to decentralized based approaches using MPLS. Therefore, this model provides a better alternative for the smooth transition of a legacy network to an SDN.

Sugam et al. [191] investigate how to leverage the centralized controller to get significant improvements in network utilization as well as to reduce packet losses and delays. They formulate the SDN controller’s optimization problem for TE with partial deployment and develop fast fully polynomial time approximation schemes (FPTAS) for the problem resolution. The reason for using FPTAS is that it is simpler to implement and runs significantly faster than a general linear programming solver for medium and large size problems. They show that deploying even a few strategically placed SDN forwarding elements in the network can lead to improved network performance.

He et al. [192] investigate near-optimal TE to optimize the TE performance over all network links shared with uncontrollable conventional traffic. Two modes are investigated. In barrier mode two forms of traffic (conventional and SDN) are routed in separated capacity spaces. In hybrid mode each link can be fully occupied by either form of traffic. Fast algorithms are proposed for both scenarios with provable approximation guarantees.

Ren et al. [193] propose a distributed algorithm for near optimal TE. In hSDN, redirecting flows from every source-destination pair through at least one SDN node, can improve TE performance and obtain flow manageability, while on the other hand leading to increasing demands of TCAM resources in SDN nodes. They formulate the TE problem as an Integer Linear Programming (ILP) model to minimize Maximum Link Utilization (MLU) and solve it in a centralized manner, where SDN waypoint selection and splitting fractions for each flow are jointly determined. They develop a distributed algorithm deriving from Lagrangian decomposition theory to effectively solve the TE problem. The simulation results indicate that, when 30% of the SDN nodes are deployed, the proposed TE-aware distributed routing algorithm obtains MLU comparable to that of full SDN, and has a limited influence on the routing efficiency.

Guo et al. [194] propose the SOTE algorithm in which OSPF weights and flow-splitting ratio of the SDN nodes can both be changed. The controller can arbitrarily split the flows coming into the SDN nodes. SOTE can obtain a lower MLU and performs better compared with legacy network and the hybrid network with fixed weight setting. SOTE shows that near optimal performance can be achieved when only 30% of the SDN nodes are deployed.

Nakahado et al [195] propose a hSDN strategy, where only edge nodes need to be replaced by SDN switches and other nodes obey conventional OSPF routing. Only edge nodes distribute the traffic incoming to the OSPF network from other networks, and other nodes operate OSPF routing. This technique can decrease network congestion ratio.

Sharma et al. [196] argue that in future networks, traditional network management techniques and controller-based mechanisms need to co-exist and work in an integrated manner to enable incremental deployment of SDN capabilities in legacy networks. They propose an integrated network management and control system framework that combines legacy network management functions such as discovery, fault detection with the end-to-end flow provisioning and control enabled by SDN.

Galan-Jimenez et al. [197] focus on Traffic Matrix (TM) Assessment problem from the perspective of an Internet Service Provider. Since the migration from legacy IP to fully-deployed SDN networks needs to be incremental due to budget and technical constraints, they propose a mixed measurement and estimation scalable solution for hybrid IP/SDN networks to accurately solve the TM Assessment problem by exploiting the availability of flow rule counters in SDN switches. The performance evaluation shows that the proposed error-tolerant solution is able to assess the TM with a negligible estimation error by only measuring a small percentage of traffic flows, overcoming other state-of-the-art algorithms proposed in the literature. The performance analysis of the proposed implementation using an emulated network environment shows that a trade-off between the quality of the assessed TM and its impact on the network in terms of control messages can be found by properly tuning the number of measured flows.

Ren et al. [198] propose a flow routing and splitting algorithm that jointly determines an appropriate SDN switch for every flow as the waypoint, and optimize the traffic splitting fractions for every SDN switch among its outgoing links to minimize the MLU. Simulation results with various SDN deployment rates indicate that, when 20% of the SDN switches are deployed, the proposed algorithm can obtain a lower MLU compared with other state-of-art works. The proposed algorithm can generate paths that are about 50% longer for every flow on the average. However, it has a limited influence on the routing efficiency.

Guo et al. [199] search for an optimal migration sequence of legacy routers to SDN-enabled routers to determine the order the routers should be migrated. They propose a genetic-heuristic algorithm to find a migration sequence of the routers that maximizes the TE

benefit. They evaluate the algorithm by conducting simulation experiments, making comparison to the greedy migration algorithm and static migration algorithms. The experiments show that the genetic algorithm outperforms the other migration algorithms in searching for a migration sequence. When properly deployed, about a migration of 40% of routers reaps most of the benefit.

Wang et al. [200] argue that the effectiveness of TE in hSDN strongly depends on both the structures of forwarding graphs and traffic distribution, while existing approaches mainly focus on the latter. They define the consistent forwarding graph, and then construct forwarding graphs with potential high throughput for effective TE, while maintaining forwarding consistency. The evaluation results show that the proposed forwarding graph construction approach improves network throughput and achieves better load balancing compared with existing simple forwarding path constructing approaches, especially with more fraction of SDN deployment.

Guo et al. [201] formulate the problem of optimizing routing for both the average case and worst case over multiple traffic matrices. They prove the problem is NP-hard and propose a heuristic algorithm and evaluate the algorithm with real traffic datasets. Through extensive experiments, they observe that the worst case performance of routing can be dramatically improved by about 30% with a little sacrifice of the average case performance by about 2% and demonstrate the effectiveness of their algorithm in optimizing both the average case and worst case performance of routing.

Segment Routing (SR) is also proposed for TE in SDN. Davoli et al. [202] have examined the TE problem in the context of an IP carrier network and proposed an SDN-based segment routing TE model. SDN controller allocates network traffic to the links according to link capacity. For flow assignment, a heuristic based approach minimizing network traversal time is used. Seremet et al. [203] shows the benefits of TE in SDN by comparing two scenarios: Segment routing in classical IP/MPLS networks and segment routing in IP/MPLS networks with SDN controller.

Wang et al. [204] propose a solution to handle the heterogeneity caused by distinct forwarding characteristics of SDN and legacy switches. They plan SDN placement to enhance the SDN controllability over the hybrid network, and conduct TE considering both the forwarding characteristics of SDN and legacy switches. The experiments with various topologies show that the SDN placement planning and hybrid forwarding yield better network performance especially in the early 70% SDN deployment.

A brief summary of traffic management schemes for hSDN is given in Table IV.

5.3 Quality-of-Service Routing

SDN provides an open control interface to support flexible network traffic scheduling strategies rendering it the ideal framework for satisfying QoS requirements of many applications such as voice and video.

Egilmez et al. [206] propose OpenQoS, a design based on an OF controller for multimedia delivery with end-to-end QoS support. It is based on QoS routing where the routes of multimedia traffic are optimized dynamically to fulfill the required QoS. They measure the performance of OpenQoS over a real test network and compare it with the performance of the current state-of-the-art, HTTP-based multi-bitrate adaptive streaming. Experimental results show that OpenQoS can guarantee seamless video delivery with little or no video artifacts experienced by the end-users. Moreover, unlike current QoS architectures, in OpenQoS the guaranteed service is handled without having adverse effects on other types of traffic in the network.

Ongaro et al. [207] exploit the use of SDN in conjunction with the OF protocol to differentiate network services with quality level assurance and to respect agreed SLAs. They define a Management and Orchestration (MANO) architecture that allows to manage the network in a modular way. They provide a seamless integration of the proposed architecture and the SDN following the separation between the control and data planes. They provide an ILP formulation of the problem of enhancing QoS and Quality of Experience (QoE) in SDN networks in terms of packet loss and delay, taking into account network constraints and the requirements of real-time applications, i.e., maximum acceptable packet loss and delay rates.

HiQoS [208] provides QoS guarantees using SDN. Moreover, HiQoS uses multiple paths between source and destination and queuing mechanisms to guarantee QoS for different types of traffic. Experimental results show that the proposed HiQoS scheme can reduce delay and increase throughput to guarantee QoS. In addition, it recovers from link failure very quickly by rerouting traffic from failed path to other available path.

Sieber et al. [209] demonstrate an implementation of a Network Services Abstraction Layer (NSAL) on top of the network control and management plane. They introduce a unified data model for both SDN and legacy devices that allows managing and configuring both networks in a unified way in order to achieve QoS for time-critical tasks (e.g. VoIP). Due to the unified data model, network operators are able to manage their network through one interface. They demonstrate a use case by implementing a VoIP scheduling application on top of the NSAL and evaluate VoIP call quality in a distributed heterogeneous network.

Lin et al. [210] propose a QoS-aware adaptive routing (QAR) in multi-layer hierarchical SDN networks. Specifically, the distributed hierarchical control plane architecture is used to minimize signaling delay in large SDN networks via three-levels design of controllers, i.e., the super, domain (or master), and slave controllers. The QAR algorithm is proposed with the aid of reinforcement learning and a QoS-aware reward function, achieving a time-efficient, adaptive, QoS-provisioning packet forwarding. Simulation results confirm that QAR outperforms the conventional Q-learning solution and provides fast convergence with QoS provisioning, facilitating the practical implementations in large-scale software service-defined networks.

Bi et al.[211] consider a hybrid Industrial network consisting of conventional routers (e.g., running OSPF) and SDN-enabled switches (e.g., running OF), and propose an intelligent QoS-aware forwarding strategy to improve QoS in industrial applications, by utilizing a single path minimum cost forwarding scheme and a K-path partition algorithm for multipath forwarding. Simulation results demonstrate that the proposed scheme guarantees the QoS requirements of industrial services and efficiently utilizes bandwidth resources by balancing traffic load in the SDN/OSPF hybrid industrial network.

Mondal et al. [212] study the problem of QoS-aware data flow management in the presence of heterogeneous flows. The optimal data flow management in this is NP-hard and they propose a game theory-based heterogeneous data flow management scheme, that is capable of reducing network delay by 77–98%, while ensuring 24–47% increase in network throughput.

Chienhung et al. [213] propose an SDN hybrid network architecture which can discover existence of legacy switches using the Spanning Tree Protocol to have global view of the SDN hybrid network. Authors enable OpenFlow switches to cooperate with legacy switches by using the Learning Bridge Protocol without requiring any modification on legacy switches. By utilizing the characteristics of SDN, SDN applications can dynamically find routing paths according to pre-defined QoS requirements and current network status.

Alouache et al. [214] propose a hSDN based geographical routing protocol with a clustering approach for vehicular networks. It takes into account three different criteria to select the best relay to send data: (1) the contact duration between vehicles, (2) the available load of each vehicle, (3) and the log of encountered communication errors embedded in each cluster head. The multi‐criteria strategy allows the selection of the most reliable vehicles by avoiding communication problems and ensuring connection availability. Once the hybrid control plane has finds the next eligible neighbor, the data plane is in charge of dividing and sending data. Simulation results show that proposed scheme A achieves good performance with respect to the average routing overhead, the packet drop rate, and the throughput.

Article Objective Technique
Egilmez et al. [206] Multimedia delivery Dynamic optimization
Ongaro et al. [207] Meet service level agreements ILP formulation
HiQoS [208] Differentiated services for Multi-path and queuing
bandwidth guarantee
Sieber et al. [209] Simpler configuration of hSDN Unified data model
Lin et al. [210] Multi-layer hierarchical SDN Minimize signaling delay
Bi et al.[211] Hybrid industrial network Multipath forwarding
Mondal et al. [212] Data Flow Management Game theory based scheme
Chienhung et al. [213] Discover legacy switches Spanning tree protocol
Alouache et al. [214] Geographical routing multi-criteria strategy
Hin et al. [215] Energy-aware routing Safely turn off equipment
TABLE V: Summary of QoS solutions in hSDN

Hin et al. [215] propose an algorithm to enable energy aware routing in hSDN. Since in real life, turning off network equipment is a delicate task as it can lead to packet losses, proposed work provides several features to safely enable energy saving services: tunneling for fast rerouting, smooth node disabling and detection of both traffic spikes and link failures. Proposed work is validated by extensive simulations and by experimentation.

A brief summary of QoS schemes for hSDN is given in Table V.

6 Hybrid SDN in Emergent Networks: Implementation and Deployment

During the 2010s the scientific and academic community has experienced a period of development of new and innovative network technologies with evolving complexity and requirements e.g., Next Generation Mobile Communication Networks (5G), Network Slicing (NS) [216], NFV [217], Cloud Computing [218], IoT [219], Blockchain [220], etc. This section analyzes the impact and use of the hSDN concept over various of these emergent technologies.

6.1 5G Mobile Communications

5G next generation mobile communication technology has been raised as the response to the increasing volume of traffic and number of connections generated by mobile communication networks and also as a need to fulfill the requirements that enable the deployment and implementation of new use cases for mobile communications and even new applications for Industry 4.0.

Fig. 12: IMT-Advanced (4th Generation) VS. IMT 2020 (5th Generation) key capabilities according to ITU-R M.2083

The International Telecommunication Union Recommendation (ITU-R) M.2083 [221] distinguishes three main usage scenarios for this technology: enhanced Mobile Broadband (eMBB) [222], massive Machine-Type Communications (mMTC) [223] and Ultra-reliable and Low Latency Communications (uRLLC) [224]; each of them with different requirements ranging from high data rates (+50Mbps) and extensive cell coverage (+10 Tbps/Km2), to low delays (<5ms) or low data transmission rates. 5G is a complex technology that requires a flexible architecture that provides the possibility to implement all the previously-mentioned requirements through the usage of novel technologies e.g., Multiple-Input and Multiple-Output (MIMO) [225], NS, NFV, SDN and, potentially, hSDN. This subsection will survey the use of the hSDN paradigm over these architectures and study the impact on its key-technology enablers (NFV, NS and Multi-Access Edge Computing (MEC) [226]).

6.1.1 Architectures and technologies

As mentioned in previous sections, SDN proposes the separation of the control and data planes enabling a centralized control of all the data flows and paths inside a network domain, that is, the fundamental role of the CUPS concept introduced by the 3rd Generation Partnership Project (3GPP) TS 129.244 [227] for Long-Term Evolution (LTE). This straight relation between the SDN technology basis and the CUPS concept was one of the main triggers in order to stablish SDN as a key-enabling technology for 5G. The standardization of the SDN architecture was raised by the ONF firstly in June 2014 in Technical Recommendation (TR)-502 [45]. It has evolved throughout different TRs until the current definition has been achieved, which is divided in various TRs: TR-518 [228], TR-521 [229] and TR-522 [230]. The ONF released the aforementioned architecture and several 5G-related Standard Definition Organizations took it as an input for the construction of the 5G architecture. In fact, it is worth to mention that the overall 5G technology evolution is flourishing around the definitions and standards of multiple SDOs e.g., 3GPP, 5G Infrastructure Public Private Partnership (5GPPP), European Telecommunications Standards Institute (ETSI), IETF, Next Generation Mobile Networks (NGMN), etc. This combination is generating several benefits, since never before in the history of mobile communications have so many SDOs get involved around the same technology; however, there are also some disadvantages derived from this approach which lead to collisions between the definitions of 5G concepts and to a slower developing road-map of the overall technology.
As a result of this conjunction of standards and definitions sources, there are numerous proposed 5G architecture recommendations with a strong inter-dependence of the involved technologies. The leading 5G architecture is the one defined by the 5GPPP in [231]. It is based on the 5G architecture defined by the 3GPP in the TS 23.501 [232] but enhancing it with the addition of key-enabler technologies: SDN, NFV and Network Slicing (NS). It explores three levels of network programmability through the usage of the SDN technology: the data plane, Transport Network and Network Function in Radio Access Network (RAN). The data plane programmability is the most dependent on SDN features as it requires to be fully integrated with the MANO plane and agnostic to the underlying HW infrastructure. Figure 13 shows the 5GPPP proposed SDN architecture for the data plane programmability. This architecture is divided in three layers: (1) WAN Resource Manager, the SDN Application in charge of triggering the SDN control plane operations and translating the orchestrator external link information to network domain-specific paths; (2) SDN Controllers, allocated in the network and RAN domains (each of them supported by SDN agents on the respective domain); and (3) the data plane, comprised of LTE small cells, Core and Edge Network Function Virtualization Infrastructure (NFVI) and Wireless LAN (WLAN) Access Points. Moreover, the advantages of introducing the SDN solution in this technology helps in the interconnection of VNFs and Network Slice instances. However, this architecture was one of the first steps towards the integration of all these technologies inside a single architecture and is not fully SDN-enabled (current SDN/NFV architectures integrate the SDN controller in the Virtual Infrastructure Manager (VIM) and derive the SDN operation management to the Network Function Virtualization Orchestrator (NFVO)).

Fig. 13: 5GPPP SDN Architecture for the data plane Programmability in the 5G Architecture (Source [231]).

SDOs have developed several 5G architecture recommendations which allow the integration of these technologies e.g., ONF’s SDN Architecture for NS TR-526 [233], ETSI-NFV architecture for SDN integration with NFV [234] or ETSI-MEC modified architecture for NFV/SDN integration with MEC [235]. These architectures do not assume the hSDN paradigm as a default but leave room for implementing the hSDN approach instead of the pure SDN vision.
The research community has made great advancements in this topic also, the first approaches were towards unifying SDN and NFV technologies in order to evolve the SDN-agnostic architectures to fully SDN-enabled architectures [236, 237, 238]. Another intermediate step before reaching 5G hSDN architectures was adding to the previous bundle of technologies the NS technology as explored by [239] and, finally, integrating all those technologies with the current 5G mobile communications architecture proposed by the 3GPP (see Table VI):

Article Brief Description
Ordonez et al. [239] Presents the key-enablers to allow the integration of NS in the SDN architecture proposed by the ONF [45] and theoretically demonstrate the what is needed and what may be supplied by NFV in order to implement it.
Yousaf et al. [20] Surveys the challenges and issues of the current 5G network requirements. Presents each technology advantages/disadvantages and explores several SW solutions for the different elements that comprise the 5G network e.g., MANO, VIM, Virtual Network Function Manager (VNFM), etc.
Sun et al. [240] Articulates the challenges around novel technologies such as MIMO, mmWave, HetNet… and their impact in the overall 5G architecture. Besides, they develop a fully functional architecture that combines the NFV, SDN, 4G LTE and Software Defined Radio (SDR) technologies.
Trivisonno et al. [19] Proposes a flexible 5G architecture focused on the challenges presented by Vehicle to Vehicle (V2V) and IoT scenarios that combines SDN and NFV with a MEC module. The proposed 4G data plane instantiation within the 5G architecture requires that the SDN data plane acknowledges how to interact with legacy protocols; therefore, this is the first partial (lacks the NS technology) 5G academic architecture towards the hSDN paradigm.
Barakabitze et al. [241] Introduces the concept of QoE-Softwarization through QoE-aware SDN-NFV architecture for delivering multimedia services over 5G networks.
Bouras et al. [242] Small survey about the combination of the NFV and SDN technologies in 5G environments.
TABLE VI: 5G SDN/NFV Academic Architecture Proposals

The final step is the constitution of fully hSDN 5G architectures that integrate all the aforementioned technologies. Irshad et al. [243] propose a 5G hybrid architecture based on optical wireless network units that aims to easily unify legacy technologies with 5G-enabled technologies all-at-once. To demonstrate the capabilities of the proposed architecture, they conduct several simulations using VMware, MiniNet [244] and the FloodLight [245] SDN controller. These demonstrations show that this theoretical architecture outperforms, at a data rate level, those architectures based on a multi-RAN approach.
A similar example can be found in [246]. In this case, the main objective is to virtualize the LTE Enhanced Packet Core (EPC) using SDN and NFV. In this architecture, the combination of both technologies is applied on each gateway in order to find the optimal path towards different sets of bearers, with the same QoS class identifier without any consequence over the QoS. Finally, two instances of a direct application of the hSDN paradigm in 5G architectures are [247] and [248]. The first of them surveys the current challenges of applying this combination of technologies to SFC (another key-enabler in 5G fully-virtualized networks) and, the latter, presents a service-based hSDN model that allows to efficiently manage wireless mesh back-hauls and achieves higher performance if compared with canonical SDN models (rates of x6 less delay).
In summary, these last architectures offer potential solutions to face 5G Key Performance Indicators such as the interfacing between the 3GPP slicing management system and other management systems, such as the Transport Network, by introducing domain-specific SDN controllers (wireless, optical, satellite…) which will allocate transport resources properly according to network slices’ requirements (see 13).

It is worth mentioning that there is a set of projects in the 5GPPP H2020, Phase-2 and Phase-3, Information and Communication Technology (ICT)-17 and ICT-19 EU programmes that aim at creating advanced 5G infrastructures and testbeds, running verticals experimentation trials and testing 5G KPIs in Europe. From these projects, two of them are in charge of generating the 5G infrastructure that will be used by 5GPPP Phase-3:Part 3 projects, such as 5G-Growth [249], 5G-Solutions [250], 5G-Heart [251]. To validate their specific vertical use cases, those are: 5G-VINNI [252] and 5G-EVE [253]. The first one proposes several architectural principles for 5G architectures that are implemented on each of the main facility sites [254], which may be migrated to the hSDN paradigm. On the other hand, 5G-EVE also proposes multiple architectures [255] for the main facility sites in order to be 3GPP Release-16 compliant, which implies the usage of OpenDayligh [96] as the SDN controller.

6.2 Cloud and Data Center Networking

Before the 2010s, the provisioning of computing and networking resources was based on the acquisition of large amounts of HW by application providers in order to create their own DC. The main drawbacks of this approach are the huge costs of buying and maintaining the whole infrastructure and that it is not an easily-scalable or flexible method. The appearance of cloud computing has introduced a shift to a new paradigm, the cloud utility model [218], where application providers do not need to buy and maintain these expensive infrastructures but, instead, are able to lease computing resources with just a couple of clicks and a lesser upfront investment. This section analyzes the current status and impact of the SDN and hSDN paradigms on cloud computing environments.

Currently, cloud computing supports three service delivery models which are usually categorized as follows:

  1. Software as a Service (SaaS): Provides any kind of SW as a cloud solution to cloud customers e.g., e-mail services, Virtual Private Network(VPN) services, schedulers, etc.

  2. Platform as a Service (PaaS): Provides a specific platform so that cloud customers are able to develop multiple operations and manage applications without the underlying complexity of maintaining the required infrastructure for developing an application.

  3. Infrastructure as a Service (IaaS): Provides access to a completely virtualized infrastructure such as Virtual Machines, SDRs and any kind of computing or networking resource.

To provide these services, cloud providers must operate several DCs among multiple locations around the globe. These DCs are comprised of a humongous quantity of switches and servers providing response to application requests through the division of their computing resources in VMs. Therefore, the Data Center Network (DCN) is usually designed is such a way that the interchange of these amounts of traffic between the VMs and containers is done in the most efficient and optimal possible way; however, legacy networks offer a poor solution to these highly complex scenarios as they are not easy to manage and scale. The SDN technology provides a clear separation of the control and data planes with a centralized control logic and, thus, with a global vision of the overall status of the network that enables a flexible, programmable and dynamic solution to the management and scalability issue of the DCNs. There are relevant cloud providers that have already adopted this approach, for instance Google and Microsoft (see Section 6.4.2 SD-WAN and Branch).

Fig. 14: Energy efficient hSDN architecture for DCNs. Edited from source: [256].

Whereas Cloud Computing, DCNs and SDN are highly researched topics, (see Table VII) there are still few published articles that make a specific mention or usage of the hSDN paradigm. Chekired et al. propose ScalCon [257], an hSDN architecture with a decentralized hierarchical control plane focused at efficiently managing DCNs with multiple SDN controllers. This architecture is similar to ParaCon [258], as it is shown bellow, but with some remarkable differences: ScalCon contemplates three different levels for the control plane which may be governed by one or more SDN controllers (Fog layer, Edge layer and Cloud layer); in ParaCon, every controller has links associations with all the switches within the network, ScalCon only allocates links associations with the switches within the corresponding cluster of a layer; and, finally, ScalCon utilizes two asynchronous algorithms with a novel multi-priority queuing model, instead of the one used in ParaCon, in order to improve the performance for path computation. Besides, a comparison between both architectures and SDN controllers ,ONOS [95] and POX [259], is driven at the end of the article (Section IV) which demonstrates that ScalCon outperforms all the previously mentioned schemes in terms of path computation latency, transmission overhead, and end-to-end delay.
Paliwal et al. [256] have created an energy efficient resource management technique intended for those DCs where the traffic patterns are highly correlated to the traffic demand over sets of operational hours. This approach aims at enhancing the advantages of combining hSDN with MPLS to conform real-time dynamic provision of network elements and energy saving in the DCN (see Figure 14 for the architectural view). The core network of the proposed architecture is composed of hybrid switches that are able to interpret both MPLS tags and OF protocol rules; besides, the Network administrator module interacts with the SDN controller through a Northbound interface developed with REST APIs. The method is based on switching specific network elements with small loads off and re-allocating the traffic to an alternate active network element taking into account the traffic demand. Simulations show that energy consumption is reduced approximately a 77% in Fat-Tree and BCube-like DCNs.

Survey Brief Description
Wang et al. [260] Performs an exhaustive exploration of the current impact of different cloud computing architectures depending on the physical DCN topology, the virtualized infrastructure management and monitoring, and the DCN routing. Moreover, a profound analysis is driven towards the capabilities of different kind of DCs: optical, electrical, wireless, SDN-based… It mentions the unique opportunity that the hSDN paradigm brings to SDN-based DCs: reducing deployment costs and overcoming the SDN techniques issues to provide optimized QoS on demand in the cloud.
Son et al. [261] Provides a detailed taxonomy of the state of art of both SDN and cloud computing. The authors address a set of topics that range from SDN-enabled cloud computing architectures to energy-efficient DCNs designs and SDN-based QoS network management methods and policies.
Jararweh et al. [262] Presents a comprehensive survey that aims at exploring the existing Software-Define Clouds and its comprising subsystems (security, storage, networking…). Besides, the authors develop a PoC towards a fully automated software-based control framework for cloud computing environments.
Toosi et al. [263] Mainly focused on the issues that arise, from a cloud-provider perspective, when interconnecting multiple DCs. It describes and provides several solutions for the interoperability scenarios that are generated under the above-mentioned circumstances multiple DCs.
Jain et al. [264] Brief survey that analyses the benefits and drawbacks of SDN/NFV-based DCs. Besides, it presents the initial development and capabilities of OpenADN, an SDN-based network application focused on application partitioning and delivery in multi-cloud scenarios.
TABLE VII: SDN & Cloud Computing surveys

There are some other minor examples of the hSDN applied to Cloud Computing environments .The authors of [265, 266] present a model for the incremental deployment of SDN in ISP and DC networks. This model combines SDN-enabled switches with upgraded legacy switches that are able to interact with the OpenFlow protocol in such a manner that it effectively reduces MLU for congestion mitigation with only 20% deployment of SDN switches. [267] and [71] introduce, respectively, a TE view for SDN/OSPF hybrid network architectures where some OSPF flows and weights are modified from the centralized SDN controller. Besides, a hybrid IP/SDN architecture known as “Source Hybrid IP/SDN (OSHI)” 6 is also described. This hybrid network architecture implements IP routing with the SDN forwarding control in the core of the network.

In conclusion, Cloud Computing and DCNs are crucial pillars of the current technological evolution towards ”Everything as a Service” and the necessity of developing networks that are highly flexible and scalable. In order to achieve these goals but without the burden of huge investments, there is an existing dependence between these technologies and the inclusion of the hSDN paradigm in them. hSDN applied in DCNs academic and research projects are starting to emerge as it is a major research area that is even being exploited by the industry (see section 6.4.2).

6.3 Internet of Things

The concept of IoT was born in 1999 by Kevin Ashton [268] in the context of supply chain management. Since then, as other technologies evolved, so did the concept of IoT and, currently, it refers to a network of interconnected devices that gathers environmental information through the interaction with its environment and even uses the existing Internet Services to generate novel communication and application services.
According to the study carried by IoT Analytics [269], in 2011 the number of interconnected devices was greater than the number human beings in the planet. As of writing, in 2020, over devices exist (46% of them are IoT devices – non mobiles phones, laptops, servers, etc.) coexist with the human race and it is expected that more devices will be used by 2025.. As it can be seen, the expected expansion and evolution of this technology has surpassed its initial goals, gathering multiple benefits and advantages that come attached within this technology. In order to achieve those advantages, IoT devices interact with back-end systems (cloud, edge and fog servers…) that process the volumes of data generated by the sensing activities. A critical aspect upon this technology is the implementation of new architectures that are able to adapt the overall network capabilities to the requirements of its inherent extremely dynamic environments e.g., highly scalable, avoidance of new potential attack surfaces and vectors, network programmability, high security for e-Health and Industry 4.0 devices due to potential data flaws, etc.
Several works have been created around IoT architectures: [270, 271, 272, 273, 274]. These architectures are characterized by the softwarization and virtualization of the network and, consequently, SDN and NFV technologies are becoming key enablers for the generation of this second phase of evolved IoT architectures. This section gives an “au courant” status of the IoT deployments that implement the hSDN paradigm.

Pure SDN integration and deployments are being vastly researched and implemented within IoT scenarios to achieve the above-mentioned upgrades over common IoT networks and other applications. Fichera et al. [275] and Tello-Oquendo et al. [276] are two examples of the usability of SDN in 5G IoT networks. Aside from these theoretical architectures, there are lots of experimental implementations which try to solve IoT security flaws. [277, 278, 279] are clear examples of frameworks that exploit SDN capabilities to achieve traffic anomalies detection and mitigation in IoT cloud networks, secure cluster formation for large IoT grids, MITM attack prevention in fog-based IoT scenarios and security improvement through a flow-based SD-GW method, respectively.

Notwithstanding, some researches are leading the approach to the hSDN paradigm in the IoT realm. On first place, Bendouda et al.[280] explore a partial approach towards a complete hSDN IoT deployment, proposing a hSDN architecture that integrates both, a centralized and a semi-distributed approach for the control plane. The “hybrid” concept of this proposal is directly related with the division of the pure SDN centralized control plane into three layers: (1) Main Controller, has a view of the global network status and coordinates lower-level controllers; : (2) Secondary Controller, acts as an entry point to address network constraints related to wireless technologies and routing processes; (3) Local Controller, lower layer controller, its main role is to manage and interchange control packages with the IoT island gateway (these nodes are selected by the Connected Dominating Set algorithm presented in the same article). The resulting architecture is capable of auto-selecting the level-3 local controllers depending on multiple network parameters (distance from gateway, average link quality, etc.) and therefore obtains remarkable performance results if compared with traditional one-layer controller-plane SDN approaches.
Lee et al. [281] discover that the hSDN paradigm may be a good model for large IoT architectures (more than 34 connected devices) if compared with an efficient set of TE, load-balancing and flow management policies in order to reduce the delay and upgrade the throughput of the overall network. The implementation of a hSDN scenario was developed due to the traditional throwbacks of pure SDN scenarios e.g scalability, availability, reliability and cost. This architecture is comprised of two kind of networks: Distributed Networks, composed of legacy devices that support OSPF and SDN-enabled networks, composed by hybrid or SDN devices. Additionally, those networks are populated with three different types of devices: legacy, hybrid (run IP routing-based protocol stack in the Distributed Network and OF forwarding-based stack in SDN) and SDN. They modify the idea of a basic forwarding table from [282] and apply it to the hybrid nodes using TCAM tables for OF-based routing entries and Static Random-Access Memory (RAM) tables for IP-based routing entries and emulate the architecture using MiniNet [244] with the Floodlight controller [245].
Saadeh et al. [283], on the other hand, propose a hSDN architecture that combines Information Centric Network (ICN) and SDN. The proposed architecture aims to expand PPUSTMAN architecture [284] (see Zhang et al. survey [285] for detailed ICN-SDN architectures information) by separating the controlling function into three controlling planes: Operational, Tactical, and Strategic planes. Additionally, each of those planes is composed of three controlling units: Control, View, and Model. This project delves into the advantages of joining both technologies in IoT scenarios:

  1. Enhancing the performance of the controllers

  2. Generation of more abstraction levels for node interactions

  3. ICNs easier deployments

  4. Better inter-paradigm operability

  5. Decrease in traffic volume using in-network caches for SDN actions publication

Lastly, technologies similar to IoT such as Internet of Vehicles (IoV) [286] are starting to implement the hSDN paradigm. [287, 288] present a clear demonstration of this path towards the research of hSDN in IoV environments.
Once again, as with previous technologies, the IoT realm is not an exception and there is still a lot of work to be done in order to have production-level IoT networks with a full hSDN implementation. However, the first projects combining both technologies are starting to arise and its advantages being discovered.

6.4 Others

6.4.1 Blockchain

Since the implementation of the Blockchain Technology in 2009 by a person (or group of people) with the pseudonym of “Satoshi Nakamoto” [289] for the renamed cryptocurrency Bitcoin, this technology has been applied to numerous applications other than pure cryptocurrencies [290, 291, 291, 292] with great benefits to the scientific community due to its four main characteristics: decentralization, persistency, anonymity and auditability. In the field of legacy and SDN networks, the most important of these characteristics might be the decentralization as it avoids traffic and security bottlenecks in high density networks such as IoT environments, mMTC scenarios, DCNs. But, at the same time, this technology has a major throwback which is crucial for all of these networks, Scalability. Public Blockchain may be addressed as a public distributed ledger or database where event records (transactions) are stored in a chain of blocks that continuously grows as soon as new block are added. Therefore, the blockchain tends to be very heavy if the amount of transactions per day is high which, as a direct consequence, makes this technology not that suitable for large networks.
Fortunately, at this point is where the SDN and the hSDN technologies appear as a pillar that helps to manage this issue in these kind of networks as, combined with Blockchain, they are capable of providing the flexibility, programmability and scalability required by them. In fact, almost all of the papers reviewed in this survey about this topic are related to IoT networks, vehicular networks and vehicle to grid scenarios. This section surveys the current research status and impact of this technology in hSDN scenarios.

In [293], T.M. Fernández-Carames and P. Fraga-Lamas carry-out a research on the applications of Blockchain in IoT architectures (BIoT) and they reference two articles where Blockchain capabilities are upgraded using SDN to control fog nodes. These novel architectures are able to deploy distributed fog nodes with a central SDN controller that enables a huge reduction of the overall delay of the network, greatly increases the throughput and balances the load when flooding attacks are launched against them. As mentioned before, the combination of these innovative technologies leads to impressive architectures that overcome the disadvantages of each of these technologies, if implemented independently e.g., Kumar Sharma et al. [294] propose a distributed SDN fog node-based cloud architecture for IoT, comprised of three layers (device, fog and cloud) that achieves high resiliency, scalability and availability with full decentralization and true redundancy if compared with traditional IoT networks. The key enablers of that architecture are each of the fog nodes which, basically, are small SDN/hSDN controlled blockchain-based distributed networks that are responsible of service delivery, data analytics of each of the device-layer isles and sending the already-processed data back to the device-layer or to the cloud layer if needed.
A similar work called DistBlockNet is proposed by almost the same authors in [295], the main difference between these two projects is that DistBlockNet is conceived as a large horizontal single-layered distributed mesh of SDN controllers, that are able to communicate with each other thanks to the Blockchain technology, instead of a three-layered vertical architecture. Moreover, this architecture is more focused on solving legacy IoT network security issues (reducing the attack window, threat prevention, data protection, mitigation of cache poisoning/ARP spoofing, Distributed Denial of Service (DDoS)/DoS attacks) than on optimizing the overall performance of the network. In any case, although both architectures are presented as pure SDN-controlled architectures, due to their characteristics, a hSDN approach could be implemented as far as the deployed legacy devices do not collide with the generated Blockchain algorithms.
More recent proposals have driven to the application of this novel combination of technologies into the field of energy trading in vehicular networks. [296] presents BEST which is a dedicated Blockchain-based secure energy trading scheme for electric vehicles (EVs) where the blockchain algorithm is applied to validate EVs’ requests in a distributed manner and where SDN is utilized to pass the verified EVs’ requests to a global SDN controller in a flexible, scalable and programmable way. This proposal includes some miner nodes that are the ones in charge of carrying out the validation operation based on several factors (dynamic pricing, connectivity records, etc.). Furthermore, Jindal et al. present SURVIVOR [297], a blockchain based edge as a service framework for secure energy trading in SDN-enabled Vehicle to Grid environments, that merges SDN and Blockchain to achieve a framework with a low-latency network backbone and secure energy transactions based on the proof-of-work method. Once again both frameworks do not collide with the hSDN paradigm. Additionally, [298] and [299] present two projects where these technologies are combined with further technologies such as 5G and vehicular ad-hoc Networks (VANETS) but, in this case, it is not that clear if an hSDN approach can be added.

In summary, Blockchain is a promising technology that all-along has lots of advantages but that, if combined with SDN and hSDN technologies, it enables great advantages if applied in large-scale and dense networks. However, it is still a novel technology if applied within the hSDN paradigm and, therefore, there are not many finished studies that address this topic.

6.4.2 Software-Defined WAN and Branch

WANs are in charge of providing connectivity between cloud services, SaaS applications, DCs and branch locations. In the past years, SaaS and cloud providers have been heavily investing in high-speed peering connections close to the enterprise WAN edge and, consequently, broadband bandwidths have raised considerably. Therefore, the overall cost of maintaining a traditional WAN with legacy protocols such as MPLS, BGP and OSPF has greatly increased. Due to these reasons, many enterprises are migrating to a novel approach, namely, Software-Defined WAN (SD-WAN), that allows them to highly reduce costs, augment the network flexibility and programmability and even extend existing MPLS bandwidth at branches by adding broadband connections and using PBR. This section surveys the integration of the hSDN paradigm in SD-WANs and Software-Defined Branch (SD-Branch). There exist two major and commonly known SD-WAN deployments implemented with the hSDN paradigm: Google’s B4 [13] and Microsoft SWAN [15]. B4 [13] is Google’s inter-DC SD-WAN solution. It aims at customizing a self-defined hSDN SD-WAN architecture that fulfils Google’s DCs unique requirements with a two-level hierarchical control framework: (1) the upper layer allocates a logically centralized TE server that looks over high-level TE policies and; (2) the lower layer is a network controller located in each of the DCs that hosts and manages local control applications and site-level traffic. Google took several design decisions before implementing this massive SD-WAN ranging from developing and building their own routers based on merchant switch silicon, to centralized TE and links with 100% utilization rates. As stated in [13, section 3.4] integrating both legacy and OpenFlow-based routing protocols, was one of the critical challenges in order to support hSDN network deployments, in fact Google needed to develop a Routing Application Proxy (RAP) that allowed the selected routing engine, Quagga [72], to communicate and interact with the OpenFlow switches (BGP/IS-IS route updates, routing-protocol packets flowing between switches and Quagga and interface updates from the switches to Quagga). Figure 15 (b) shows a more detailed view of this procedure.
B4 is a clear example of the advantages of the migration to the hSDN paradigm and of how it can serve to implement real global-level solutions, Hong et al. analyze in [15] the evolution of Google’s hSDN WAN since its beginning up to 2018, when B4 was already comprised of 33 sites (see Figure 15) and supports services with a required availability of 99.99% and with links utilization rates near to 100%.

Fig. 15: B4 Google’s Hybrid SD-WAN. (a) B4 Topology as of January, 2018; (b) OpenFlow Integration with Legacy Routing Protocols. Edited from Sources [15, 13]

SWAN [300] is Microsoft’s SD-WAN inter-DC centralized traffic engineering solution. It is primarily focused on solving the shortcomings of today’s most used TE practices with MPLS in WANs: (1) Poor Efficiency, low volumes of traffic are carried if compared with the total capacity of the links; this leads to a poor link utilization rate (e.g., the average utilization of half the links is under 30% and of three in four links is under 50% in an average production inter-DC WAN); (2), poor sharing, in traditional inter-DC WANs there is a limited support for flexibly resource allocation. To achieve these goals, SWAN comprises a global network view implementing network agents with the FloodLight OpenFlow controller [245], which helps to find globally optimized bandwidth path assignments and uses fine-grained policy rules to carry more high-priority traffic (e.g., interactive traffic) while maintaining the overall fairness among services of the same class. As an example, SWAN establishes a 5-minute period before service brokers exchange data that allows to predict the forthcoming traffic and, therefore, use the theoretically “free” bandwidth for other types of traffic (in their paper, they distinguish between three types of services/traffic: interactive, elastic and background). Consequently, the remaining resources are used more efficiently and effectively.
In the initial design of SWAN, only OF-enabled Switches were used (Arista 7050Ts and IBM Blade G8264s), however, Hong et al. stated in their paper that “(…) We use OpenFlow switches, though any switch that permits direct programming of forwarding state (e.g., MPLS Explicit Route Objects [3]) may be used.”. This statement leaves room to a hSDN SD-WAN implementation. Besides, the initial design was slightly smaller than Google’s approach, with just 5 DCs all around three continents (China, USA and EU); nonetheless, SWAN is able to carry 60% more traffic if compared to MPLS TE link utilization rates and, likewise, only a 4% of the flows deviate over 5% from their fair share. In MPLS TE, 20% of the flows deviate by that much, and the worst-case deviation is much higher.
The main differences between these two hSDN SD-WAN inter-DC implementations draw from the premise that they are facing different challenges. B4 aims at developing custom switches and procedures to directly integrate legacy routing protocols with a pure SDN environment, while SWAN develops mechanisms for the congestion-free data plane updates and for effectively using the limited forwarding table capacity of commodity switches.

Besides, the appearance of IoT environments inside the enterprise branches, the early adoption of mMTC changes and the shift of enterprise workloads to cloud providers has led to big changes in traditional traffic patterns and to VLAN sprawl and an increase in network complexity. These facts require a new concept beyond SD-WAN, a concept that includes the enterprise branch routing, security and Local Area Networks as well. This concept is known as SD-Branch.

The SD-Branch concept has not yet fully impacted in the academic community but, in the enterprise world there are a few solutions starting to arise [301, 302, 303, 304]. Aruba’s SD-Branch solution [301] includes SD-WAN gateways supporting broadband, LTE and MPLS WAN connections, and a cloud-based management Graphical User Interface (GUI) for WAN and security configuration. The branch security is enforced through a stateful firewall based on DPI, IPSec VPNs, Web content classification and reputation and cloud security integration, among others. Cisco’s Network Function Virtualization Infrastructure Software (NFVIS) [302] is a full operating system for the SD-Branch. NFVIS provides data flow acceleration for the VNFs used to implement routing, SD-WAN and security branch services. Fortinet’s Secure SD-Branch [303] is focused on providing a secure and manageable branch for enterprises. It integrates several security-specific modules such as firewall, secure wireless access points, secure Ethernet and network access control. Versa’s SD-Branch [304] relies on the FlexVNF operating system to execute routing, SD-WAN and security functions on Versa boxes (white boxes are also supported). The proposed architecture enables the deployment of automated services for WAN and branch scenarios.
In summary, the SD-Branch concept is still young and needs to be adopted and researched by the academic community as a promising technology for future flexible networks. Moreover, B4 and SWAN represent a clear path of migration from legacy WANs to hybrid SD-WANs and they demonstrate the capabilities and advantages of the hSDN paradigm.

7 Simulation Tools and Testbeds

As in any other technological domain, researchers looking into the hybrid SDN paradigm require Simulation Tools, Emulation Tools and testbeds that allow them to evaluate several hSDN scenarios and its efficiency and behavior over specific conditions. If pure SDN experiments are to be carried-out, the most commonly known tools/platforms are: Mininet, EstiNet, NS-3 and, more recently, MiniNEXT [244, 305, 306, 307]. The most used platform is MiniNet as it implements a lightweight kernel that enables the creation of multiple custom network topologies that integrate diverse devices over a simple laptop. MiniNet defines its SW as a “network emulation orchestration system” that orchestrates several Open vSwitch [41] instances and connects them as defined by the user.
Nonetheless, from a hSDN perspective, these tools lack some crucial functionalities that are key-enablers in order to be able to simulate a hybrid legacy and SDN scenario. Rashid et al. analyze some of these challenges in their hSDN survey [31], stating that the main lacking functionalities are related to the implementation of the required protocols for the communication between the SDN controller and the legacy devices and to the network policy implementation. Moreover, Habib et al. [308] does an extensive research and comparison over the existing SDN simulation tools and testbeds and classify them into three categories depending on how the solution is deployed/implemented: centralized, distributed and remote testbeds. This subsection surveys the state of art in SDN/hSDN simulation tools and testbeds.

Platform Family First Release Year Open Source Architecture Type Legacy Protocols Support Virtualization Technology Scalability
MiniNet [244] Simulation Tool 2010 Virtual Centralized None KVM Low
MiniNet CE [309] Simulation Tool 2013 Virtual Centralized None KVM Medium
Distributed MiniNet [310] Simulation Tool 2013 Virtual Distributed None KVM High
EstiNet [305] Simulation Tool 2013 x Virtual Centralized Low
None (kernel re-entering
simulation)
High
NS-3 [306] Simulation Tool 2006 Virtual Centralized Low KVM Medium
MiniNEXT [307] Simulation Tool 2014 Virtual Distributed Medium KVM Low
MaxiNet [311] Simulation Tool 2014 Virtual Centralized None KVM High
DOT [312] TestBed 2014 Virtual
Distributed (centralized
Orchestrator)
High
Containers (Docker),
QEU/ KVM
High
ONE [313] Both 2018 Virtual Remote High Containers (Docker) High
RISE [314] TestBed 2013 x Hybrid Remote Medium Local - kernel Low
Ofelia [315] TestBed 2014 x Hybrid Remote Medium Xen-based VMs Medium
OTG [316] TestBed 2017 Physical Centralized Low None Low
SDNetkit [308] Simulation Tool 2017 Virtual Centralized High NetKit VMs Medium
NextLAB [317] TestBed 2019 Hybrid Centralized High KVM Low
Kentucky U. [66] N/A 2019 x Hybrid N/A High KVM Medium
TABLE VIII: Simulation Tools and Testbeds Comparison

Table VIII shows a classification of the most significant hSDN/SDN simulation tools and testbeds up to the date. We classify these solutions by:

  1. Family: Distinguishes between “Simulation Tool” and “Testbed” to have a clear view of what kind of operations is the user going to be capable to perform on them.

  2. First Release Year: Clarifies the maturity and evolution of each platform.

  3. Open Source: Distinguishes between OpenSource platforms and commercial or non-OpenSource platforms.

  4. Architecture: Separates platforms into two categories:

    • Virtual: Platforms without a physical network deployment.

    • Hybrid: Platforms with a physical network implementation combined with virtual network elements.

  5. Type: We have used the same classification as in [308]:

    • Centralized: Platforms running locally.

    • Distributed: Platforms running on a distributed environment.

    • Remote Testbeds: Physical networks that offer the possibility of experimenting SDN/hSDN using virtual overlay networks.

  6. Legacy Protocols Support: In hSDN, it is important to be able to communicate with Legacy network elements. We define four levels:

    • None: No support at all of Legacy communication protocols.

    • Low: Only one Legacy communication protocol is supported i.e. EstiNet only supports STP.

    • Medium: More than one Legacy communication protocol is supported but the integration is not fully compatible.

    • High: All or almost all the legacy communication protocols are supported and the integration is fully compatible.

  7. Virtualization technology: Currently this parameter is highly meaningful depending on the kind of experiment and the amount of available resources that a researcher has in order to carry-out its research.

  8. Scalability: Clarifies the degree of scalability of the platform.

As explained above, MiniNet [244] is the most widely known and used tool for SDN simulation. However, its latest stable version (MiniNET v.2.3.0d6) has no support for hybrid SDN scenarios where OpenFlow and Legacy routing protocols coexist. MiniNet Cluster Edition (CE) [309] and Distributed MiniNet [310] platforms enable the execution of multiple MiniNet instances in one single system and to execute MiniNet over a distributed scenario but, as the original SW, both of them lack the functionalities to run other than OpenFlow-controlled devices.
Moreover, EstiNet is the only Commercial platform in Table VIII but, even though it is capable of supporting several SDN controllers as Floodlight, the only accepted Legacy routing protocol is STP. Its leading capability is the usage of the kernel re-entering technique to simulate scenarios and its high scalability (see [305] for further comparison between EstiNet and MiniNet). NS-3, in its latest stable version, enables certain degree of interaction between the simulated components and the real topology-enabled network components (see [306] for further comparison between NS-3 and MiniNet). MaxiNet [311] was created as a MiniNet plugin with the aim of emulating large-scale network with a high density of devices e.g. DC networks, but, as a MiniNet complement, it inherits the lack of hSDN simulation capabilities from the original platform. At last, MiniNext [307] is the only Simulation platform that supports, to some degree, legacy routing protocols. It was designed as an extension layer for MiniNet in order to allow the creation of more complex networks and the integration of traditional routing engines (e.g. Quagga , BIRD…[72, 318]) and NAT and network management components (e.g. DHCP). The main issue of MiniNext is that is no longer supported and it only works with MiniNet v.2.1.0.

Arup Raton et al. [312] created one of the first hSDN distributed testbeds, focused on improving and upgrading MiniNet scalability. The Distributed OpenFlow testbed (DOT) provides the possibility to create an emulated topology over a cluster of multiple physical machines, establishing an orchestration hierarchy in which one of the machines is the so called DOT Manager, in charge of managing the whole emulation process, and the rest of them are the DOT Nodes. The big advantage of this testbed, compared with the previous Simulation Tools, is that it allows the usage of two different kind of virtualization techniques (Linux Containers and Kernel Virtual Machines (KVM)) and adds full support for hybrid-networks, allowing the emulation of non-OpenFlow Switches as well as allowing the inclusion of physical switches, which are key-enablers for hSDN environments.

The Open Network Emulator (ONE) [313] is a Microsoft platform that was open-sourced in 2018 which, originally, was a collaboration project between Microsoft Azure and Microsoft Research teams called CrystalNET [319] (named after a fortuneteller’s crystal ball, as it was going to be able to “reveal” the future of a network) focused on the scope of emulating large-scale cloud networks. Currently, it is an Azure [320] product that has been integrated as an IaaS platform that allows to deploy realistic test environments for building network automation tools and testing the network behavior. ONE is highly scalable as it allows to create the virtual resources using VMs and Containers technology and is highly flexible as it supports the usage of a huge number of networks devices SW images from a wide range of different vendors. ONE also accepts real hardware devices and, hence, legacy routing protocols. These capabilities make this testbed a trustworthy platform for those researchers that are looking to develop hSDN experiments without owning a physical infrastructure.

RISE, proposed by Kanaumi et al. in [314], is a Wide Area OpenFlow Network (WA-OFN) hybrid Japanese testbed built on top of the JGN2plus and Japan GigaBit Network (JGN-X) [321] networks that aims at allowing multiple users/customers to create WA-OFNs over existing real networks (RISE features FlowVisor [322] to manage and control the underlying networks). Although its implementation over existing real networks may seem a good approach to obtain reliable network metrics and behavior, it is technologically limited as some of the devices within these networks do not support MAC-in-MAC and MPLS-based tunneling. RISE users are allowed to create their own wide-area hybrid network topologies in this testbed, namely Existing Virtual Networks, and it also enables the isolation of a user resources by assigning them a logical slice of the network (Network Slice). Some of the conducted demonstrations on the RISE testbed are shown in [323, 324].
The RISE project achieved a federation of networks with other national-level OpenFlow testbeds such as Ofelia (EU) [315] and other European, USA and South-American projects (See Figure 16). Furthermore, based on the collected knowledge and experiences from this testbed, a new testbed was proposed to launch ICT experiments in a distributed, large-scale environment called JOSE [325].

Fig. 16: Ofelia worldmap connectivity scheme (source [315])

OFELIA [315] is a pan-European collaborative OpenFlow testbed that connects individual testbeds by a common Layer-2 infrastructure. It is comprised of seven different facilities (iMinds, U. of Bristol, ETHZ, i2CAT, TUB, CreateNet and CNIT) each of them consisting on a self-defined infrastructure (OpenFlow switches, OpenFlow controllers, legacy switches and computing resources) and with an experimentation specialty ranging from large-scale emulation and NetFPGA farms to Optical OpenFlow Experiments. It incorporates three significant tools: (1) Web-Based GUI to manage user authentication, experiment onboarding requests and network status; (2) Virtualization Technology Aggregate Manager [326] responsible of the system control; and (3) the FlowVisor Aggregate Manager which uses FlowVisor as an agent to provision resources on the OPN and uses a northbound API to communicate with the Web-Based GUI. These tools enable the three fundamental pillars of the OFELIA testbed: resource diversity and flexibility for the experimenters, federation extensibility and ease of experiment orchestration, control and management. Notwithstanding, it is not fully capable of supporting legacy devices.

Joshua A. et al. propose the SDN On-The-Go (OTG) hSDN physical testbed in [316]. Unlike the previous testbeds, OTG main goal is to create a portable and economical solution that fulfills the requirements of those academic researchers willing to implement SDN/hSDN experiments that transcend the Simulation Tools capabilities (precise and repeatable metrics, concise real-world behavior, etc.). To that end, the OTG testbed is comprised of a reduced set of components, namely: four SDN Zodiac FX switches, four RaspberryPi 3 hosts and a Kangaroo+ mini-pc as the SDN Controller. Thanks to this implementation, this testbed converges on a low-cost (full set cost less than $1000), portable and standalone hSDN testbed capable of generating several topologies e.g. Four switches and two hosts partial mesh, Four switches and four hosts hybrid fattree with direct WAN access, etc. As demonstrated in [316] OTG shows a more stable performance when compared with MiniNet if running the same experiments. Additionally, as it is a modular testbed, it can be expanded and various HW extensions can be added to the initial set of components. The big shortcoming of the OTG testbed is that it does not explicitly integrate hSDN support, therefore, if required, it has to be manually added.

Fig. 17: SDNetkit Hybrid Network Topologies examples. Left: Hybrid node topology. Right: ISP complete Hybrid Network Topology. Edited from source [308]

.

SDNetkit is presented in [308] as a solution built over Netkit [327], a freely OpenSource lightweight network emulator based on User-Mode Linux. The main difference between this Simulation/Emulation solution and the previous ones is that it extends the Legacy networking and routing protocols of the Netkit Network Emulator in order to be SDN-capable (it was not conceived as an independent SDN tool). SDNetkit supports OpenFlow 1.3 and various routing engines such as Quagga [72], enabling the creation of interoperability hybrid scenarios where legacy and SDN-enabled devices and protocols coexist. More specifically, this emulation platform is capable of executing Open vSwitch [41] instances with the RYU [328] framework as the controller in parallel with legacy devices.
It is important to remark that SDNetkit is a modular platform that allows to extend its own functionalities by adding new SW to its repositories: “(…) e.g. it is possible to add other SDN frameworks for implementing custom controllers, other SDN-enabled switch implementations, or general software for testing SDN”.

SDNetkit inherits the virtualization technology used by the Netkit emulator which, basically, is a form of VMs that require no administrative priviliges and consume a small amount of computing resources on the host system. As it can be observed in Figure 17, SDNetkit can create multiple different hSDN topologies ranging from: a network topology that uses one or more hybrid nodes (nodes that simultaneously run OpenFlow and legacy routing protocols) to a complete hybrid network topology where legacy devices coexist with SDN-enabled devices. Consequently, all these functionalities, extension possibilities and legacy/SDN support make this platform one of the best choices if hSDN scenarios are to be emulated.

The Nextlab testbed [317], on the contrary, is focused on creating a platform that allows its users to share their resources in order to build a global hSDN community and testing environment. It is conceived with a layered architecture (see Figure 1 in [329]) composed of four main components: (1) Web-GUI, based on OvS-Mesh tool (similar to the GUI of MiniNet); (2) Topology Creator, takes the input topology generated by the user in the Web GUI and translates it into a pre-configured PicOS script that can be executed on Pica8 switches; (3) Open Networking Operating System (ONOS), as the SDN Controller and (4) Pica8 Switch, hosts the virtually created topologies.
NextLab brings, with this architecture, a set of advantages that enable the configuration of hybrid topologies (it uses Open vSwitch instances for the virtual elements)and QoS policies. Besides, due to the usage of ONOS as the SDN controller, users can share their computing resources and add them to the NextLab global network. As a consequence, NextLab users can develop hSDN experiments with minimum cost and contribute to a community where similar experiments are being carried out. The two initial nodes of this tesbed were located at Vitry-sur-Seine (France) and at Hanoi (Vietnam). Chu et al. [317] paper, execute different tests to compare the performance of NextLab and MiniNet. NextLab consumes less CPU in all scenarios.

There are also other hSDN testbeds built with more specific objectives rather than to be used to deploy and test hSDN environments. SDN/OSPF Traffic Engineering (SOTE) by Guo et al. [329] proposes a heuristic algorithm formulated as a Mixed Integer Non-Linear Programming Model (MINPM) which minimizes the MLU, if compared to traditional routing algorithms, comprising OSPF [330] link weight configuration with the traffic splitting ratio in SDN nodes. To demonstrate the capabilities of their algorithm they implement an hSDN testbed with a single SDN controller and several hSDN switches (see Section 6.5 in [329]). Furthermore, the authors of [331] propose a Network-assisted HTTP Adaptive Streaming algorithm to increase the QoE in hSDN networks. To do so, they simulate a hSDN network using MiniNet [244] , the FloodLight Controller [245] and an Open vSwitch [41] operating in standalone mode to simulate a legacy switch.
Shi et al. [66], as detailed in section 2.3.1, have created a hSDN testbed as a real application case over a campus-like topology in order to demonstrate how to integrate the SDN paradigm with a real Legacy Network and all its challenges. In [63] Poularakis et al. describe the development of the testbed using a commercial SDN dataplane, ONOS [95], and Open vSwitch combined with in order to study the control of Mobile Ad Hoc Networks (MANET) and demonstrate its feasibility. Less remarkable but worth to mention, other hSDN testbeds and tools are HybNET, Telekinesis and ClosedFlow respectively [139, 332, 333]. To end, if interested in a deeper view over the current state of art of cross-technological SDN and network emulator testbeds and tools, see Piccialli et al. survey [334].

8 Industry and Standardization Perspectives

White papers by many companies and documents of standardization groups have supported deployment of hSDN. This section briefly describes current state of deployment, white papers for hSDN deployment and documents from standardization groups ONF and IETF.

According to IDC, the North American SD-WAN market was $145 million in 2017 and will grow to $1.8 billion in 2022 [335]. According to 2019 Cisco Global Networking Trends Survey [336], over 58% of organizations globally have already deployed SD-WAN in some form, and over 94% of respondents believe they will deploy some form of basic or more advanced SD-WAN implementation within the next two years.

Major networking equipment manufacturers, network consulting firms and network operators have supported deployment of hSDN. In 2012 Huawei launched the industry’s first hSDN controller and SDN-capable router [337]. Huawei was also the first to pass the OF 1.2 interoperability tests managed by the Open Networking Foundation. Entuity reports that many organizations do not adopt SDN due to costs including retraining staff, HW costs, SW licenses and hidden costs of business continuity during initial deployment and advocates incremental deployment of hSDN [338]. Allied Telesis proposes a migration path to SDN in an enterprise network [339]. It recommends migration in three phases. The first phase achieves unified management of the network. The second phase uses OpenFlow hybrid where some nodes support OpenFlow while others don’t. The third phase is optimum SDN deployment. It indicates that transition from a hybrid network to SDN may be gradual or never completed. NEC describes various hSDN introduction models that satisfy a diverse range of customer needs [340]. It indicates that by introducing SDN to only the advantageous area in an existing network, the effects and benefits of SDN can be leveraged without affecting the existing network.

In the SDN hybrid network architecture designed by HP [341], the hybrid SDN controller retains control over all packet forwarding on the data plane and chooses to delegate forwarding decisions to controlled switches in order to reduce complexity of forwarding decisions controller makes and to reduce the amount of traffic on the control plane between the switches and controller.
The Verizon SDN/NFV reference architecture [342] describes a range approaches to SDN deployment and supports hSDN as a practical solution. In this hSDN architecture, SDN controllers cooperate with existing forwarding boxes and vendor-specific domain controllers. AT&T advocates hSDN networks for cloud networks and big data applications. They indicate that the transition to SDN will be a multi-phased process and having a clear technology road-map will help enterprises transition in small increments with minimal disruption to their business goals.

As businesses continue to deploy new applications and move to virtualized environments, they will need to connect them to legacy applications and data stores. Businesses should base their decisions on network architecture that provide the features and capabilities they need now as well as in the future. They will demand a network architecture that is easy to integrate with SDN controllers. They will also prefer open standards and interoperability between their environments and network devices. Several ways of integrating SDN into the Data Center using Juniper SDN gateways are described in a Juniper white paper [343] on SDN integration.

IBM developed a reference architecture designed to help organizations compare and contrast various SDN products [344]. IBM has defined many use cases (such as micro-segmentation) and more than 100 requirements to help adapt SDN to different environments. Requirements include the need to align network security policies to the new SDN-driven capabilities, perhaps leveraging metadata from virtual machines in addition to the standard IP address segmentation rules.

Recent improvements in the Intel micro-architecture enable network service providers to gain more flexibility and control over customer offerings through the use of SDN [345]. By virtualizing network functions on Intel architecture, network service providers can employ techniques such as deep packet inspection, geographic LB, and power management to optimize bandwidth. Practical implementation issues of SDN and NFV in WAN are described in a white paper from Wind River [346]. An Intel/HP/Wind River reference design demonstrates that performance of a standard open source Open vSwitch can improve by an order of magnitude by fusing Intel’s Data Plane Development Kit with Wind River’s Open Virtualization Profile and running the software on industry-standard hardware supplied by HP.

A Gigamon white paper [347] indicates that, while some organizations may have the resources for a new network deployment of SDN, the majority will likely deploy hybrid networks based on traditional infrastructure side by side with SDN capable routers and switches and dynamically provisioned workloads. For most organizations, their networking infrastructure vendors will have offerings that will allow a gradual migration to SDN.

Cisco white paper [348] describes SDN and network programmability use cases for the defense and intelligence communities. Use cases include latency-based routing, dynamic QoS, diversion networking, central policy management for access control lists and call admission control in multiple security domain networks.

An SEL white paper [349] examines the benefits of using SDN to easily interconnect and manage traffic on Ethernet networks that communicate using IEC 61850 technology. A case study from the Itaipu Dam in South America, one of the world’s largest hydroelectric facilities, is used to illustrate these benefits.

A Lumina white paper [350] describes an extensible software platform that enables the automation of legacy network elements using model driven frameworks, shielding the complexity of underlying south bound interfaces and enabling northbound applications. Lumina’s platform uses a micro-services architecture to extend the capabilities of the OpenDayLight based SDN controller to enable better integration with business layers.

There is limited work on hSDN standardization. ONF document [351] describes various scenarios and issues to be addressed to ensure successful migration from hSDN to SDN. The discussion includes traditionally controlled and SDN controlled ports, traditional underlay with SDN overlay, explicit invocation of traditional forwarding from SDN and control plane connectivity for hSDN. RFC 7149 [352] indicates that SDN will be deployed incrementally and raises interesting questions such as how SDN affects the lifetime of legacy systems. Legacy systems may be obsolete rapidly due to their HW and SW limitations compared with SDN.

9 Open Research Challenges in hSDN

Based on our in-depth survey, we identified the following areas where there is limited work or satisfactory solutions are not available.

  • [wide, labelwidth=labelindent=0pt]

  • P4-capable devices: The P4 language enables programmers to define a variety of protocols and data plane behaviours to be implemented in P4-enabled nodes. Of course, in hSDN deployments, these custom features and protocols must be carefully designed and implemented to avoid disrupting the switching and routing operations of legacy L2 and L3 devices.

  • QoS in hSDN: This is closely related to node discovery and more challenging than node discovery. Any QoS approach needs to discover all the nodes and allocate resources on the nodes. Having limited control over the nodes render QoS provisioning challenging. Existing work covers the topic superficially, with some initial contributions related to discovery protocols [65], and to resource allocation in specific scenarios such as hybrid MPLS/OF networks [353].

  • Controller Placement Problem (CPP) PoCs: As stated in section 2.3.8 this problem can lead to critical issues in SDN and hSDN architectures and, therefore, it needs to be properly addressed and studied. To the best of our knowledge, there are only two works that analyze this problem proposing solutions for hSDN environments [87, 86]. We identify two main open challenges for the CPP in hybrid legacy/SDN networks:

    • Deployment model. The CPP problem should be tackled by taking into account the deployment model (cf. Section 2.1) and execution (either static or progressive deployment of SDN devices within a legacy network). In particular, a progressive deployment might require adaptive solutions of the problem, as done in dynamic contexts such as vehicular networks [354], or the so-called Software-Defined Drone networks [355] and Software-Defined Satellite Networking [356].

    • Legacy middleboxes. The impact of legacy devices on the control channel should be also carefully addresses. Thus, in the formulation of the CPP, the position and the type (e.g., router, firewall, IDS, etc.) of the legacy devices the control traffic must traverse should be also taken into account. Indeed, these factors might affect the end-to-end latency on the control channel, hence the optimal placement of the SDN controller.

  • Security: This topic is of vital importance in any kind of network but specially in hSDN and SDN networks as the attack surface might be increased. Despite all the surveyed works in Section 3, there still exists a lack of research on the following challenges:

    • Security by design. hSDN deployments integrate legacy and SDN technologies, each of them with its specific security-related issues and benefits. A service-based deployment model (cf. Section 2.1) focused on security services can be adopted for a secure network design. However, it might contrast the fulfillment of other requirements (e.g., QoS, budget) that might impose different deployment approaches.

    • Integrated security solutions. As discussed in Section 3.1, there are a number of vulnerabilities that affect hSDN networks. To reduce the attack surface of hSDN deployments, integrated solutions that combine the benefits of hardware appliances and SDN programmability are required. However, only a limited number of solutions in this regard have been proposed so far (cf. Section 3.3). Approaches based on the NFV paradigm look promising  [114, 115], however the proposed placement models are limited to NFV-enabled devices, without considering security hardware appliances when provisioning security services.

  • Emergent Technologies: A wide range of innovative technologies have been addressed in this survey. Due to their emergent nature there are few works developing hSDN solutions for each of these technologies. Thereupon, several key challenges for hSDN applied on these technologies remain unresolved:

    • hSDN 5G architectures. As demonstrated in Section 6 there are a couple of works approaching this challenge but there is still a lot to be done in order to achieve a full hSDN 5G architecture or even to know if this approach brings more advantages than potential disadvantages. An integral hSDN 5G Architecture PoC or testbed would be a promising solution as it would help to clarify the above-addresed issues. Therefore, an in-deep study of the advantages that hSDN brings to 5G Architectures would help to establish the base to develop these further 5G-hSDN architecture testbeds.

    • Cloud-native hSDN architectures with high energy efficiency. Cloud-native environments are becoming a requirement for many technologies and services i.e. 5G architecture is designed with a cloud-native approach. As shown in section 6.2 and in Table VII, there has been a lot of research on this topic for pure SDN but it is still an innovative research area that needs to analyze the energy efficiency management in DCNs with hSDN. hSDN cloud-native proposals with a focus in energy efficiency [256] demonstrate the potential within this combination of technologies, however these approaches are conditioned to a limited range of DCNs topology types.

    • hSDN and MEC-based architectures. MEC/edge is a disrupting technology with a high interest for Communication Providers and Operators as it allows to reduce application latencies and the overall connection speeds. This technology is highly dependant of virtualization technologies and, consequently, SDN and hSDN are an important approach. At the time of writing, there were no works combining this two technologies. Besides, some SDOs such as ETSI are looking for PoCs related to MEC/edge since 2016 [357].

    • hSDN budget solutions for Internet of Everything (IoX) environments. Day by day, the use cases of IoX scenarios is increasing but the number of works relating hSDN with these technologies is just starting to arise. We think that there is a big research/opportunity gap within this topic as the benefits obtained from combining these technologies, as shown in Section 6.3, are numerous.

    • SD-Branch hSDN opensource solutions. SD-Branch is a young concept that begins to have an increasing interest for the industry and that can be highly exploited by the research community as a promising technology for flexible networks. Therefore, an opensource solution that is combined with hSDN would be an innovative approach as there are no analogous platforms up to the date.

  • Simulation Tools and Testbeds: Although there is a clear emergence and development of pure SDN simulation tools and testbeds that support multiple legacy protocols, there is still a lack of open dedicated simulation tools that allow to experiment with hSDN. DOT and ONE [312, 313] may be considered the first step towards resolving this particular challenge.

  • Resilience and Traffic Engineering: There is limited work on resilience and fault-tolerance of legacy routers in hSDN mostly due to lack of adequate protocols for discovering legacy routers. Current traffic engineering work focuses on providing SDN benefits using a small set of SDN capable routers. However, these approaches fail to provide the foundations for services such as QoS. Proposed traffic engineering approaches need to be improved to provide better resilience, fault-tolerance and improved estimation error to enable QoS based services.

  • Standardization of hSDN: Standardization efforts for hSDN are limited considering the size of hSDN deployments. Standardization on various issues such as traffic measurement interfaces, QoS primitives supported, ways for handling legacy devices, algorithms for node discovery, primitives for provisioning security services will enable faster deployment of hSDN by promoting interoperable hardware and software by various vendors.

    As already discussed in Section 2.1, there is not yet an accepted standard for NBI between the controller and SDN applications and network services that utilize the controller [358]. This forces application developers to either choose a specific controller platform, which might not meet their needs completely, or to modify their applications to work on different controllers. In this regard, the progressive integration of SDN technologies in legacy networks, hence the increasing need for common interfaces, is pushing for an advancement in the direction of standard NBIs [359, 360]. Nevertheless, despite a few scattered initiatives, vendor specific and ad-hoc NBIs are still a major barrier to the development of portable SDN applications.

10 Conclusions

In contrast to the large and well-known network providers who have already adopted SDN, there are many organizations and enterprises who simply cannot afford a transition from legacy networking to the SDN. Keeping the traditional network infrastructure and updating it incrementally to an SDN infrastructure seems an affordable and technically feasible solution. The so-called Hybrid SDN (hSDN) is now considered as a widely accepted transition phase and a compromise solution which have the advantages of both SDN and legacy networking solutions.

This paper focuses on presenting a comprehensive state-of-the-art survey of hybrid SDN. It first studies hSDN models in both control and data planes along with control plane optimization in terms of placement and scalability problems. The related security and privacy issues and existing vulnerabilities and threats are another investigated domain. The investigation is then supported by exploring possible threats detection and mitigation approaches and reviewing recent security modules and cyber defence frameworks.
This paper also discusses hSDN network management. More specifically, it probes network update in both legacy and SDN settings. Network automation with respect to network telemetry for management plane and network auto configuration are other topics which are reviewed elaborately. Besides, reliability, resiliency, fault-tolerance, and load balancing are also surveyed. The paper then goes into effect traffic engineering topic and inquire into traffic measurement, traffic management and quality of service routing. The implementation and deployment of hSDN is another topic which is studied in this paper.
We observed there are few topics which did not receive enough attention in the related body of literature on hSDN. Correspondingly, recent hSDN use cases in 5G mobile networks, cloud and data center networking, IoT connectivity, Blockchain, SD-WAN, and finally SD-Branch are presented in detail. Existing simulation tools and public testbeds are another overlooked topics which reviewed and compared in this paper. Current status of standards and business issues in hSDN along with future research directions are the last topics inspected in this paper.

In conclusion, we believe that hSDN has come a long way. However, there are few domains which need further investigation. While different hSDN models and architectures, security, and traffic engineering solutions have been studied in depth, the existing literature on managing hybrid networks, privacy issues, and new hSDN standards and business models are sparse and need further investigations.

References

  • [1] J. S. Turner and D. E. Taylor, “Diversifying the internet,” in GLOBECOM ’05. IEEE Global Telecommunications Conference, 2005., vol. 2, 2005, pp. 6 pp.–760.
  • [2] J. Van der Merwe and C. Kalmanek, “Network programmability is the answer,” in Workshop on Programmable Routers for the Extensible Services of Tomorrow (PRESTO 2007), Princeton, NJ, 2007.
  • [3] N. McKeown, “Software-defined networking,” INFOCOM keynote talk, vol. 17, no. 2, pp. 30–32, 2009.
  • [4] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker, and J. Turner, “Openflow: enabling innovation in campus networks,” ACM SIGCOMM Computer Communication Review, vol. 38, no. 2, pp. 69–74, 2008.
  • [5] L. Yang, R. Dantu, T. Anderson, and R. Gopal, “Forwarding and control element separation (forces) framework,” Internet Requests for Comments, RFC Editor, RFC 3746, April 2004.
  • [6] P. Bosshart, D. Daly, G. Gibb, M. Izzard, N. McKeown, J. Rexford, C. Schlesinger, D. Talayco, A. Vahdat, G. Varghese et al., “P4: Programming protocol-independent packet processors,” ACM SIGCOMM Computer Communication Review, vol. 44, no. 3, pp. 87–95, 2014.
  • [7] “Open networking foundation,” https://www.opennetworking.org/, 2011, Last accessed: 2020-05-28.
  • [8] T. Wang, F. Liu, and H. Xu, “An efficient online algorithm for dynamic sdn controller assignment in data center networks,” IEEE/ACM Transactions on Networking, vol. 25, no. 5, pp. 2788–2801, 2017.
  • [9] L. Velasco, A. Asensio, J. Berral, A. Castro, and V. López, “Towards a carrier sdn: An example for elastic inter-datacenter connectivity,” Optics express, vol. 22, no. 1, pp. 55–61, 2014.
  • [10] J. Liu, J. Li, G. Shou, Y. Hu, Z. Guo, and W. Dai, “Sdn based load balancing mechanism for elephant flow in data center networks,” in 2014 International Symposium on Wireless Personal Multimedia Communications (WPMC).   IEEE, 2014, pp. 486–490.
  • [11] M. Banikazemi, D. Olshefski, A. Shaikh, J. Tracey, and G. Wang, “Meridian: an sdn platform for cloud network services,” IEEE Communications Magazine, vol. 51, no. 2, pp. 120–127, 2013.
  • [12] J. Mambretti, J. Chen, and F. Yeh, “Next generation clouds, the chameleon cloud testbed, and software defined networking (sdn),” in 2015 International Conference on Cloud Computing Research and Innovation (ICCCRI).   IEEE, 2015, pp. 73–79.
  • [13] S. Jain, A. Kumar, S. Mandal, J. Ong, L. Poutievski, A. Singh, S. Venkata, J. Wanderer, J. Zhou, M. Zhu, J. Zolla, U. Hölzle, S. Stuart, and A. Vahdat, “B4: Experience with a globally-deployed software defined wan,” in Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, ser. SIGCOMM ’13, 2013, p. 3–14.
  • [14] X. Jin, Y. Li, D. Wei, S. Li, J. Gao, L. Xu, G. Li, W. Xu, and J. Rexford, “Optimizing bulk transfers with software-defined optical wan,” in Proceedings of the 2016 ACM SIGCOMM Conference, 2016, pp. 87–100.
  • [15] C.-Y. Hong, S. Mandal, M. Al-Fares, M. Zhu, R. Alimi, K. N. B., C. Bhagat, S. Jain, J. Kaimal, S. Liang, K. Mendelev, S. Padgett, F. Rabe, S. Ray, M. Tewari, M. Tierney, M. Zahn, J. Zolla, J. Ong, and A. Vahdat, “B4 and after: Managing hierarchy, partitioning, and asymmetry for availability and scale in google’s software-defined wan,” in Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, ser. SIGCOMM ’18, 2018, p. 74–87.
  • [16] S. Costanzo, L. Galluccio, G. Morabito, and S. Palazzo, “Software defined wireless networks: Unbridling sdns,” in 2012 European Workshop on Software Defined Networking.   IEEE, 2012, pp. 1–6.
  • [17] R. Riggio, M. K. Marina, J. Schulz-Zander, S. Kuklinski, and T. Rasheed, “Programming abstractions for software-defined wireless networks,” IEEE Transactions on Network and Service Management, vol. 12, no. 2, pp. 146–162, 2015.
  • [18] I. F. Akyildiz, S.-C. Lin, and P. Wang, “Wireless software-defined networks (w-sdns) and network function virtualization (nfv) for 5g cellular systems: An overview and qualitative evaluation,” Computer Networks, vol. 93, pp. 66–79, 2015.
  • [19] R. Trivisonno, R. Guerzoni, I. Vaishnavi, and D. Soldani, “Sdn-based 5g mobile networks: architecture, functions, procedures and backward compatibility,” Transactions on Emerging Telecommunications Technologies, vol. 26, no. 1, pp. 82–92, 2015.
  • [20] F. Z. Yousaf, M. Bredel, S. Schaller, and F. Schneider, “Nfv and sdn—key technology enablers for 5g networks,” IEEE Journal on Selected Areas in Communications, vol. 35, no. 11, pp. 2468–2478, Nov 2017.
  • [21] MARKETSANDMARKETS, “Software defined networking market by sdn type (open sdn, sdn via overlay, and sdn via api), component (solutions and services), end user (data centers, service providers, and enterprises), and region - global forecast to 2023,” Published on March 2019, Last accessed: 2020-02-17. [Online]. Available: https://www.marketsandmarkets.com/Market-Reports/software-defined-networking-sdn-market-655.html
  • [22] S. Schmid and J. Suomela, “Exploiting locality in distributed sdn control,” in Proceedings of the second ACM SIGCOMM workshop on Hot topics in software defined networking, 2013, pp. 121–126.
  • [23] Y. Jimenez, C. Cervello-Pastor, and A. J. Garcia, “On the controller placement for designing a distributed sdn control layer,” in 2014 IFIP Networking Conference.   IEEE, 2014, pp. 1–9.
  • [24] M. Aslan and A. Matrawy, “Adaptive consistency for distributed sdn controllers,” in 2016 17th International Telecommunications Network Strategy and Planning Symposium (Networks).   IEEE, 2016, pp. 150–157.
  • [25] S. Khorsandroo and A. S. Tosun, “White box analysis at the service of low rate saturation attacks on virtual sdn data plane,” in 2019 IEEE 44th LCN Symposium on Emerging Topics in Networking (LCN Symposium).   IEEE, 2019, pp. 100–107.
  • [26] ——, “Time inference attacks on software defined networks: Challenges and countermeasures,” in 2018 IEEE 11th International Conference on Cloud Computing (CLOUD).   IEEE, 2018, pp. 342–349.
  • [27] F. Telecom, “Google leans heavily on sdn for reliability, velocity and availability of its network,” Published by Mike Robuckon on Dec 6, 2018, Last accessed: 2020-03-22. [Online]. Available: https://www.fiercetelecom.com/telecom/google-leans-heavily-sdn-for-reliability-velocity-and-availability-its-network
  • [28] T. next platform, “How google wants to rewire the internet,” Published by Timothy Prickett Morgan on July 17, 2017, Last accessed: 2020-04-05. [Online]. Available: https://www.nextplatform.com/2017/07/17/google-wants-rewire-internet/
  • [29] S. Sezer, S. Scott-Hayward, P. K. Chouhan, B. Fraser, D. Lake, J. Finnegan, N. Viljoen, M. Miller, and N. Rao, “Are we ready for sdn? implementation challenges for software-defined networks,” IEEE Communications Magazine, vol. 51, no. 7, pp. 36–43, 2013.
  • [30] Y. Sinha, K. Haribabu et al., “A survey: hybrid sdn,” Journal of Network and Computer Applications, vol. 100, pp. 35–55, 2017.
  • [31] R. Amin, M. Reisslein, and N. Shah, “Hybrid sdn networks: A survey of existing approaches,” IEEE Communications Surveys Tutorials, vol. 20, no. 4, pp. 3259–3306, 2018.
  • [32] W. Wang, W. He, and J. Su, “Boosting the benefits of hybrid sdn,” in 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).   IEEE, 2017, pp. 2165–2170.
  • [33] Sandhya, Y. Sinha, and K. Haribabu, “A survey: Hybrid sdn,” Journal of Network and Computer Applications, vol. 100, pp. 35 – 55, 2017.
  • [34] R. Amin, N. Shah, B. Shah, and O. Alfandi, “Auto-configuration of acl policy in case of topology change in hybrid sdn,” IEEE Access, vol. 4, pp. 9437–9450, 2016.
  • [35] X. Huang, S. Cheng, K. Cao, P. Cong, T. Wei, and S. Hu, “A survey of deployment solutions and optimization strategies for hybrid sdn networks,” IEEE Communications Surveys Tutorials, vol. 21, no. 2, pp. 1483–1507, 2019.
  • [36] S. Vissicchio, L. Vanbever, and O. Bonaventure, “Opportunities and research challenges of hybrid software defined networks,” ACM SIGCOMM Computer Communication Review, vol. 44, pp. 70–75, 04 2014.
  • [37] O. N. Foundation, “Openflow switch specification version 1.5.1,” https://www.opennetworking.org/wp-content/uploads/2014/10/openflow-switch-v1.5.1.pdf, March 2015.
  • [38] N. Katta, O. Alipourfard, J. Rexford, and D. Walker, “Infinite CacheFlow in Software-Defined Networks,” in Proceedings of the Third Workshop on Hot Topics in Software Defined Networking, 2014.
  • [39] R. Doriguzzi-Corin, D. Siracusa, E. Salvadori, and A. Schwabe, “Empowering network operating systems with memory management techniques,” in NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium, 2016.
  • [40] A. Marsico, R. Doriguzzi-Corin, and D. Siracusa, “An effective swapping mechanism to overcome the memory limitation of sdn devices,” in 2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), 2017.
  • [41] B. Pfaff, J. Pettit, T. Koponen, E. Jackson, A. Zhou, J. Rajahalme, J. Gross, A. Wang, J. Stringer, P. Shelar, K. Amidon, and M. Casado, “The design and implementation of open vswitch,” in 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15).   Oakland, CA: USENIX Association, May 2015, pp. 117–130.
  • [42] D. Parniewicz, R. Doriguzzi Corin, L. Ogrodowczyk, M. Rashidi Fard, J. Matias, M. Gerola, V. Fuentes, U. Toseef, A. Zaalouk, B. Belter, E. Jacob, and K. Pentikousis, “Design and implementation of an openflow hardware abstraction layer,” in Proceedings of the 2014 ACM SIGCOMM Workshop on Distributed Cloud Computing, 2014.
  • [43] R. Kandoi, “Deploying software-defined networks: a telco perspective,” 2015.
  • [44] B. S. Networks, “Big virtual switch,” https://www.bigswitch.com/sites/default/files/sdnresources/ bvsdatasheet.pdf, 2013.
  • [45] Open Networking Foundation, “SDN architecture. Issue 1,” https://www.opennetworking.org/images/stories/downloads/sdn-resources/technical-reports/TR_SDN_ARCH_1.0_06062014.pdf, June 2014, Last accessed: 2020-04-30.
  • [46] E. Haleplidis, K. Pentikousis, S. Denazis, J. H. Salim, D. Meyer, and O. Koufopavlou, “Software-defined networking (sdn): Layers and architecture terminology,” Internet Requests for Comments, RFC Editor, RFC 7426, January 2015.
  • [47] R. Enns, M. Bjorklund, J. Schoenwaelder, and A. Bierman, “Network configuration protocol (netconf),” Internet Requests for Comments, IETF, RFC 6241, June 2011. [Online]. Available: http://www.rfc-editor.org/rfc/rfc6241.txt
  • [48] O. N. Foundation, “Openflow switch specification version 1.3.0,” https://www.opennetworking.org/wp-content/uploads/2014/10/openflow-spec-v1.3.0.pdf, June 2012.
  • [49] S. Hares and R. White, “Software-defined networks and the interface to the routing system (i2rs),” IEEE Internet Computing, vol. 17, no. 4, pp. 84–88, 2013.
  • [50] M. Szalay, L. Toka, G. Rétvári, G. Pongrácz, L. Csikor, and D. P. Pezaros, “Harmless: Cost-effective transitioning to sdn,” in Proceedings of the SIGCOMM Posters and Demos.   2018 IFIP Networking Conference and Workshops, 2017, pp. 91–93.
  • [51] L. Csikor, M. Szalay, G. Rétvári, G. Pongrácz, D. P. Pezaros, and L. Toka, “Transition to sdn is harmless: Hybrid architecture for migrating legacy ethernet switches to sdn,” IEEE/ACM Transactions on Networking, vol. 28, no. 1, pp. 275–288, 2020.
  • [52] T. P. L. Consortium, “P4 language specification,” https://p4.org/p4-spec/docs/P4-16-working-spec.html, 2020.
  • [53] W. Feng, Z.-L. Zhang, C. Liu, and J. Chen, “Clé: Enhancing security with programmable dataplane enabled hybrid sdn,” in Proceedings of the 15th International Conference on Emerging Networking EXperiments and Technologies, 2019.
  • [54] P. Bosshart, D. Daly, G. Gibb, M. Izzard, N. McKeown, J. Rexford, C. Schlesinger, D. Talayco, A. Vahdat, G. Varghese, and D. Walker, “P4: Programming protocol-independent packet processors,” SIGCOMM Comput. Commun. Rev., vol. 44, no. 3, p. 87–95, 2014.
  • [55] I. Martinez-Yelmo, J. Alvarez-Horcajo, M. Briso-Montiano, D. Lopez-Pajares, and E. Rojas, “Arp-p4: deep analysis of a hybrid sdn arp-path/p4runtime switch,” Telecommunication Systems, vol. 72, no. 4, pp. 555–565, 2019.
  • [56] Intel, “Netcope p4,” https://www.intel.com/content/www/us/en/programmable/solutions/partners/partner-profile/netcope-technologies–a-s-/ip/netcope-p4.html, 2020.
  • [57] H. Wang, R. Soulé, H. T. Dang, K. S. Lee, V. Shrivastav, N. Foster, and H. Weatherspoon, “P4fpga: A rapid prototyping framework for p4,” ser. SOSR ’17, 2017.
  • [58] S. Ibanez, G. Brebner, N. McKeown, and N. Zilberman, “The p4-¿netfpga workflow for line-rate packet processing,” in Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, ser. FPGA ’19, 2019.
  • [59] Z. Cao, H. Su, Q. Yang, J. Shen, M. Wen, and C. Zhang, “P4 to fpga-a fast approach for generating efficient network processors,” IEEE Access, vol. 8, pp. 23 440–23 456, 2020.
  • [60] Z. N. Abdullah, I. Ahmad, and I. Hussain, “Segment routing in software defined networks: A survey,” IEEE Communications Surveys & Tutorials, vol. 21, no. 1, pp. 464–486, 2018.
  • [61] I. Šeremet and S. Čaušević, “Advancing multiprotocol label switching traffic engineering with segment routing in software defined network environment,” in 2020 19th International Symposium INFOTEH-JAHORINA (INFOTEH), 2020, pp. 1–6.
  • [62] W. Zhang, W. Lei, and S. Zhang, “A multipath transport scheme for real-time multimedia services based on software-defined networking and segment routing,” IEEE Access, vol. 8, pp. 93 962–93 977, 2020.
  • [63] K. Poularakis, Q. Qin, K. M. Marcus, K. S. Chan, K. K. Leung, and L. Tassiulas, “Hybrid sdn control in mobile ad hoc networks,” in 2019 IEEE International Conference on Smart Computing (SMARTCOMP), 2019, pp. 110–114.
  • [64] E. Kaljic, A. Maric, and P. Njemcevic, “An implementation of a deeply programmable sdn switch based on a hybrid fpga/cpu architecture,” in 2019 18th International Symposium INFOTEH-JAHORINA (INFOTEH).   IEEE, 2019, pp. 1–6.
  • [65] J. Alvarez-Horcajo, E. Rojas, I. Martinez-Yelmo, M. Savi, and D. Lopez-Pajares, “Hddp: Hybrid domain discovery protocol for heterogeneous devices in sdn,” IEEE Communications Letters, 2020.
  • [66] P. Shi, S. Rivera, L. Pike, Z. Fei, J. Griffioen, and K. Calvert, “Enabling Shared Control and Trust in Hybrid SDN/Legacy Networks,” in 2019 28th International Conference on Computer Communication and Networks (ICCCN), 2019, pp. 1–9.
  • [67] P. Lin, J. Bi, and H. Hu, “Btsdn: Bgp-based transition for the existing networks to sdn,” Wireless Personal Communications, vol. 86, no. 4, pp. 1829–1843, 2016.
  • [68] T. Feng and J. Bi, “Openrouteflow: Enable legacy router as a software-defined routing service for hybrid sdn,” in 2015 24th International Conference on Computer Communication and Networks (ICCCN), 2015, pp. 1–8.
  • [69] O. Tilmans and S. Vissicchio, “Igp-as-a-backup for robust sdn networks,” in 10th International Conference on Network and Service Management (CNSM) and Workshop, 2014, pp. 127–135.
  • [70] J. Alvarez-Horcajo, I. Martinez-Yelmo, E. Rojas, J. A. Carral, and D. Lopez-Pajares, “New cooperative mechanisms for software defined networks based on hybrid switches,” Transactions on Emerging Telecommunications Technologies, vol. 28, no. 8, p. e3150, 2017.
  • [71] S. Salsano, P. L. Ventre, L. Prete, G. Siracusano, M. Gerola, and E. Salvadori, “Oshi-open source hybrid ip/sdn networking (and its emulation on mininet and on distributed sdn testbeds),” in 2014 Third European Workshop on Software Defined Networks.   IEEE, Sep. 2014, pp. 13–18.
  • [72] Quagga, “Quagga routing suite,” https://www.nongnu.org/quagga/, 2012, Last accessed: 2020-04-17.
  • [73] F. Kuliesius and M. Giedraitis, “Sdn/legacy hybrid network control system,” in 2019 Eleventh International Conference on Ubiquitous and Future Networks (ICUFN).   IEEE, 2019, pp. 504–509.
  • [74] M. Caesar, D. Caldwell, N. Feamster, J. Rexford, A. Shaikh, and J. van der Merwe, “Design and implementation of a routing control platform,” in Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation-Volume 2.   USENIX Association, 2005, pp. 15–28.
  • [75] S. Vissicchio, L. Vanbever, and J. Rexford, “Sweet little lies: Fake topologies for flexible routing,” in proceedings of the 13th ACM Workshop on Hot Topics in Networks, 2014, pp. 1–7.
  • [76] M. Birk, G. Choudhury, B. Cortez, A. Goddard, N. Padi, A. Raghuram, K. Tse, S. Tse, A. Wallace, and K. Xi, “Evolving to an sdn-enabled isp backbone: key technologies and applications,” IEEE Communications Magazine, vol. 54, no. 10, pp. 129–135, 2016.
  • [77] D. Levin, M. Canini, S. Schmid, F. Schaffert, and A. Feldmann, “Panopticon: Reaping the benefits of incremental SDN deployment in enterprise networks,” in 2014 USENIX Annual Technical Conference (USENIX ATC 14).   Philadelphia, PA: USENIX Association, Jun. 2014, pp. 333–345. [Online]. Available: https://www.usenix.org/conference/atc14/technical-sessions/presentation/levin
  • [78] M. Caria, A. Jukan, and M. Hoffmann, “Sdn partitioning: A centralized control plane for distributed routing protocols,” IEEE Transactions on Network and Service Management, vol. 13, no. 3, pp. 381–393, 2016.
  • [79] K. Poularakis, Q. Qin, E. Nahum, M. Rio, and L. Tassiulas, “Bringing sdn to the mobile edge,” in 2017 IEEE SmartWorld, Ubiquitous Intelligence Computing, Advanced Trusted Computed, Scalable Computing Communications, Cloud Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), 2017.
  • [80] J. Lu, Z. Zhang, T. Hu, P. Yi, and J. Lan, “A survey of controller placement problem in software-defined networking,” IEEE Access, vol. 7, pp. 24 290–24 307, 2019.
  • [81] S. Saraswat, V. Agarwal, H. P. Gupta, R. Mishra, A. Gupta, and T. Dutta, “Challenges and solutions in software defined networking: A survey,” Journal of Network and Computer Applications, vol. 141, pp. 23–58, 2019.
  • [82] B. P. R. Killi and S. V. Rao, “Controller placement in software defined networks: A comprehensive survey,” Computer Networks, vol. 163, p. 106883, 2019.
  • [83] A. Kumari and A. S. Sairam, “A survey of controller placement problem in software defined networks,” arXiv preprint arXiv:1905.04649, 2019. [Online]. Available: https://arxiv.org/abs/1905.04649v1
  • [84] S. Mohanty, K. Kanodia, B. Sahoo, and K. Kurroliya, “A simulated annealing strategy for reliable controller placement in software defined networks,” in 2020 7th International Conference on Signal Processing and Integrated Networks (SPIN), 2020, pp. 844–849. [Online]. Available: http://hdl.handle.net/2080/3519
  • [85] G. Schütz and J. Martins, “A comprehensive approach for optimizing controller placement in software-defined networks,” Computer Communications, 2020.
  • [86] Z. Guo, W. Chen, Y.-F. Liu, Y. Xu, and Z.-L. Zhang, “Joint switch upgrade and controller deployment in hybrid software-defined networks,” IEEE Journal on Selected Areas in Communications, vol. 37, no. 5, pp. 1012–1028, 2019.
  • [87] J. Yuan, E. Li, C. Kang, F. Chang, T. Yuan, and X. Li, “Latency-based dynamic controller assignment in hybrid sdns: Considering the impact of legacy routers,” Future Internet, vol. 11, no. 8, p. 168, 2019.
  • [88] A. Abuarqoub, “A review of the control plane scalability approaches in software defined networking,” Future Internet, vol. 12, no. 3, p. 49, 2020.
  • [89] M. Antikainen, T. Aura, and M. Särelä, “Spook in your network: Attacking an sdn with a compromised openflow switch,” in Secure IT Systems, 2014, pp. 229–244.
  • [90] Po-Wen Chi, Chien-Ting Kuo, Jing-Wei Guo, and Chin-Laung Lei, “How to detect a compromised sdn switch,” in Proceedings of the 2015 1st IEEE Conference on Network Softwarization (NetSoft), 2015, pp. 1–6.
  • [91] A. Akhunzada, “Securing software defined networks: Taxonomy, requirements, and open issues,” IEEE Communications Magazine, pp. 36–44, 04 2015.
  • [92] S. Scott-Hayward, C. Kane, and S. Sezer, “Operationcheckpoint: Sdn application control,” in 2014 IEEE 22nd International Conference on Network Protocols, 2014.
  • [93] A. L. Aliyu, P. Bull, and A. Abdallah, “A trust management framework for network applications within an sdn environment,” in 2017 31st International Conference on Advanced Information Networking and Applications Workshops (WAINA), 2017, pp. 93–98.
  • [94] R. Doriguzzi-Corin, E. Salvadori, P. A. Aranda Gutiérrez, C. Stritzke, A. Leckey, K. Phemius, E. Rojas, and C. Guerrero, “NetIDE: Removing Vendor Lock-in in SDN,” in Network Softwarization (NetSoft), 2015 1st IEEE Conference on, April 2015, pp. 1–2.
  • [95] P. Berde, M. Gerola, J. Hart, Y. Higuchi, M. Kobayashi, T. Koide, B. Lantz, B. O’Connor, P. Radoslavov, W. Snow, and G. Parulkar, “ONOS: Towards an Open, Distributed SDN OS,” in Proceedings of the Third Workshop on Hot Topics in Software Defined Networking, ser. HotSDN ’14.   New York, NY, USA: Association for Computing Machinery, 2014, p. 1–6.
  • [96] J. Medved, R. Varga, A. Tkacik, and K. Gray, “Opendaylight: Towards a model-driven sdn controller architecture,” in Proceeding of IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks 2014, June 2014, pp. 1–6.
  • [97] R. Doriguzzi-Corin, P. A. Aranda Gutiérrez, E. Rojas, H. Karl, and E. Salvadori, “Reusability of Software-Defined Networking Applications: A Runtime, Multi-Controller Approach,” in Proceedings of the 12th Conference on International Conference on Network and Service Management, 2016.
  • [98] K. Benton, L. J. Camp, and C. Small, “Openflow vulnerability assessment,” ser. HotSDN ’13, 2013.
  • [99] R. Sherwood, G. Gibb, K.-K. Yap, G. Appenzeller, M. Casado, N. McKeown, and G. Parulkar, “Can the Production Network Be the Testbed?” in Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation, 2010.
  • [100] A. Al-Shabibi, M. De Leenheer, M. Gerola, A. Koshibe, G. Parulkar, E. Salvadori, and B. Snow, “Openvirtex: Make your virtual sdns programmable,” in Proceedings of the Third Workshop on Hot Topics in Software Defined Networking, 2014.
  • [101] R. Doriguzzi-Corin, E. Salvadori, M. Gerola, M. Suñé, and H. Woesner, “A Datapath-Centric Virtualization Mechanism for OpenFlow Networks,” in 2014 Third European Workshop on Software Defined Networks, 2014.
  • [102] X. Jin, J. Gossels, J. Rexford, and D. Walker, “CoVisor: A Compositional Hypervisor for Software-defined Networks,” in Proceedings of the 12th USENIX Conference on Networked Systems Design and Implementation, ser. NSDI’15, 2015.
  • [103] D. Kreutz, F. M. Ramos, and P. Verissimo, “Towards Secure and Dependable Software-Defined Networks,” in Proceedings of the Second ACM SIGCOMM Workshop on Hot Topics in Software Defined Networking, 2013.
  • [104] A. Dixit, K. Kogan, and P. Eugster, “Composing Heterogeneous SDN Controllers with Flowbricks,” in 2014 IEEE 22nd International Conference on Network Protocols, 2014, pp. 287–292.
  • [105] J. C. Mogul, A. AuYoung, S. Banerjee, L. Popa, J. Lee, J. Mudigonda, P. Sharma, and Y. Turner, “Corybantic: Towards the Modular Composition of SDN Control Programs,” in Proceedings of the Twelfth ACM Workshop on Hot Topics in Networks, 2013.
  • [106] R. Sahay, G. Blanc, Z. Zhang, and H. Debar, “Aroma: An sdn based autonomic ddos mitigation framework,” computers & security, vol. 70, pp. 482–499, 2017.
  • [107]

    P. Harikrishna and A. Amuthan, “Sdn-based ddos attack mitigation scheme using convolution recursively enhanced self organizing maps,” in

    SADHANA-ACADEMY PROCEEDINGS IN ENGINEERING SCIENCES, vol. 45, no. 1.   Springer, 2020.
  • [108] S. Hameed and H. A. Khan, “Leveraging sdn for collaborative ddos mitigation,” in 2017 International Conference on Networked Systems (NetSys).   IEEE, 2017, pp. 1–6.
  • [109] T. A. Pascoal, Y. G. Dantas, I. E. Fonseca, and V. Nigam, “Slow tcam exhaustion ddos attack,” in ICT Systems Security and Privacy Protection, 2017, pp. 17–31.
  • [110] Juniper, “Security Bulletin: Junos OS: EX4300 Series: Denial of Service upon receipt of large number of specific valid packets on management interface,” 2019. [Online]. Available: https://kb.juniper.net/InfoCenter/index?page=content&id=JSA10938&actp=METADATA
  • [111] R. Millman, “Vulnerability in Cisco routers could allow DoS attacks,” 2018. [Online]. Available: https://www.scmagazineuk.com/vulnerability-cisco-routers-allow-dos-attacks/article/1494131
  • [112] H. Huang, S. Guo, J. Wu, and J. Li, “Service chaining for hybrid network function,” IEEE Transactions on Cloud Computing, vol. 7, no. 4, pp. 1082–1094, 2019.
  • [113] Z. Ning, N. Wang, and R. Tafazolli, “Deep reinforcement learning for nfv-based service function chaining in multi-service networks : Invited paper,” in 2020 IEEE 21st International Conference on High Performance Switching and Routing (HPSR), 2020, pp. 1–6.
  • [114] A. Shameli-Sendi, Y. Jarraya, M. Pourzandi, and M. Cheriet, “Efficient provisioning of security service function chaining using network security defense patterns,” IEEE Transactions on Services Computing, vol. 12, no. 4, pp. 534–549, 2019.
  • [115] R. Doriguzzi-Corin, S. Scott-Hayward, D. Siracusa, M. Savi, and E. Salvadori, “Dynamic and application-aware provisioning of chained virtual security network functions,” IEEE Transactions on Network and Service Management, vol. 17, no. 1, pp. 294–307, 2020.
  • [116] C. Kim, P. Bhide, E. Doe, H. Holbrook, A. Ghanwani, D. Daly, M. Hira, and B. Davie, “In‐band Network Telemetry,” 2016. [Online]. Available: https://p4.org/assets/INT-current-spec.pdf
  • [117] D. Ding, M. Savi, and D. Siracusa, “Estimating Logarithmic and Exponential Functions to Track Network Traffic Entropy in P4,” in Proceedings of the IFIP Network Operations and Management Symposium (NOMS), 2020.
  • [118] S. Scott-Hayward, S. Natarajan, and S. Sezer, “A survey of security in software defined networks,” IEEE Communications Surveys Tutorials, vol. 18, no. 1, pp. 623–654, 2016.
  • [119] S. T. Ali, V. Sivaraman, A. Radford, and S. Jha, “A survey of securing networks using software defined networking,” IEEE Transactions on Reliability, vol. 64, no. 3, pp. 1086–1097, 2015.
  • [120] H. Jmila and G. Blanc, “Designing security-aware service requests for nfv-enabled networks,” in 2019 28th International Conference on Computer Communication and Networks (ICCCN), 2019, pp. 1–9.
  • [121] P. Quinn and T. Nadeau, “Problem statement for service function chaining,” Internet Requests for Comments, RFC Editor, RFC 7498, April 2015.
  • [122] DPDK, “Data plane development kit,” https://www.dpdk.org/, jun 2018, Last accessed: 2020-05-28.
  • [123] L. Rizzo, “Netmap: a novel framework for fast packet i/o,” in 21st USENIX Security Symposium (USENIX Security 12), 2012, pp. 101–112.
  • [124] D. Barach, L. Linguaglossa, D. Marion, P. Pfister, S. Pontarelli, and D. Rossi, “High-speed software data plane via vectorized packet processing,” IEEE Communications Magazine, vol. 56, no. 12, pp. 97–103, 2018.
  • [125] S. Miano, R. Doriguzzi-Corin, F. Risso, D. Siracusa, and R. Sommese, “Introducing smartnics in server-based data plane processing: The ddos mitigation use case,” IEEE Access, vol. 7, pp. 107 161–107 170, 2019.
  • [126] H. Pan, H. Guan, J. Liu, W. Ding, C. Lin, and G. Xie, “The FlowAdapter: Enable Flexible Multi-Table Processing on Legacy Hardware,” in Proceedings of the Second ACM SIGCOMM Workshop on Hot Topics in Software Defined Networking, 2013.
  • [127] R. S. D. Gamayunov, I. Platonov, “Toward network access control with software-defined networking,” 2013. [Online]. Available: http://www.researchgate.net/profile/Ruslan_Smeliansky/publication/236148336_Toward_Network_Access_Control_With_Software-Defined_Networking/links/00b7d51caj777e947a000000.pdf
  • [128] A. Fiessler, C. Lorenz, S. Hager, and B. Scheuermann, “FireFlow - High Performance Hybrid SDN-Firewalls with OpenFlow,” in 2018 IEEE 43rd Conference on Local Computer Networks (LCN), 2018.
  • [129] D. Ding, M. Savi, G. Antichi, and D. Siracusa, “An incrementally-deployable p4-enabled architecture for network-wide heavy-hitter detection,” IEEE Transactions on Network and Service Management, vol. 17, no. 1, pp. 75–88, 2020.
  • [130] K.T. Cheng, P.W. Tsai, A.C. Risdianto, T.C. Ling, S.W. Lee, C.S. Yang, “Implementations of BGP Security Modules in Hybrid-SDN network,” in Proceedings of the 16th APAN Research Workshop, 2019.
  • [131] M. M. I. Fahad Ubaid, Rashid Amin, “Mitigating Address Spoofing Attacks in Hybrid SDN,” International Journal of Advanced Computer Science and Applications, vol. 8, no. 4, 2017.
  • [132] A. Sebbar, K. ZKIK, Y. Baadi, M. Boulmalf, and M. D. ECH-CHERIF El KETTANI, “Using advanced detection and prevention technique to mitigate threats in sdn architecture,” in 2019 15th International Wireless Communications Mobile Computing Conference (IWCMC), 2019, pp. 90–95.
  • [133] K. Zkik, A. Sebbar, Y. Baadi, A. Belhadi, and M. Boulmalf, “An efficient modular security plane am-secp for hybrid distributed sdn,” in 2019 International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), 2019, pp. 354–359.
  • [134] Zkik, Karim, EL Hajji, Said, and Orhanou, Ghizlane, “A centralized secure plan for detecting and mitigation incidents in hybrid sdn,” MATEC Web Conf., vol. 189, p. 10015, 2018.
  • [135] R. Amin, N. Shah, and W. Mehmood, “Enforcing optimal acl policies using k-partite graph in hybrid sdn,” Electronics, vol. 8, no. 6, p. 604, May 2019. [Online]. Available: http://dx.doi.org/10.3390/electronics8060604
  • [136] C. Lorenz, D. Hock, J. Scherer, R. Durner, W. Kellerer, S. Gebert, N. Gray, T. Zinner, and P. Tran-Gia, “An sdn/nfv-enabled enterprise network architecture offering fine-grained security policy enforcement,” IEEE Communications Magazine, vol. 55, no. 3, pp. 217–223, 2017.
  • [137] I. E. T. F. (IETF), “BGPsec Protocol Specification,” 2017. [Online]. Available: https://tools.ietf.org/html/rfc8205
  • [138] N. Arora, H. Zhang, C. Lumezanu, J. Rhee, G. Jiang, and H. Lu, “Hybrid network management,” Sep. 20 2016, uS Patent 9,450,823.
  • [139] H. Lu, N. Arora, H. Zhang, C. Lumezanu, J. Rhee, and G. Jiang, “Hybnet: Network manager for a hybrid network infrastructure,” in Proceedings of the Industrial Track of the 13th ACM/IFIP/USENIX International Middleware Conference, 2013.
  • [140] M. Reitblatt, N. Foster, J. Rexford, C. Schlesinger, and D. Walker, “Abstractions for network update,” ACM SIGCOMM Computer Communication Review, vol. 42, no. 4, pp. 323–334, 2012.
  • [141] S. Ghorbani and M. Caesar, “Walk the line: consistent network updates with bandwidth guarantees,” in Proceedings of the first workshop on Hot topics in software defined networks, 2012, pp. 67–72.
  • [142] S. Vissicchio, L. Vanbever, L. Cittadini, G. G. Xie, and O. Bonaventure, “Safe update of hybrid sdn networks,” IEEE/ACM Transactions on Networking, vol. 25, no. 3, pp. 1649–1662, 2017.
  • [143] K.-T. Foerster, S. Schmid, and S. Vissicchio, “Survey of consistent software-defined network updates,” IEEE Communications Surveys & Tutorials, vol. 21, no. 2, pp. 1435–1461, 2018.
  • [144] P. Francois, C. Filsfils, J. Evans, and O. Bonaventure, “Achieving sub-second igp convergence in large ip networks,” ACM SIGCOMM Computer Communication Review, vol. 35, no. 3, pp. 35–44, 2005.
  • [145] A. Shaikh, R. Dube, and A. Varma, “Avoiding instability during graceful shutdown of multiple ospf routers,” IEEE/ACM Transactions on Networking, vol. 14, no. 3, pp. 532–542, 2006.
  • [146] P. Francois and O. Bonaventure, “Avoiding transient loops during the convergence of link-state routing protocols,” IEEE/ACM Transactions on Networking, vol. 15, no. 6, pp. 1280–1292, 2007.
  • [147] J. Moy, P. Pillay-Esnault, and A. Lindem, “Graceful ospf restart,” RFC 3623, Tech. Rep., 2003.
  • [148] A. Ammireddy, “Expedited graceful ospf restart,” Nov. 1 2012, uS Patent App. 13/096,642.
  • [149] M. Shand and L. Ginsberg, “Restart signaling for is-is,” RFC 5306, October, Tech. Rep., 2008.
  • [150] R. Keralapura, C.-N. Chuah, and Y. Fan, “Optimal strategy for graceful network upgrade,” in Proceedings of the 2006 SIGCOMM workshop on Internet network management, 2006, pp. 83–88.
  • [151] S. Raza, Y. Zhu, and C.-N. Chuah, “Graceful network operations,” in IEEE INFOCOM 2009.   IEEE, 2009, pp. 289–297.
  • [152] ——, “Graceful network state migrations,” IEEE/ACM Transactions on Networking, vol. 19, no. 4, pp. 1097–1110, 2011.
  • [153] L. Vanbever, S. Vissicchio, C. Pelsser, P. Francois, and O. Bonaventure, “Seamless network-wide igp migrations,” in Proceedings of the ACM SIGCOMM 2011 conference, 2011, pp. 314–325.
  • [154] ——, “Lossless migrations of link-state igps,” IEEE/ACM Transactions on Networking, vol. 20, no. 6, pp. 1842–1855, 2012.
  • [155] F. Le, G. G. Xie, D. Pei, J. Wang, and H. Zhang, “Shedding light on the glue logic of the internet routing architecture,” in Proceedings of the ACM SIGCOMM 2008 conference on Data communication, 2008, pp. 39–50.
  • [156] L. Vanbever, S. Vissicchio, L. Cittadini, and O. Bonaventure, “When the cure is worse than the disease: The impact of graceful igp operations on bgp,” in 2013 Proceedings IEEE INFOCOM.   IEEE, 2013, pp. 2220–2228.
  • [157] Y. Wang, E. Keller, B. Biskeborn, J. Van Der Merwe, and J. Rexford, “Virtual routers on the move: live router migration as a network-management primitive,” ACM SIGCOMM Computer Communication Review, vol. 38, no. 4, pp. 231–242, 2008.
  • [158] E. Keller, J. Rexford, and J. E. van der Merwe, “Seamless bgp migration with router grafting.” in NSDI, 2010, pp. 235–248.
  • [159] S. Vissicchio, L. Vanbever, C. Pelsser, L. Cittadini, P. Francois, and O. Bonaventure, “Improving network agility with seamless bgp reconfigurations,” IEEE/ACM Transactions on Networking, vol. 21, no. 3, pp. 990–1002, 2012.
  • [160] R. Alimi, Y. Wang, and Y. R. Yang, “Shadow configuration as a network management primitive,” in Proceedings of the ACM SIGCOMM 2008 conference on Data communication, 2008, pp. 111–122.
  • [161] J. Rexford, D. Clark, and A. Vahdat, “A purpose-built global network: Google’s move to sdn.” Communications of the ACM, vol. 59, no. 3, pp. 46–54, 2016.
  • [162] M. Reitblatt, N. Foster, J. Rexford, and D. Walker, “Consistent updates for software-defined networks: Change you can believe in!” in Proceedings of the 10th ACM Workshop on Hot Topics in Networks, 2011, pp. 1–6.
  • [163] S. Brandt, K.-T. Förster, and R. Wattenhofer, “On consistent migration of flows in sdns,” in IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications, 2016, pp. 1–9.
  • [164] K.-T. Forster and R. Wattenhofer, “The power of two in consistent network updates: Hard loop freedom, easy flow migration,” in 2016 25th International Conference on Computer Communication and Networks (ICCCN), 2016, pp. 1–9.
  • [165] J. Zheng, H. Xu, G. Chen, and H. Dai, “Minimizing transient congestion during network update in data centers,” in 2015 IEEE 23rd International Conference on Network Protocols (ICNP), 2015, pp. 1–10.
  • [166] S. Dudycz, A. Ludwig, and S. Schmid, “Can’t touch this: Consistent network updates for multiple policies,” in 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), 2016, pp. 133–143.
  • [167] K.-T. Förster, R. Mahajan, and R. Wattenhofer, “Consistent updates in software defined networks: On dependencies, loop freedom, and blackholes,” in 2016 IFIP Networking Conference (IFIP Networking) and Workshops, 2016, pp. 1–9.
  • [168] A. Ludwig, S. Dudycz, M. Rost, and S. Schmid, “Transiently secure network updates,” ACM SIGMETRICS Performance Evaluation Review, vol. 44, no. 1, pp. 273–284, 2016.
  • [169] K.-T. Foerster, A. Ludwig, J. Marcinkowski, and S. Schmid, “Loop-free route updates for software-defined networks,” IEEE/ACM Transactions on Networking, vol. 26, no. 1, pp. 328–341, 2017.
  • [170] J. Edelman, S. S. Lowe, and M. Oswalt, Network Programmability and Automation: Skills for the Next-Generation Network Engineer.   ” O’Reilly Media, Inc.”, 2018.
  • [171] D. Rafique and L. Velasco, “Machine learning for network automation: Overview, architecture, and applications [invited tutorial],” Journal of Optical Communications and Networking, vol. 10, no. 10, pp. D126–D143, 2018.
  • [172] F. Paolucci, A. Sgambelluri, F. Cugini, and P. Castoldi, “Network telemetry streaming services in sdn-based disaggregated optical networks,” Journal of Lightwave Technology, vol. 36, no. 15, pp. 3142–3149, 2018.
  • [173] Y. Zhang, X. Gong, Y. Hu, W. Wang, and X. Que, “Sdnmp: Enabling sdn management using traditional nms,” in 2015 IEEE International Conference on Communication Workshop (ICCW), 2015, pp. 357–362.
  • [174] R. Katiyar, P. Pawar, A. Gupta, and K. Kataoka, “Auto-configuration of sdn switches in sdn/non-sdn hybrid network,” in Proceedings of the Asian Internet Engineering Conference, 2015, pp. 48–53.
  • [175] R. Amin, N. Shah, B. Shah, and O. Alfandi, “Auto-configuration of acl policy in case of topology change in hybrid sdn,” IEEE Access, vol. 4, pp. 9437–9450, 2016.
  • [176] C. Chu, K. Xi, M. Luo, and H. J. Chao, “Congestion-aware single link failure recovery in hybrid sdn networks,” in 2015 IEEE Conference on Computer Communications (INFOCOM), 2015, pp. 1086–1094.
  • [177] R. Yasunaga, Y. Nakayama, T. Mochida, Y. Kimura, T. Yoshida, and K. Suzuki, “Optimal load balancing method for symmetrically routed hybrid sdn networks,” in 2015 21st Asia-Pacific Conference on Communications (APCC), 2015, pp. 234–238.
  • [178] T. Shozi, P. Mudali, M. O. Adigun, and H. Kobo, “A data plane multi-path load balancing mechanism for hybrid software defined networks in different topologies,” in 2019 International Conference on Advances in Big Data, Computing and Data Communication Systems (icABCD), 2019, pp. 1–6.
  • [179] X. Jia, Y. Jiang, and J. Zhu, “Link fault protection and traffic engineering in hybrid sdn networks,” in IEEE INFOCOM 2018 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), 2018, pp. 853–858.
  • [180] Z. Yang and K. L. Yeung, “Sdn candidate selection in hybrid ip/sdn networks for single link failure protection,” IEEE/ACM Transactions on Networking, vol. 28, no. 1, pp. 312–321, 2020.
  • [181] E. Rosen, A. Viswanathan, and R. Callon, “Multiprotocol label switching architecture,” Internet Requests for Comments, RFC Editor, RFC 3031, January 2001. [Online]. Available: http://www.rfc-editor.org/rfc/rfc3031.txt
  • [182] E. Mannie, “Generalized multi-protocol label switching (gmpls) architecture,” Internet Requests for Comments, RFC Editor, RFC 3945, October 2004.
  • [183] M. R. Abbasi, A. Guleria, and M. S. Devi, “Traffic engineering in software defined networks: A survey,” Journal of Telecommunications and Information Technology, 2016.
  • [184] I. F. Akyildiz, A. Lee, P. Wang, M. Luo, and W. Chou, “Research challenges for traffic engineering in software defined networks,” IEEE Network, vol. 30, no. 3, pp. 52–58, 2016.
  • [185] T. Yingying Cheng and X. Jia, “Compressive traffic monitoring in hybrid sdn,” IEEE Journal on Selected Areas in Communications, vol. 36, no. 12, pp. 2731–2743, 2018.
  • [186] M. Polverini, A. Baiocchi, A. Cianfrani, A. Iacovazzi, and M. Listanti, “The power of sdn to improve the estimation of the isp traffic matrix through the flow spread concept,” IEEE Journal on Selected Areas in Communications, vol. 34, no. 6, pp. 1904–1913, 2016.
  • [187] J. Zhang, M. Ye, Z. Guo, C.-Y. Yen, and H. J. Chao, “Cfr-rl: Traffic engineering with reinforcement learning in sdn,” arXiv preprint arXiv:2004.11986, 2020.
  • [188] Y. Tian, W. Chen, and C. Lea, “Monitor placement for link latency measurement in hybrid sdns,” IEEE Transactions on Network and Service Management, pp. 1–1, 2020.
  • [189] M. Casado, T. Koponen, S. Shenker, and A. Tootoonchian, “Fabric: A retrospective on evolving sdn,” in Proceedings of the First Workshop on Hot Topics in Software Defined Networks, ser. HotSDN ’12, 2012, p. 85–90.
  • [190] X. Tu, X. Li, J. Zhou, and S. Chen, “Splicing mpls and openflow tunnels based on sdn paradigm,” in 2014 IEEE International Conference on Cloud Engineering, 2014, pp. 489–493.
  • [191] S. Agarwal, M. Kodialam, and T. V. Lakshman, “Traffic engineering in software defined networks,” in 2013 Proceedings IEEE INFOCOM, 2013, pp. 2211–2219.
  • [192] J. He and W. Song, “Achieving near-optimal traffic engineering in hybrid software defined networks,” in 2015 IFIP Networking Conference (IFIP Networking), 2015, pp. 1–9.
  • [193] C. Ren, S. Bai, Y. Wang, and Y. Li, “Achieving near-optimal traffic engineering using a distributed algorithm in hybrid sdn,” IEEE Access, vol. 8, pp. 29 111–29 124, 2020.
  • [194] Y. Guo, Z. Wang, X. Yin, X. Shi, and J. Wu, “Traffic engineering in sdn/ospf hybrid network,” in 2014 IEEE 22nd International Conference on Network Protocols, 2014, pp. 563–568.
  • [195] Y. Nakahodo, T. Naito, and E. Oki, “Implementation of smart-ospf in hybrid software-defined network,” in 2014 4th IEEE International Conference on Network Infrastructure and Digital Content, 2014, pp. 374–378.
  • [196] P. Sharma, S. Banerjee, S. Tandel, R. Aguiar, R. Amorim, and D. Pinheiro, “Enhancing network management frameworks with sdn-like control,” in 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013), 2013, pp. 688–691.
  • [197] J. Galán-Jiménez, M. Polverini, and A. Cianfrani, “A scalable and error-tolerant solution for traffic matrix assessment in hybrid ip/sdn networks,” IEEE Transactions on Network and Service Management, vol. 17, no. 1, pp. 251–264, 2020.
  • [198] C. Ren, S. Wang, J. Ren, X. Wang, T. Song, and D. Zhang, “Enhancing traffic engineering performance and flow manageability in hybrid sdn,” in 2016 IEEE Global Communications Conference (GLOBECOM), 2016, pp. 1–7.
  • [199] Y. Guo, Z. Wang, X. Yin, X. Shi, J. Wu, and H. Zhang, “Incremental deployment for traffic engineering in hybrid sdn network,” in 2015 IEEE 34th International Performance Computing and Communications Conference (IPCCC), 2015, pp. 1–8.
  • [200] W. Wang, W. He, and J. Su, “Enhancing the effectiveness of traffic engineering in hybrid sdn,” in 2017 IEEE International Conference on Communications (ICC), 2017, pp. 1–6.
  • [201] Y. Guo, Z. Wang, X. Yin, X. Shi, and J. Wu, “Optimize routing in hybrid sdn network with changing traffic,” in 2017 26th International Conference on Computer Communication and Networks (ICCCN), 2017, pp. 1–8.
  • [202] L. Davoli, L. Veltri, P. L. Ventre, G. Siracusano, and S. Salsano, “Traffic engineering with segment routing: Sdn-based architectural design and open source implementation,” in 2015 Fourth European Workshop on Software Defined Networks, 2015, pp. 111–112.
  • [203] I. Šeremet and S. Čaušević, “Advancing multiprotocol label switching traffic engineering with segment routing in software defined network environment,” in 2020 19th International Symposium INFOTEH-JAHORINA (INFOTEH), 2020, pp. 1–6.
  • [204] W. Wang, W. He, and J. Su, “Boosting the benefits of hybrid sdn,” in 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), 2017, pp. 2165–2170.
  • [205] Y. Sinha, S. Bhatia, V. S. Shekhawat, and G. S. S. Chalapathi, “Mpls based hybridization in sdn,” in 2017 Fourth International Conference on Software Defined Systems (SDS), 2017, pp. 156–161.
  • [206] H. E. Egilmez, S. T. Dane, K. T. Bagci, and A. M. Tekalp, “Openqos: An openflow controller design for multimedia delivery with end-to-end quality of service over software-defined networks,” in Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, 2012, pp. 1–8.
  • [207] F. Ongaro, E. Cerqueira, L. Foschini, A. Corradi, and M. Gerla, “Enhancing the quality level support for real-time multimedia applications in software-defined networks,” in 2015 International Conference on Computing, Networking and Communications (ICNC), 2015, pp. 505–509.
  • [208] J. Yan, H. Zhang, Q. Shuai, B. Liu, and X. Guo, “Hiqos: An sdn-based multipath qos solution,” China Communications, vol. 12, no. 5, pp. 123–133, 2015.
  • [209] C. Sieber, A. Blenk, D. Hock, M. Scheib, T. Höhn, S. Köhler, and W. Kellerer, “Network configuration with quality of service abstractions for sdn and legacy networks,” in 2015 IFIP/IEEE International Symposium on Integrated Network Management (IM), 2015, pp. 1135–1136.
  • [210] S. Lin, I. F. Akyildiz, P. Wang, and M. Luo, “Qos-aware adaptive routing in multi-layer hierarchical software defined networks: A reinforcement learning approach,” in 2016 IEEE International Conference on Services Computing (SCC), 2016, pp. 25–33.
  • [211] Y. Bi, G. Han, C. Lin, Y. Peng, H. Pu, and Y. Jia, “Intelligent quality of service aware traffic forwarding for software-defined networking/open shortest path first hybrid industrial internet,” IEEE Transactions on Industrial Informatics, vol. 16, no. 2, pp. 1395–1405, 2020.
  • [212] A. Mondal and S. Misra, “Flowman: Qos-aware dynamic data flow management in software-defined networks,” IEEE Journal on Selected Areas in Communications, vol. 38, no. 7, pp. 1366–1373, 2020.
  • [213] C. Lin, K. Wang, and G. Deng, “A qos-aware routing in sdn hybrid networks,” Procedia Computer Science, vol. 110, pp. 242 – 249, 2017, 14th International Conference on Mobile Systems and Pervasive Computing (MobiSPC 2017) / 12th International Conference on Future Networks and Communications (FNC 2017) / Affiliated Workshops.
  • [214] L. Alouache, N. Nguyen, M. Aliouat, and R. Chelouah, “Hsdn-gra: A hybrid software-defined networking-based geographic routing protocol with multi-agent approach,” International Journal of Communication Systems, vol. 33, no. 15, p. e4521, 2020.
  • [215] N. Huin, M. Rifai, F. Giroire, D. Lopez Pacheco, G. Urvoy-Keller, and J. Moulierac, “Bringing energy aware routing closer to reality with sdn hybrid networks,” in GLOBECOM 2017 - 2017 IEEE Global Communications Conference, 2017, pp. 1–7.
  • [216] NGMN Alliance, “Description of Network Slicing Concept v.1.0.8,” https://www.ngmn.org/wp-content/uploads/Publications/2016/161010_NGMN_Network_Slicing_framework_v1.0.8.pdf, September 2016, Last accessed: 2020-04-13.
  • [217] ETSI IGS NFV, “Network Functions Virtualisation (NFV); Architectural Framework v.1.2.1,” https://docbox.etsi.org/ISG/NFV/Open/Publications_pdf/Specs-Reports/NFV%20002v1.2.1%20-%20GS%20-%20NFV%20Architectural%20Framework.pdf, December 2014, Last accessed: 2020-04-13.
  • [218] T. Dillon, C. Wu, and E. Chang, “Cloud computing: Issues and challenges,” in 2010 24th IEEE International Conference on Advanced Information Networking and Applications, April 2010, pp. 27–33.
  • [219] K. Patel, S. Patel, P. Scholar, and C. Salazar, “Internet of things-iot: Definition, characteristics, architecture, enabling technologies, application & future challenges,” International Journal of Engineering Science and Computing (IJESC), vol. 6, pp. 6122–6131, 05 2016.
  • [220] M. Pilkington, In Research Handbook on Digital Transformations. Blockchain technology: principles and applications.   Edward Elgar, September 2016, Last accessed: 2020-04-13.
  • [221] ITU, “IMT Vision – Framework and overall objectives of the future development of IMT for 2020 and beyond,” https://www.itu.int/dms_pubrec/itu-r/rec/m/R-REC-M.2083-0-201509-I!!PDF-E.pdf, September 2015, Last accessed: 2020-04-14.
  • [222] A. Anand, G. de Veciana, and S. Shakkottai, “Joint scheduling of urllc and embb traffic in 5g wireless networks,” IEEE/ACM Transactions on Networking, pp. 1–14, 2020.
  • [223] C. Bockelmann, N. K. Pratas, G. Wunder, S. Saur, M. Navarro, D. Gregoratti, G. Vivier, E. De Carvalho, Y. Ji, C. Stefanovic, P. Popovski, Q. Wang, M. Schellmann, E. Kosmatos, P. Demestichas, M. Raceala-Motoc, P. Jung, S. Stanczak, and A. Dekorsy, “Towards massive connectivity support for scalable mmtc communications in 5g networks,” IEEE Access, vol. 6, pp. 28 969–28 992, 2018.
  • [224] M. Alsenwi, N. H. Tran, M. Bennis, A. Kumar Bairagi, and C. S. Hong, “embb-urllc resource slicing: A risk-sensitive approach,” IEEE Communications Letters, vol. 23, no. 4, pp. 740–743, 2019.
  • [225] Jian Yang and S. Roy, “On joint transmitter and receiver optimization for multiple-input-multiple-output (mimo) transmission systems,” IEEE Transactions on Communications, vol. 42, no. 12, pp. 3221–3231, 1994.
  • [226] ETSI IGS MEC, “Multi-access Edge Computing (MEC); MEC Testing Framework v.2.1.1,” https://www.etsi.org/deliver/etsi_gr/MEC-DEC/001_099/025/02.01.01_60/gr_MEC-DEC025v020101p.pdf, June 2019, Last accessed: 2020-04-14.
  • [227] 3rd Generation Partnership Project, “Interface between the Control Plane and the User Plane nodes v16.5.0 Release 16),” https://www.etsi.org/deliver/etsi_ts/129200_129299/129244/16.05.00_60/ts_129244v160500p.pdf, November 2020, Last accessed: 2021-02-15.
  • [228] Open Networking Foundation, “Relationship of SDN and NFV. Issue 1,” https://www.opennetworking.org/images/stories/downloads/sdn-resources/technical-reports/onf2015.310_Architectural_comparison.08-2.pdf, October 2015, Last accessed: 2020-04-30.
  • [229] ——, “SDN architecture. Issue 1.1,” https://www.opennetworking.org/images/stories/downloads/sdn-resources/technical-reports/TR-521_SDN_Architecture_issue_1.1.pdf, June 2016, Last accessed: 2020-04-30.
  • [230] ——, “SDN Architecture for Transport Networks.” https://www.opennetworking.org/wp-content/uploads/2014/10/SDN_Architecture_for_Transport_Networks_TR522.pdf, March 2015, Last accessed: 2020-04-30.
  • [231] A. Zafeiropoulos, A. Gavras, A. Tzanakaki, A. Albanese, A. Kousaridas, A. Weit, B. Sayadi, B. Jou, C. Bernardos, C. Benzaid, C. Mannweiler, D. Camps-Mur, D. Breitgand, D. Estevez, D. Navratil, D. Mi, D. Lopez, D. Klonidis, E. Mutafungwa, E. Fotopoulou, E. Kafetzakis, E. Pateromichelakis, E. Biton, F. Tesema, G. Kalfas, H. Karl, J. Bartelt, J. Gutiérrez, J. Cosmas, J. Thomson, J. Giménez, J. Alcaraz Calero, J. Mangues-Bafalluy, K. Katsalis, L. Gallo, M. Gramaglia, M. Spada, M. Salih, N. Nikaein, N. Jawad, N. Maletic, Ö. Bulakci, P. Demestichas, P. Hasselmeyer, Q. Wang, Q. Wei, R. Ustok, R. Blom, S. Pontarelli, S. Keskin, S. Salsano, S. Parker, T. Deiss, U. Acar, X. Li, and Y. Zhang, 5G PPP Architecture Working Group: View on 5G Architecture v.3.0.   Belgium: European Commission, February 2020, vol. Version 3.0.
  • [232] 3rd Generation Partnership Project, “System architecture for the 5G System (5GS). v.16.4.0,” https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=3144, March 2020, Last accessed: 2020-04-30.
  • [233] Open Networking Foundation, “Applying SDN Architecture to 5G Slicing. Issue 1,” https://www.opennetworking.org/images/stories/downloads/sdn-resources/technical-reports/Applying_SDN_Architecture_to_5G_Slicing_TR-526.pdf, April 2016, Last accessed: 2020-04-30.
  • [234] ETSI GS NFV-EVE, “Network Functions Virtualisation (NFV); Ecosystem; Report on SDN Usage in NFV Architectural Framework. v.1.1.1,” https://www.etsi.org/deliver/etsi_gs/NFV-EVE/001_099/005/01.01.01_60/gs_nfv-eve005v010101p.pdf, December 2015, Last accessed: 2020-04-30.
  • [235] ETSI GR MEC, “Mobile Edge Computing (MEC); Deployment of Mobile Edge Computing in an NFV environment v.1.1.1,” https://www.etsi.org/deliver/etsi_gr/MEC/001_099/017/01.01.01_60/gr_MEC017v010101p.pdf, February 2018, Last accessed: 2020-04-30.
  • [236] T. Wood, K. Ramakrishnan, J. Hwang, G. Liu, and W. Zhang, “Toward a software-based network: integrating software defined networking and network function virtualization,” IEEE Network, vol. 29, no. 3, pp. 36–41, May 2015.
  • [237] A. Hakiri and P. Berthou, “Leveraging sdn for the 5g networks: Trends, prospects and challenges,” 2015.
  • [238] R. F. Moyano, D. F. Cambronero, and L. B. Triana, “A user-centric sdn management architecture for nfv-based residential networks,” Computer Standards & Interfaces, vol. 54, pp. 279–292, 2017.
  • [239] J. Ordonez-Lucena, P. Ameigeiras, D. Lopez, J. J. Ramos-Munoz, J. Lorca, and J. Folgueira, “Network slicing for 5g with sdn/nfv: Concepts, architectures, and challenges,” IEEE Communications Magazine, vol. 55, no. 5, pp. 80–87, May 2017.
  • [240] S. Sun, M. Kadoch, L. Gong, and B. Rong, “Integrating network function virtualization with sdr and sdn for 4g/5g networks,” IEEE Network, vol. 29, no. 3, pp. 54–59, May 2015.
  • [241] A. A. Barakabitze, L. Sun, I.-H. Mkwawa, and E. Ifeachor, “A novel qoe-aware sdn-enabled, nfv-based management architecture for future multimedia applications on 5g systems,” arXiv preprint arXiv:1904.09917, 2019.
  • [242] C. Bouras, A. Kollia, and A. Papazois, “Sdn & nfv in 5g: Advancements and challenges,” in 2017 20th Conference on Innovations in Clouds, Internet and Networks (ICIN).   IEEE, March 2017, pp. 107–111.
  • [243] M. N. Irshad, L. Du, I. A. Khoso, T. B. Javed, and M. M. Aslam, “A hybrid solution of sdn architecture for 5g mobile communication to improve data rate transmission,” in 2019 28th Wireless and Optical Communications Conference (WOCC).   IEEE, May 2019, pp. 1–5.
  • [244] B. Lantz, B. Heller, and N. McKeown, “A network in a laptop: Rapid prototyping for software-defined networks,” in Proceedings of the 9th ACM SIGCOMM Workshop on Hot Topics in Networks, ser. Hotnets-IX.   New York, NY, USA: Association for Computing Machinery, 2010.
  • [245] FloodLight Controller, “Floodlight SDN OpenFlow Controller v1.2,” https://github.com/floodlight/floodlight, February 2016, Last accessed: 2020-04-16.
  • [246] A. Tawbeh, H. Safa, and A. R. Dhaini, “A hybrid sdn/nfv architecture for future lte networks,” in 2017 IEEE International Conference on Communications (ICC).   IEEE, May 2017, pp. 1–6.
  • [247] J. Zhang, Z. Wang, N. Ma, T. Huang, and Y. Liu, “Enabling efficient service function chaining by integrating nfv and sdn: architecture, challenges and opportunities,” IEEE Network, vol. 32, no. 6, pp. 152–159, November 2018.
  • [248] J. Nunez-Martinez, J. Baranda, and J. Mangues-Bafalluy, “A service-based model for the hybrid software defined wireless mesh backhaul of small cells,” in 2015 11th International Conference on Network and Service Management (CNSM), Nov 2015, pp. 390–393.
  • [249] 5G GROWTH, “5G GROWTH: 5G-enabled Growth in Vertical Industries,” https://5growth.eu/project/, 2019, Last accessed: 2020-09-24.
  • [250] 5G SOLUTIONS, “5G SOLUTIONS: 5G Solutions for European Citizens,” https://www.5gsolutionsproject.eu/, 2019, Last accessed: 2020-09-24.
  • [251] 5G HEART, “5G HEART: 5G HEalth AquacultuRe and Transport validation trials,” https://5gheart.org/, 2019, Last accessed: 2020-09-24.
  • [252] 5G VINNI, “5G VINNI: Verticals Innovation Infrastructure,” https://www.5g-vinni.eu, 2018, Last accessed: 2020-05-03.
  • [253] 5G EVE, “5G EVE: European Validation platform for Extensive trials,” https://www.5g-eve.eu/concept/, 2018, Last accessed: 2020-05-03.
  • [254] 5G VINNI, “D1.1 Design of infrastructure architecture and subsystems,” https://zenodo.org/record/2668754#.Xq7x-y8RpQI, December 2018, Last accessed: 2020-05-03.
  • [255] 5G EVE, “D2.1: Initial detailed architectural and functional site facilities description,” https://zenodo.org/record/3540439#.Xq7t6S8RpQI, October 2018, Last accessed: 2020-05-03.
  • [256] M. Paliwal and D. Shrimankar, “Effective resource management in sdn enabled data center network based on traffic demand,” IEEE Access, vol. 7, pp. 69 698–69 706, 2019.
  • [257] D. A. Chekired, M. A. Togou, and L. Khoukhi, “A hybrid sdn path computation for scaling data centers networks,” in 2018 IEEE Global Communications Conference (GLOBECOM).   IEEE, Dec 2018, pp. 1–6.
  • [258] K. Qiu, S. Huang, Q. Xu, J. Zhao, X. Wang, and S. Secci, “Paracon: A parallel control plane for scaling up path computation in sdn,” IEEE Transactions on Network and Service Management, vol. 14, no. 4, pp. 978–990, Dec 2017.
  • [259] S. Kaur, J. Singh, and N. S. Ghumman, “Network programmability using pox controller,” in ICCCS International Conference on Communication, Computing & Systems, IEEE, vol. 138, 2014, p. 70.
  • [260] B. Wang, Z. Qi, R. Ma, H. Guan, and A. V. Vasilakos, “A survey on data center networking for cloud computing,” Computer Networks, vol. 91, pp. 528–547, 2015.
  • [261] J. Son and R. Buyya, “A taxonomy of software-defined networking (sdn)-enabled cloud computing,” ACM Computing Surveys (CSUR), vol. 51, no. 3, pp. 1–36, May 2018.
  • [262] Y. Jararweh, M. Al-Ayyoub, E. Benkhelifa, M. Vouk, A. Rindos et al., “Software defined cloud: Survey, system and evaluation,” Future Generation Computer Systems, vol. 58, pp. 56–74, 2016.
  • [263] A. N. Toosi, R. N. Calheiros, and R. Buyya, “Interconnected cloud computing environments: Challenges, taxonomy, and survey,” ACM Computing Surveys (CSUR), vol. 47, no. 1, pp. 1–47, May 2014.
  • [264] R. Jain and S. Paul, “Network virtualization and software defined networking for cloud computing: a survey,” IEEE Communications Magazine, vol. 51, no. 11, pp. 24–31, November 2013.
  • [265] D. K. Hong, Y. Ma, S. Banerjee, and Z. M. Mao, “Incremental deployment of sdn in hybrid enterprise and isp networks,” in Proceedings of the Symposium on SDN Research, ser. SOSR ’16.   Association for Computing Machinery, 2016.
  • [266] S. Khorsandroo and A. S. Tosun, “An experimental investigation of sdn controller live migration in virtual data centers,” in 2017 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN), 2017, pp. 309–314.
  • [267] Y. Guo, Z. Wang, X. Yin, X. Shi, and J. Wu, “Traffic engineering in sdn/ospf hybrid network,” in 2014 IEEE 22nd International Conference on Network Protocols.   IEEE, Oct 2014, pp. 563–568.
  • [268] K. Ashton et al., “That ‘internet of things’ thing,” RFID journal, vol. 22, no. 7, pp. 97–114, 2009.
  • [269] IoT Analytics, “IoT Analytics: Market Insights for the Internet of Things,” https://iot-analytics.com, 2019, Last accessed: 2020-04-28.
  • [270] I. Farris, T. Taleb, Y. Khettab, and J. Song, “A Survey on Emerging SDN and NFV Security Mechanisms for IoT Systems,” IEEE Communications Surveys Tutorials, vol. 21, no. 1, pp. 812–837, 2019.
  • [271] M. Ambrosin, A. Compagno, M. Conti, C. Ghali, and G. Tsudik, “Security and privacy analysis of national science foundation future internet architectures,” IEEE Communications Surveys Tutorials, vol. 20, no. 2, pp. 1418–1442, 2018.
  • [272] S. Kraijak and P. Tuwanut, “A survey on iot architectures, protocols, applications, security, privacy, real-world implementation and future trends,” WiCOM, January 2015.
  • [273] G. Marques, N. Garcia, and N. Pombo, A Survey on IoT: Architectures, Elements, Applications, QoS, Platforms and Security Concepts.   Cham: Springer International Publishing, 2017, pp. 115–130.
  • [274] S. Krčo, B. Pokrić, and F. Carrez, “Designing iot architecture(s): A european perspective,” in 2014 IEEE World Forum on Internet of Things (WF-IoT), March 2014, pp. 79–84.
  • [275] S. Fichera, M. Gharbaoui, P. Castoldi, B. Martini, and A. Manzalini, “On experimenting 5G: Testbed set-up for SDN orchestration across network cloud and IoT domains,” in 2017 IEEE Conference on Network Softwarization (NetSoft), July 2017.
  • [276] L. Tello-Oquendo, S.-C. Lin, I. F. Akyildiz, and V. Pla, “Software-defined architecture for qos-aware iot deployments in 5g systems,” Ad Hoc Networks, vol. 93, p. 101911, 2019.
  • [277] S. S. Bhunia and M. Gurusamy, “Dynamic attack detection and mitigation in IoT using SDN,” in 2017 27th International Telecommunication Networks and Applications Conference (ITNAC), Nov 2017.
  • [278] C. Gonzalez, S. M. Charfadine, O. Flauzac, and F. Nolot, “SDN-based security framework for the IoT in distributed grid,” in 2016 International Multidisciplinary Conference on Computer and Energy Science (SpliTech), July 2016, pp. 1–5.
  • [279] C. Li, Z. Qin, E. Novak, and Q. Li, “Securing SDN Infrastructure of IoT–Fog Networks From MitM Attacks,” IEEE Internet of Things Journal, vol. 4, no. 5, pp. 1156–1164, 2017.
  • [280] D. Bendouda, A. Rachedi, and H. Haffaf, “An hybrid and proactive architecture based on SDN for Internet of Things,” in 2017 13th International Wireless Communications and Mobile Computing Conference (IWCMC), June 2017, pp. 951–956.
  • [281] A. Lee, X. Wang, H. Nguyen, and I. Ra, “A hybrid software defined networking architecture for next-generation iots.” KSII Transactions on Internet & Information Systems, vol. 12, no. 2, 2018.
  • [282] J. Zhang, K. Xi, M. Luo, and H. J. Chao, “Load balancing for multiple traffic matrices using sdn hybrid routing,” in 2014 IEEE 15th International Conference on High Performance Switching and Routing (HPSR), July 2014, pp. 44–49.
  • [283] H. Saadeh, W. Almobaideen, K. E. Sabri, and M. Saadeh, “Hybrid SDN-ICN Architecture Design for the Internet of Things,” in 2019 Sixth International Conference on Software Defined Systems (SDS), June 2019, pp. 96–101.
  • [284] H. K. Saadeh, W. Almobaideen, and K. E. Sabri, “Ppustman: Privacy-aware publish/subscribe iot mvc architecture using information centric networking,” Modern Applied Science, vol. 12, no. 5, p. 128, 2018.
  • [285] Q. Zhang, X. Wang, M. Huang, K. Li, and S. K. Das, “Software defined networking meets information centric networking: A survey,” IEEE Access, vol. 6, pp. 39 547–39 563, 2018.
  • [286] M. Gerla, E. Lee, G. Pau, and U. Lee, “Internet of vehicles: From intelligent grid to autonomous cars and vehicular clouds,” in 2014 IEEE World Forum on Internet of Things (WF-IoT), March 2014, pp. 241–246.
  • [287] W. Zhu, D. Gao, W. Zhao, H. Zhang, and H.-P. Chiang, “Sdn-enabled hybrid emergency message transmission architecture in internet-of-vehicles,” Enterprise Information Systems, vol. 12, no. 4, pp. 471–491, 2018.
  • [288] L. Alouache, N. Nguyen, M. Aliouat, and R. Chelouah, “Toward a hybrid sdn architecture for v2v communication in iov environment,” in 2018 Fifth International Conference on Software Defined Systems (SDS), April 2018, pp. 93–99.
  • [289] S. Nakamoto, “Bitcoin: A peer-to-peer electronic cash system,” https://git.dhimmel.com/bitcoin-whitepaper/, Manubot, Tech. Rep., 2019, Last accessed: 2020-04-24.
  • [290] M. Sharples and J. Domingue, “The blockchain and kudos: A distributed system for educational record, reputation and reward,” in Adaptive and Adaptable Learning, K. Verbert, M. Sharples, and T. Klobučar, Eds.   Cham: Springer International Publishing, 2016, pp. 490–496.
  • [291] T.-T. Kuo, H.-E. Kim, and L. Ohno-Machado, “Blockchain distributed ledger technologies for biomedical and health care applications,” Journal of the American Medical Informatics Association, vol. 24, no. 6, pp. 1211–1220, 09 2017.
  • [292] S. Huckle, R. Bhattacharya, M. White, and N. Beloff, “Internet of things, blockchain and shared economy applications,” Procedia Computer Science, vol. 98, pp. 461 – 466, 2016.
  • [293] T. M. Fernández-Caramés and P. Fraga-Lamas, “A review on the use of blockchain for the internet of things,” IEEE Access, vol. 6, pp. 32 979–33 001, 2018.
  • [294] P. K. Sharma, M. Chen, and J. H. Park, “A software defined fog node based distributed blockchain cloud architecture for iot,” IEEE Access, vol. 6, pp. 115–124, 2018.
  • [295] P. K. Sharma, S. Singh, Y. Jeong, and J. H. Park, “DistBlockNet: A Distributed Blockchains-Based Secure SDN Architecture for IoT Networks,” IEEE Communications Magazine, vol. 55, no. 9, pp. 78–85, 2017.
  • [296] R. Chaudhary, A. Jindal, G. S. Aujla, S. Aggarwal, N. Kumar, and K.-K. R. Choo, “BEST: Blockchain-based secure energy trading in SDN-enabled intelligent transportation system,” Computers & Security, vol. 85, pp. 288 – 299, 2019.
  • [297] A. Jindal, G. S. Aujla, and N. Kumar, “SURVIVOR: A blockchain based edge-as-a-service framework for secure energy trading in SDN-enabled vehicle-to-grid environment,” Computer Networks, vol. 153, pp. 36 – 48, 2019.
  • [298] L. Xie, Y. Ding, H. Yang, and X. Wang, “Blockchain-Based Secure and Trustworthy Internet of Things in SDN-Enabled 5G-VANETs,” IEEE Access, vol. 7, pp. 56 656–56 666, 2019.
  • [299] J. Gao, K. O. O. Agyekum, E. B. Sifah, K. N. Acheampong, Q. Xia, X. Du, M. Guizani, and H. Xia, “A Blockchain-SDN enabled Internet of Vehicles Environment for Fog Computing and 5G Networks,” IEEE Internet of Things Journal, pp. 1–1, 2019.
  • [300] C.-Y. Hong, S. Kandula, R. Mahajan, M. Zhang, V. Gill, M. Nanduri, and R. Wattenhofer, “Achieving high utilization with software-driven wan,” in Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, ser. SIGCOMM ’13, 2013, p. 15–26.
  • [301] Aruba Networks, “SD-Branch Whitepaper: Revolutionizing the branch for todays digital era,” https://www.arubanetworks.com/assets/so/SO_SDBranch.pdf, 2019, Last accessed: 2020-04-26.
  • [302] CISCO, “SD-Branch Whitepaper: Transforming branch infrastructure for the digital economy,” https://www.cisco.com/c/dam/en/us/solutions/collateral/enterprise-networks/sd-branch/sd-branch-whitepaper.pdf, 2019, Last accessed: 2020-04-26.
  • [303] fortiNet, “SD-Branch Whitepaper: Secures the Network Edge at the Branch,” https://www.fortinet.com/content/dam/fortinet/assets/solution-guides/sb-sd%20branch-solution.pdf, 2019, Last accessed: 2020-04-26.
  • [304] Versa Networks, “SD-Branch Whitepaper: Software-Defined WAN and Security for the Enterprise,” https://go.versa-networks.com/l/633831/2019-02-06/4vy9/633831/18996/versa_pb_enterpriseprodbroch__01_2__1_.pdf, 2019, Last accessed: 2020-04-26.
  • [305] S. Wang, C. Chou, and C. Yang, “Estinet openflow network simulator and emulator,” IEEE Communications Magazine, vol. 51, no. 9, pp. 110–117, 2013.
  • [306] T. R. Riley, George F.and Henderson, The ns-3 Network Simulator.   Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 15–34.
  • [307] B. Schlinker, “MiniNEXT: MiniNet Extension,” https://github.com/USC-NSL/miniNExT, 2014, Last accessed: 2020-04-15.
  • [308] H. Mostafaei, G. Lospoto, R. di Lallo, M. Rimondini, and G. Di Battista, “Sdnetkit: A testbed for experimenting sdn in multi-domain networks,” in 2017 IEEE Conference on Network Softwarization (NetSoft), 2017, pp. 1–6.
  • [309] V. Antonenko and R. Smelyanskiy, “Global network modelling based on mininet approach.” in Proceedings of the Second ACM SIGCOMM Workshop on Hot Topics in Software Defined Networking, ser. HotSDN ’13.   New York, NY, USA: Association for Computing Machinery, 2013, p. 145–146.
  • [310] B. Lantz and B. O’Connor, “A mininet-based virtual testbed for distributed sdn development,” in Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, ser. SIGCOMM ’15.   New York, NY, USA: Association for Computing Machinery, 2015, p. 365–366.
  • [311] P. Wette, M. Dräxler, A. Schwabe, F. Wallaschek, M. H. Zahraee, and H. Karl, “Maxinet: Distributed emulation of software-defined networks,” in 2014 IFIP Networking Conference, 2014, pp. 1–9.
  • [312] A. R. Roy, M. F. Bari, M. F. Zhani, R. Ahmed, and R. Boutaba, “Dot: Distributed openflow testbed,” SIGCOMM Comput. Commun. Rev., vol. 44, no. 4, p. 367–368, Aug. 2014.
  • [313] H. Niaz, S. Addagatla, N. Raje, Y. Zhu, and J. Padhye, “Open network emulator (one), production grade emulation at scale,” SIGCOMM, 2018.
  • [314] Y. KANAUMI, S. ichi SAITO, E. KAWAI, S. ISHII, K. KOBAYASHI, and S. SHIMOJO, “Rise: A wide-area hybrid openflow network testbed,” IEICE Transactions on Communications, vol. E96.B, no. 1, pp. 108–118, 2013.
  • [315] M. Suñé, L. Bergesio, H. Woesner, T. Rothe, A. Köpsel, D. Colle, B. Puype, D. Simeonidou, R. Nejabati, M. Channegowda, M. Kind, T. Dietz, A. Autenrieth, V. Kotronis, E. Salvadori, S. Salsano, M. Körner, and S. Sharma, “Design and implementation of the ofelia fp7 facility: The european openflow testbed,” Computer Networks, vol. 61, pp. 132 – 150, 2014, special issue on Future Internet Testbeds – Part I.
  • [316] J. Alcorn, S. Melton, and C. E. Chow, “Sdn on-the-go (otg) physical testbed,” in 2017 IEEE Conference on Dependable and Secure Computing, 2017, pp. 202–208.
  • [317] T.-T. Chu, V. Tong, H. A. Tran, S. Souihi, D. Q. Tran, and A. Mellouk, “Nextlab: A new hybrid testbed and development platform for software-defined networking,” in Proceedings of the Tenth International Symposium on Information and Communication Technology, ser. SoICT 2019.   New York, NY, USA: Association for Computing Machinery, 2019, p. 186–190.
  • [318] BIRD, “The bird internet routing daemon,” https://bird.network.cz, 2010, Last accessed: 2020-04-17.
  • [319] H. H. Liu, Y. Zhu, J. Padhye, J. Cao, S. Tallapragada, N. P. Lopes, A. Rybalchenko, G. Lu, and L. Yuan, “Crystalnet: Faithfully emulating large production networks,” in Proceedings of the 26th Symposium on Operating Systems Principles, ser. SOSP ’17.   New York, NY, USA: Association for Computing Machinery, 2017, p. 599–613.
  • [320] Microsoft, “Azure: Microsoft cloud-computing platform,” https://azure.microsoft.com/en-gb/, 2010, Last accessed: 2020-04-17.
  • [321] Japan National Institute of Information and Communications Technology, “Japan Gigabit Network,” https://testbed.nict.go.jp/jgn/english/index.html, April 2011, Last accessed: 2020-04-20.
  • [322] R. Sherwood, G. Gibb, K.-K. Yap, G. Appenzeller, M. Casado, N. McKeown, and G. Parulkar, “Flowvisor: A network virtualization layer,” OpenFlow Switch Consortium, Tech. Rep, vol. 1, p. 132, 2009.
  • [323] Y. Kanaumi, S. Saito, and E. Kawai, “Deployment of a programmable network for a nation wide r d network,” in 2010 IEEE/IFIP Network Operations and Management Symposium Workshops, 2010, pp. 233–238.
  • [324] ——, “Toward large-scale programmable networks: Lessons learned through the operation and management of a wide-area openflow-based network,” in 2010 International Conference on Network and Service Management, 2010, pp. 330–333.
  • [325] Y. Teranishi, Y. Saito, S. Murono, and N. Nishinaga, “Jose: An open testbed for field trials of large-scale iot services,” Journal of the National Institute of Information and Communications Technology, vol. 62, no. 2, 2015.
  • [326] D. Adami, B. Martini, M. Gharbaoui, P. Castoldi, G. Antichi, and S. Giordano, “Effective resource control strategies using openflow in cloud data center,” in 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013), 2013, pp. 568–574.
  • [327] M. Pizzonia and M. Rimondini, “Netkit: Easy emulation of complex networks on inexpensive hardware,” in Proceedings of the 4th International Conference on Testbeds and Research Infrastructures for the Development of Networks & Communities, ser. TridentCom ’08.   Brussels, BEL: ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 2008.
  • [328] F. Tomonori, “Introduction to ryu sdn framework,” Open Networking Summit, pp. 1–14, 2013.
  • [329] Y. Guo, Z. Wang, Z. Liu, X. Yin, X. Shi, J. Wu, Y. Xu, and H. J. Chao, “Sote: Traffic engineering in hybrid software defined networks,” Computer Networks, vol. 154, pp. 60 – 72, 2019.
  • [330] J. Moy, “Ospf version 2,” Internet Requests for Comments, RFC Editor, STD 54, April 1998. [Online]. Available: http://www.rfc-editor.org/rfc/rfc2328.txt
  • [331] H. Nguyen, A. Lee, and I. Ra, “Network-assisted http adaptive streaming for hybrid software defined network,” in Proceedings of the 2018 Conference on Research in Adaptive and Convergent Systems, ser. RACS ’18.   Association for Computing Machinery, 2018, p. 100–105.
  • [332] C. Jin, C. Lumezanu, Q. Xu, Z.-L. Zhang, and G. Jiang, “Telekinesis: Controlling legacy switch routing with openflow in hybrid networks,” in Proceedings of the 1st ACM SIGCOMM Symposium on Software Defined Networking Research, ser. SOSR ’15, 2015.
  • [333] R. Hand and E. Keller, “Closedflow: Openflow-like control over proprietary devices,” in Proceedings of the Third Workshop on Hot Topics in Software Defined Networking, ser. HotSDN ’14.   New York, NY, USA: Association for Computing Machinery, 2014, p. 7–12.
  • [334] P.-W. Tsai, F. Piccialli, C.-W. Tsai, M.-Y. Luo, and C.-S. Yang, “Control frameworks in network emulation testbeds: A survey,” Journal of Computational Science, vol. 22, pp. 148 – 161, 2017.
  • [335] I. W. Paper, “Flexible network-based, enterprise-class software-defined solutions for hybrid public/private network connectivity,” [online] Available: https://www.business.att.com/content/dam/attbusiness/reports/att-sd-wan-solutions-white-paper.pdf, 2018.
  • [336] C. W. P. W. Paper, “2019 global networking trends report,” [online] Available: https://www.cisco.com/c/dam/en/us/solutions/collateral/enterprise-networks/digital-network-architecture/global-nw-trends-survey.pdf, 2019.
  • [337] D. Zheng, “Huawei enterprise business: Four-dimensional sdn deployment,” Huawei Techn. Co. Ltd., Tech. Rep., April, 2013.
  • [338] J. Roper, “Software defined networking: Should do now? should do never? simply don’t know!” Entuity, Netw. Manag. Company, Apple Valley, MN, USA, Rep. Accessed: Apr, vol. 19, 2018.
  • [339] A. T. W. Paper, “A migration path to software-defined networking (sdn) in an enterprise network,” [online] Available: https://www.alliedtelesis.com/sites/default/files/-migrating_sdn_in_ent_network_reva.pdf, April 2016.
  • [340] N. W. Paper, “Sdn component stack and hybrid introduction models,” [online] Available: https://www.necam.com/docs/?id=c2e5a040-cdf1-4fd7-b63e-6eea4b1f7a7b, 2014.
  • [341] H. W. Paper, “Hp sdn hybrid network architecture: Scalable low-risk network deployments using hybrid sdn,” [online] Available: http://arubanetworks.com/aruba/attachments/aruba/SDN/43/1/4AA5-6738ENW.PDF, 2015.
  • [342] V. W. Paper, “Sdn-nfv reference architecture version 1.0,” [online] Available: http://innovation.verizon.com/content/dam/vic/PDF/Verizon_SDN-NFV_Reference_Architecture.pdf., February 2016.
  • [343] J. W. Paper, “Integrating sdn into the data center,” [online] Available: https://www.juniper.net/assets/us/en/local/pdf/whitepapers/2000542-en.pdf, 2015.
  • [344] I. W. Paper, “Software defined networking in the new business frontier,” [online] Available: https://www.ibm.com/downloads/cas/RWJLRAWE, 2015.
  • [345] ——, “Software defined networking and software-based services with intel processors,” [online] Available: https://www.intel.com/content/dam/doc/white-paper/communications-ia-software-defined-networking-paper.pdf, 2012.
  • [346] W. R. W. Paper, “Practical implementation of sdn and nfv in the wan,” [online] Available: https://www.windriver.com/whitepapers/networking/practical-implementation-of-sdn-nfv-in-the-wan/, 2015.
  • [347] G. W. Paper, “The road to sdn is paved with visibility and many good intentions,” [online] Available: https://www.gigamon.com/content/dam/resource-library/english/white-paper/wp-the-road-to-sdn-is-paved-with-visibility-and-many-good-intentions.pdf, 2015.
  • [348] C. W. Paper, “Software-defined networking and network programmability: Use cases for defense and intelligence communities,” [online] Available: https://www.cisco.com/c/dam/en_us/solutions/industries/docs/gov/software_defined_networking.pdf, 2014.
  • [349] S. W. Paper, “Sdn advantages for ethernet-based control,” [online] Available: https://cdn.selinc.com/assets/Literature/Publications/White20Papers/0030_SDNAdvantages_SW_20190627.pdf?v=20190627-200056, 2019.
  • [350] L. W. Paper, “Lumina extension adaptation platform,” [online] Available: https://cdn2.hubspot.net/hubfs/6333906/lumina-leap-data-sheet.pdf, 2019.
  • [351] O. N. F. W. Paper, “Framework for sdn: Scope and requirements,” [online] Available: https://www.opennetworking.org/wp-content/uploads/2014/10/Framework_for_SDN_-Scope_and_Requirements.pdf, April 2016.
  • [352] M. Boucadair and C. Jacquenet, “Software-defined networking: A perspective from within a service provider environment.” RFC, vol. 7149, pp. 1–20, March 2014.
  • [353] M. M. Tajiki, B. Akbari, N. Mokari, and L. Chiaraviglio, “Sdn-based resource allocation in mpls networks: A hybrid approach,” Concurrency and Computation: Practice and Experience, vol. 31, no. 8, p. e4728, 2019.
  • [354] S. Toufga, S. Abdellatif, H. T. Assouane, P. Owezarski, and T. Villemur, “Towards dynamic controller placement in software defined vehicular networks,” Sensors, vol. 20, no. 6, p. 1701, 2020.
  • [355] M. Alharthi, A. M. Taha, and H. S. Hassanein, “Dynamic controller placement in software defined drone networks,” in 2019 IEEE Global Communications Conference (GLOBECOM), 2019, pp. 1–6.
  • [356] S. Wu, X. Chen, L. Yang, C. Fan, and Y. Zhao, “Dynamic and static controller placement in software-defined satellite networking,” Acta Astronautica, vol. 152, pp. 49–58, 2018.
  • [357] ETSI MEC, “MEC Proofs of Concept,” https://www.etsi.org/technologies/multi-access-edge-computing/mec-poc, 2016, Last accessed: 2020-10-17.
  • [358] Z. Latif, K. Sharif, F. Li, M. M. Karim, S. Biswas, and Y. Wang, “A comprehensive survey of interface protocols for software defined networks,” Journal of Network and Computer Applications, vol. 156, p. 102563, 2020.
  • [359] D. King, C. Rotsos, I. Busi, F. Zhang, and N. Georgalas, “Transport northbound interface: The need for specification and standards coordination,” in 2017 International Conference on Optical Network Design and Modeling (ONDM), 2017.
  • [360] J. Moeyersons, P.-J. Maenhaut, F. Turck, and B. Volckaert, “Pluggable sdn framework for managing heterogeneous sdn networks,” International Journal of Network Management, vol. 30, no. 2, p. e2087, 2020.