Integrated NFV/SDN Architectures: A Systematic Literature Review

1. Introduction

Proprietary network hardware equipment is everywhere in businesses, homes, and data center networks. Each vendor explores and exploits the maximum capability of its platforms as a way to meet the performance, reliability, and availability requirements demanded by the various types of users. However, such an approach has resulted in incompatibility between different manufacturer’s technologies. Restricted licensing agreements and proprietary source code have further contributed to this limitation.

Because of such incompatibility of platforms, as well as the need for network engineers to add new features into their networks (e.g., firewalls, load balancing), they often need to purchase new equipment from different vendors. Each item of equipment is responsible for a share of the traffic processing that requires specific management strategies. Difficulties in the management and configuration of such heterogeneous environments are the norm. The requirement to allow such flexibility in configuration and the deployment of new network functions must be met in new business and engineering models. The complexities of current networking environments result in high operational (OPEX) and capital (CAPEX) expenditure costs (ETSI, 2012).

Network Functions Virtualization (NFV) aims at solving these problems by transferring networking functions from vendor-specific and proprietary hardware appliances to software hosted on Common-Off-The-Shelf (COTS) systems (a.k.a commodity platforms), i.e., with standard processing, memory, and storage components. These systems usually provide the network services in virtual machines (e.g., Virtual Network Functions - VNFs), each one performing different operations (e.g., firewall, packet inspection, routing, etc.) (ETSI, 2015). NFV has the potential to allow cost reduction and the increase in speed of network expansion. Also, NFV has the potential to increase network flexibility for fast service delivery, an option difficult to achieve with traditional methods (Szabó et al., 2015).

Software-Defined Networking (SDN) is a new network paradigm. Its main feature is the separation of the network control plane from the data plane, compared to current networks where the IP layer integrates both planes vertically into the network devices (McKeown et al., 2008). In the SDN control plane, represented by a software called SDN Controller, which is responsible for decisions on how to handle the underlying network traffic concerning network policies and rules. The SDN Controller can run on COTS systems, separated from the forwarding devices. The data plane, deployed as network devices, is responsible for forwarding data according to a set of rules. The SDN controller allows the creation and management of such rules through an Application Programming Interface (API) in the Northbound interface. It does have direct control over the data plane elements through protocols in the Southbound interface. Such a separation provides some definite advantages, such as simplification and flexibility in network policy enforcement, facilitating network configuration, development, and fostering innovation (Kreutz et al., 2015). It also brings research and development challenges that have attracted researchers from both industry and academia.

According to Mijumbi et al. (2016) (Mijumbi et al., 2016), “NFV and SDN have a lot in common since they both advocate for a passage toward open software and network hardware”. Even with different purposes, NFV and SDN do indeed represent complementary paradigms and technologies capable of providing one consolidated solution. To this end, SDN can provide connectivity between VNFs in a flexible and automated way, thus simplifying network management. On the other hand, NFV can make use of SDN as part of a Service Function Chaining (SFC). In this case, both SDN Controllers and Management Applications can run as VNFs in a scalable environment and hence benefit from essential features, such as availability, reliability, and elasticity.

Some studies are tackling the integration of NFV and SDN in different environments (e.g., Cloud Computing, Wide Area Network, Customer Premise Equipment, 5G, etc.). Industrial and academic research studies address several challenges, such as reliability, overall performance, and scalability. Those studies use distinct architectural design rationale and functional and non-functional requirements. Although NFV/SDN architectures have clear potential benefits, they are still at an early stage of development. There are several open research questions to be answered and development issues to be addressed (Kreutz et al., 2015; Mijumbi et al., 2016; Batista et al., 2015).

It is worth emphasizing that there have been some initial efforts to review the body of knowledge on NFV and SDN, but most efforts treat them in isolation. Mijumbi et al. (2016) (Mijumbi et al., 2016) and Gil and Botero (2016) (d. J. Gil Herrera and Vega, 2016) surveyed the state-of-the-art in NFV, whereas Kreutz et al. (2015) (Kreutz et al., 2015) presented a survey exclusively on SDN. Furthermore, both Li and Chen (2015) (Li and Chen, 2015) and López et al. (2015) (López et al., 2015) propose studies to integrate both technologies. However, there is still a need for a detailed vision of the different integrated NFV/SDN architectures (e.g., target environment, problems to solve, and architectural designs) as well as trends for research and development. We argue that the research community would benefit from an in-depth view of the NFV/SDN architectural designs so that researchers can have a clear picture of the past relevant studies as well as the current challenges. Therefore, our study fills an important gap by providing an in-depth view of the SDN/NFV architectures, as well as highlighting the challenges to further advances in this topic.

In this work, we cover the state-of-the-art of integrated NFV/SDN architectures. We aim at: i) investigating the characteristics (target environment and problems to solve) of integrated NFV/SDN solutions and current practices; ii) comparing their architectural designs (i.e., NFV framework design and tools, SDN APIs, and placement of SDN elements); and iii) identifying the challenges and the possibilities for improving them. To this end, we conducted a Systematic Literature Review (SLR) to provide an overview of this research area, based on a well-known methodological framework introduced by Kitchenham et al. (2009) (Kitchenham et al., 2009).

SLR is an evidence-based approach used to identify, evaluate, and interpret all available evidence about a focused topic, in a repeatable and impartial manner (Kitchenham, 2004). For this, the SLR framework follows a predefined protocol with a set of steps to perform sources and studies selection and data extraction. In the end, results are synthesized from this well-defined approach by comparing the individual studies and providing consistent evidence of the research questions being posed.

Our original contributions are three-fold. First, this SLR provides an in-depth understanding of both state-of-the-art and state-of-the-practice of NFV/SDN architectural solutions, highlighting their main characteristics (e.g., potential deployment scenarios and problems raised) and their underlying architectural designs. Second, the SLR study identifies trends for future research and development as well as open research issues and challenges. Last, but not least, our SLR provides the necessary details for replicating it or broadening its scope in the future.

The remainder of the article is organized as follows. Section 2 describes NFV and SDN technologies. Section 3 introduces the details of the adopted SLR. It explicitly defines the steps of the protocol and the strategies to retrieve the evidence, to allow this SLR to be reproduced and criticized by other professionals. Section 4 describes the search process that resulted from the SLR execution. Section 5 describes the target environments and problems addressed by the studies. Section 6 describes a taxonomy to organize the various decision-making levels for the design of NFV/SDN architectures. Such a taxonomy was derived by analyzing implementations found in the researched literature and reference architectures proposed by vendors and standardization bodies. Sections 7 and 8 provide the technical details of this taxonomy. Section 9 presents the mainly auxiliary tools used in the studies to implement NFV/SDN architectures. Section 10 presents a brief description of the results obtained. Section 11 lists the challenges involved in developing NFV/SDN solutions. Section 12 describes some threats to the validity of this study, to evaluate the quality of this research. Finally, Section 13 concludes the article.

2. Background

2.1. Network Functions Virtualization (NFV)

NFV is transforming the computer and communication networks industry. NFV allows customers to transfer the networking functions from vendor-specific and proprietary hardware appliances to software hosted on COTS platforms (ETSI, 2012).

NFV provides the network services in virtual machines (VMs) working in Cloud infrastructures, where each VM performs different network operations (e.g., firewall, intrusion detection, Deep Packet Inspection, load balancing, etc.) (ETSI, 2015). Some benefits of deploying network services as virtual functions are (ETSI, 2012):

Flexibility in the allocation of network functions in general-purpose hardware;
Rapid implementation and deployment of new network services;
Support of multiple versions of service and multi-tenancy scenarios;
Reduction in CAPEX costs by managing energy usage efficiently;
Automation of the operational processes, thus improving efficiency and reducing OPEX costs.

From 2012, the European Telecommunications Standards Institute (ETSI) has led the standardization process for NFV technology through the NFV Industry Specification Group (NFV ISG). The NFV ISG has already published tens of specifications documents, such as requirements, use cases, terminologies, proofs of concept, and the like (ETSI, 2014). These specifications allow researchers and engineers to have a clear picture of the elements of a particular NFV infrastructure.

Figure 1 illustrates the high-level architecture for NFV, which comprises of three main functional blocks, as detailed below.

Virtual Network Functions (VNFs)::: VNF is the virtualization of a certain network function, which should operate independently of the others. It may run on one or more virtual machines. A particular VNF can also be divided into several sub-functions called VNF Components (VNFCs). Elemental Management Systems (EMSs) can be used for VNF monitoring;
NFV Infrastructure (NFVI)::: NFVI comprises of all hardware and software required to deploy, operate, and monitor VNFs. To this end, NFVI has a virtualization layer necessary for abstracting the hardware resources (processing, storage, and network connectivity). It ensures the independence of the VNF software from the physical resources. The virtualization layer is usually composed of the server (e.g., Xen, KVM, VMware, etc.) and the network (e.g., VXLANs, NVGRE, OpenFlow, etc.) hypervisors. The NFVI Point of Presence (NFVI-PoP) defines a location for Network Function deployments as one or many VNFs.
NFV Management and Orchestration (MANO)::: MANO comprises three components: i) The Virtualized Infrastructure Manager (VIM), which manages and controls the interaction of VNFs with physical resources under its control (e.g., allocation, deallocation, and inventory); ii) the VNF Manager (VNFM), which is responsible for managing the VNF life-cycle (e.g., initialization, suspension, and termination); and iii) the NFV Orchestrator (NFVO), which is responsible for realizing network services on NFVI. It also performs monitoring operations of the NFVI as a way to collect information for operations and performance management.

Figure 1. NFV Architecture (ETSI, 2014).

Another component to be considered as part of the NFV framework is the Operations Support Systems and Business Support Systems (OSS/BSS). This element comprises the legacy management systems and assists MANO in the execution of network policies, either automatically or manually.

2.2. Software-Defined Networking (SDN)

SDN is a new network paradigm that was designed to overcome the difficulty in developing and testing new solutions and protocols in production environments, where the underlying code running in business switches and routers are proprietary and closed (McKeown et al., 2008).

According to Kreutz et al. (2015), currently, both control and data planes are integrated into most commercial networking devices, which makes IP networks difficult to manage. Due to this, operators need to configure network policies into each device individually, often using low-level commands that are specific to the manufacturer. Further, automatic reconfiguration mechanisms, necessary for network adaptation during failures and load changes, are non-existent in today’s networks. Such issues reduce the flexibility for deploying new network services and management strategies, as well as hindering development and innovation.

The main feature of the SDN paradigm is the separation of the control and data planes. It has clear advantages where network programmability is achieved through the centralization of the control plane in conjunction with the availability of open APIs, thus making easier the process of creating and deploying new network applications. SDN provides simplification and flexibility in network policy enforcement, facilitating network configuration and management (Kreutz et al., 2015).

The control plane, represented by a software called the SDN Controller, is responsible for decisions on how to handle network traffic, assuming the role of the “brain” of the network. The SDN Controller can run on COTS platforms, separated from the network equipment. The data plane, represented by the network devices, is responsible for forwarding traffic according to a set of rules (Kreutz et al., 2015). Such rules are created at and managed by the SDN Controller. The SDN controller has a global view of the network topology and has direct control over the data plane elements through a southbound protocol, such as OpenFlow (ONF, 2015) (detailed in Section 9.5.1).

Figure 2 shows the given three layers of an SDN architecture and the APIs responsible for the interaction between them. The SDN Northbound API is responsible for providing support for communication between the application layer and the control plane layer. It also includes support for SDN Applications, such as traffic engineering, routing, firewall, quality of service, etc. The Southbound API is responsible for the communication between the SDN Controllers and switches.

Figure 2. SDN Reference Architecture (McKeown et al., 2008; Kreutz et al., 2015; ONF, 2015).

3. Systematic Literature Review Planning

This section presents the adopted plan to perform the SLR. This phase aims to define the way the review is executed, including the research questions and the procedure for sources and studies selection.

3.1. Research Questions

To identify the state-of-the-art of integrated NFV/SDN solutions and open issues, we initially focus our study on the following research questions.

Q1) In which environments are the integrated NFV/SDN solutions applied?

This question aims at mapping the actual environments (e.g., Enterprise Networks, WANs, CPEs, Data Centers, Wireless Networks, etc.) in which the proposed integrated NFV/SDN solutions have been tested and deployed.
Q2) What are the problems that such integrated NFV/SDN solutions are trying to solve?

This question aims at identifying the issues (e.g., middlebox and network virtualization, vCPE, reliability, scalability, dynamic service function chaining, performance, etc.) that NFV/SDN solutions are trying to tackle. We also aim to classify them according to their respective target environments.
Q3) What are the differences among the design architectures of integrated NFV/SDN solutions?

This question seeks clarification on how the studies have been proposing design architectures of integrated NFV/SDN solutions. Answering this question requires a classification of the proposed architectures using a subset of their characteristics extracted from the ETSI documentation (ETSI, 2015), published in December 2015. These characteristics include NFV framework design and tools, SDN Northbound and Southbound APIs, the placement of the SDN elements in the NFV Framework, the use of multiple SDN controllers, and the like. We aim at identifying the differences among such designs based on the types of target environments and general objectives, including their main advantages and disadvantages.

Group 1 Group 2 Term 1 Software-Defined Networking Network Function Virtualisation Term 2 Software Defined Networking Network Function Virtualization Term 3 Software Defined Network NFV Term 4 Software-Defined Network Term 5 SDN

Table 1. List of Terms and Synonyms.

3.2. Sources Selection

To find the relevant evidence to answer the research questions, a set of sources must be selected to perform the search of primary studies. We now describe the criteria used to select such sources, the search strings, and the sources identification.

For the selection criteria of sources, we considered the availability of articles on the Web and the existence of advanced search mechanisms using keyword and filters based on content type (conference publications, journals, and magazines, etc.) and year of publication. We considered only studies in the English language. Therefore, we selected the following web search engines: ACM Digital Library, Engineering Village, IEEE Xplore, Science Direct, Scopus, Springer Links, and Web of Science.

To compose our search string, we considered the keywords listed in Table 1, where each group represents a keyword with its synonyms. The general form of the search string is shown as follows:

Search String::: (([G1,T1] OR [G1,T2] OR [G1,T3] OR [G1,T4] OR [G1,T5]) AND ([G2,T1] OR [G2,T2] OR [G2,T3]))

The literature searches were performed manually using all the selected web search engines. Table 2 shows the composition of the search string per web search engine. Regarding ACM Digital Library’s search queries, the symbol “+” replaces the logical operator “AND” while the space symbol replaces the logical operator “OR”. In Web of Science’s search queries, the symbol “TS” determines that search is limited to the following fields within a record: Title, Abstract, Author Keyword, Keywords Plus.

Web Search Engine Search String ACM Digital Library +(“Software-Defined Networking” “Software Defined Networking” “Software Defined Network” “Software-Defined Network” “SDN”) +(“Network Function Virtualisation” “Network Function Virtualization” “NFV”)
Engineering Village, IEEE Xplore, Science Direct, Scopus, and Springer Links (“Software-Defined Networking” OR “Software Defined Networking” OR “Software Defined Network” OR “Software-Defined Network” OR “SDN”) AND (“Network Function Virtualisation” OR “Network Function Virtualization” OR “NFV”)
Web of Science TS=((“Software-Defined Networking” OR “Software Defined Networking” OR “Software Defined Network” OR “Software-Defined Network” OR “SDN”) AND (“Network Function Virtualisation” OR “Network Function Virtualization” OR “NFV”))

Table 2. List of Search Strings per Web Search Engine.

3.3. Procedure for Studies Selection

A priori, all studies in the English language obtained from web search engines were selected as primary studies. These primary studies then went through a studies selection and evaluation process, based on three stages. One researcher (M. Bonfim) was assigned to evaluate the selected studies. An article is included for further processing in the next steps when it is approved in the previous one. Otherwise, the article is discarded.

Below, we describe the three stages of the studies selection and evaluation process:

Stage 1: Eliminate studies selected as primary studies based on exclusion criteria. An article will be only included in the following stages if it proposes an integrated NFV/SDN solution. This stage considers only information provided in abstract and conclusion.
Stage 2: Eliminate studies selected in Stage 1 based on exclusion criteria. An article will be only included for the following stages if it proposes an integrated NFV/SDN solution and describes its architecture design. This stage evaluates all content of the articles.
Stage 3: In this stage, studies selected at Stage 2 pass for a quality screening. An article will be excluded if it does not meet the following quality criteria:
- QC1: Is there a clear statement of the goals (i.e., target environments and problems to solve) of the research?
- QC2: Is the architecture design well detailed? In other words, is it possible identify the used tools, the place of SDN elements and NFV framework design?
- QC3: Are the experiments realized to evaluate the ideas presented in the study?

Each criterion has three possible responses: Yes, Partly, or No. “Yes” responses count for 1 (one) point, “Partly” count for 0.5 points and “No” count for 0 (zero) points. To be accepted, a paper must obtain a score of greater or equal to 2 (two) as described in equation 1:

(1)

Q C 1 + Q C 2 + Q C 3 \geq 2.0

Finally, at the end of execution, we included some reports from Proof of Concepts (PoCs) registered in ETSI NFV ISG PoC Projects ¹¹1http://www.etsi.org/technologies-clusters/technologies/nfv/nfv-poc, regarding NFV/SDN solutions provided by different vendors and carrier networks.

4. Search Results

The initial search was performed in April 2016. Initially, a total of 1644 articles were identified (Identity Phase) as primary studies. In the Identity Phase, 907 duplicate findings were removed from the result set. It is important to emphasize that we do not delimit a specific range of years for the searching process. Then we started the execution of the three stages of selection, as described in Subsection 3.3.

In Stage 1 (Screening Phase), having reviewed all abstracts and conclusions, we considered only articles that proposed an integrated NFV/SDN solution. In this case, we selected 138 studies and discarded 769. In Stage 2 (Screening Phase), we considered only records included in Stage 1. After evaluating all the content of articles, we considered only those that described the design of the integrated NFV/SDN solution. In this case, we selected 88 studies and discarded 50. In Stage 3 (Eligibility Phase), studies selected in Stage 2 were passed for a quality screening, described in Subsection 3.3. In this stage, we discarded 40 studies that did not meet the quality criteria, with the result that 48 studies were included (Included Phase) for data extraction.

At the end of the execution process, twelve (12) Proof of Concepts (PoCs) reports (registered in ETSI NFV ISG PoC Projects ²²2http://www.etsi.org/technologies-clusters/technologies/nfv/nfv-poc) are included, generating a total of 60 studies for data extraction. The PoCs are part of the Hot Topic 01³³3http://nfvwiki.etsi.org/index.php?title=HT01_-_Use_of_SDN_in_an_NFV_architectural_framework - “Use of SDN in an NFV architectural framework”, which includes SDN/NFV solutions provided by different vendors and carrier networks. Table 3 lists these PoCs.

Identifier Title POC#1 CloudNFV - Open NFV Framework Project. (Telefonica et al., 2014) POC#2 Service Chaining for NW function selection in Carrier Networks. (NTT et al., 2014) POC#8 Automated Network Orchestration. (Telekom et al., 2014) POC#13 Multi-Layered Traffic Steering for Gi-LAN. (Telefonica et al., 2016) POC#16 NFVIaaS with Secure SDN-controlled WAN Gateway. (AT&T et al., 2015) POC#21 Network intensive and compute intensive hardware acceleration. (BT et al., 2015) POC#23 E2E orchestration of Virtualised LTE Core-Network functions. (Telecom et al., 2014) POC#26 Virtual EPC with SDN functions in Mobile Backhaul Networks. (Italia et al., 2015) POC#27 VoLTE Service based on vEPC and vIMS architecture. (Unicom et al., 2015) POC#28 SDN Controlled VNF Forwarding graph. (DT et al., 2015) POC#34 SDN-enabled Virtual EPC Gateway. (Telenor et al., 2016) POC#38 Full layer-7 stack fulfillment, activation, and orchestration of VNFs in carrier networks. (Telstra et al., 2016)

Table 3. List of Proof of Concepts (PoCs) included in the execution process.

The following step is the data collection for each selected work in the Included Phase. One (1) researcher (M. Bonfim) performed the data extraction, extracting the following properties from each study: (i) author’s identification; (ii) article type (conference paper, journal article, report, etc.); (iii) publication description (title, ISSN, date, DOI, etc.); (iv) work’s title and abstract; (v) environments in which the NFV/SDN solution is applied; (vi) problems that NFV/SDN solution try to solve; (vii) technical aspects related to NFV/SDN solution proposed, and (viii) quality criteria evaluation.

After performing the SLR, during the article’s revision, another 14 relevant references were found and/or recommended by experts in the field to complement the SLR findings. The total number of research papers is now 74, including new references from 2016 until 2017. These papers respected the same Exclusion (Stage 2), and Quality Criteria (Stage 3) adopted previously. All results and discussions presented in this article were derived from these 74 studies.

There is a growing interest from both the academia and the industry in integrating NFV and SDN technologies. The number of studies increased from 1 research paper in 2013 to 43 in 2015, and this number has been rising since then. The researchers prefer scientific conferences to publish their studies (35 articles), followed by journals (26 articles) and PoC reports (12 articles).

The reader can find more details regarding the data collection (extracting documents) at the GitHub link⁴⁴4Link for data collection archives: https://github.com/michelsb/SLRNFVSDNFiles.git.

5. Application for Nfv/sdn Architectures

This section aims at answering the first and second research questions, thus relating the different environments where the NFV/SDN architectures found are applied, in addition to identifying the main problems those architectures are trying to solve.

First, it is necessary to present the definitions for the following terms, which will eventually appear in the studies.

Reliability::: Relates to fault tolerance, disaster recovery, and full isolation;
Elasticity in Network Function::: It is the possibility to scale network services dynamically at runtime in an automated fashion (Szabó et al., 2015);
Flexibility::: Service providers can design the layout of service chains without considering the physical network (Ding et al., 2015);
Security::: It means privacy, authentication, or authorization;
Scalability::: Service providers can augment the number of service chains without worrying about constraining flow tables (Ding et al., 2015);
Dynamic Service Chaining::: Applying different policy-based traffic steering to flows in a certain SFC;
QoS Management::: Trying to optimize the use of network capacity through QoS techniques.

5.1. Unifying Computer and Network Resources

This concept aims at creating an abstraction layer (AL) on both the computational and network resources as a way to provide a unique and centralized view of the whole environment. The NFV/SDN solutions initially focused on creating this Abstraction Layer (AL) to delivery intelligent network services - regarding performance and reliability - for different customer profiles such as end, retail, and enterprise users, as well as Over The Top (OTT) providers, and developers (ETSI, 2015).

The AL is responsible for two orchestration functions (see Figure 3), namely Resource Orchestration (RO) and Network Service Orchestration (NSO) (ETSI, 2014). RO utilizes NFV VIM component and SDN to perform a global resource management with resource virtualization provisioning and managing. On the other hand, the NFVO uses the NSO functions to implement the life-cycle management of Network Services (VNFFGs), coordinating groups of VNF instances as network services. These functions use the services exposed by the VNFM and by the RO allowing joint instantiation and configuration along with connectivity and dynamic changes management.

In this context, different architectures try to achieve this unification, such as UNIFY (Császár et al., 2013; Sköldström et al., 2014) and T-NOVA (T-NOVA Project, 2015), both EU-funded 7^th Framework Programme (FP7) projects.

Figure 3. Abstraction layer for unifying computer and network resources.

T-NOVA aims at implementing an SDN-based MANO framework to manage a federated network and cloud resources. Its primary objective is to deliver third-party network functions (NFs) to operator’s customers in an automated and optimized manner, introducing a “Network Function Store”. For NF management, an Orchestrator platform was developed on top of common open source components such as OpenStack (Rackspace Cloud Computing, 2016) and OpenDaylight (Foundation, 2016). Besides, the WICM (WAN Infrastructure and Connectivity Manager) provides network connectivity between NFVI-PoPs (Points of Presence) and manages traffic steering in virtual networks.

On the other hand, UNIFY seeks to unite computer and network resources in a common management framework. The UNIFY NFV/SDN architecture aims at creating and managing the dynamic end-to-end network services from the home and enterprise networks to the operator’s data center. It provides a MANO framework that integrates both Cloud and WAN domains and includes three layers, namely the Service Layer (SL), the Orchestration Layer (OL), and the Infrastructure Layer (IL). Figure 4 shows our simplified view of the UNIFY architecture, highlighting the main functional components of ETSI’s NFV (left box) and ONF’s SDN (right box) reference models.

The SL comprises business management concerned with service life-cycle, providing Operation Support System (OSS) and Business Support System (BSS) functions related to services from different tenants (e.g., enterprise users, service providers, etc.). The SL also executes the NFVO and VNFM functions for the NS and VNF life-cycle management. It is noteworthy that the SL management functions are infrastructure-agnostic, dealing only with the offered services (Sonkoly et al., 2015).

The OL acts as a VIM component - providing resource orchestration (RO module) to deliver virtual resources views to SL - and policy enforcement. It also includes the Controller Adapter (CA), and a multi-domain, multi-technology, and multi-vendor controller. The CA provides computing and networking abstraction by collecting virtualized resources from lower layer domain-specific controllers and organizing them into a global virtualized resource view. The CA offers an independent technology control to the RO module. Finally, in the OL, SDN is used for the creation and integration of virtual networks in both domains.

The IL manages all IT and network resources (physical and virtual) needed for the VNF execution. For this, it uses two types of domain-specific controllers, the Compute Controller to manage computational resources, and the Network Controller to manage network resources. The IL considers different kinds of resources such as SDN enabled network nodes (e.g., OpenFlow switches), and cloud-enabled data centers (e.g., OpenStack, CloudStack, etc.).

The UNIFY architecture has been used as a basis for the implementation of several NFV/SDN solutions to different problems, such as Middleboxes Virtualization and virtualized Customers Premises Equipment (vCPE) (see Sections 5.3 and 5.4).

5.2. On-demand and Application-specific Traffic Steering

Traffic Steering is the ability to direct users’ requests to the appropriate service/content sources. Traffic steering might be based on many factors such as the available networking resources and capabilities on the client and server side, user’s permissions and location, and the like. For example, a certain user may request a video streaming service that has stringent application performance requirements. In this case, on-demand and application-specific traffic steering could guarantee efficient network resource usage and better Quality of Experience (QoE) for the user.

In the NFV Framework context, SDN can enhance traffic steering between VNFs, providing dynamic service chaining. With the separation of control and data planes, SDN enables the exchange of information between the application and network layers, allowing users’ services to have an overview of the general state of the network, and to make intelligent decisions (to meet service requirements) on how to steer traffic through VNFs better.

Figure 5. NFV/SDN architecture for application-specific traffic steering.

Carella et al. (2015) (Carella et al., 2015) proposed an NFV/SDN architecture to provide a cross-layer interface between the application and network layers, to allow the deployment of network services with on-demand and application-specific traffic steering. Figure 5 shows our detailed view of this architecture. It comprises three layers: Application, Control, and Infrastructure. The Application Layer consists of network services. These services must interface with the Control Plane API to communicate their network requirements (e.g., bandwidth, maximum latency). The Control Plane has a global view of computer and network resources and provides the traffic steering capabilities to the Application Layer. Its main component is the Cross-Layer Orchestrator (CLO) that acts as an NFVO and VNFM to manage the lifecycle of services over the cloud (OpenStack-based) and WAN domains (OpenFlow-based). The CLO was implemented over the OpenSDNCore Orchestrator (Fraunhofer FOKUS, 2016), using Java programming language.

5.3. Middleboxes Virtualization

According to (Sherry et al., 2012), middleboxes have been increasingly used in enterprise networks (45% of network devices). They are deployed to increase performance (e.g., traffic shaping, load balancing, and TCP optimization) and to provide security functionalities (e.g., firewalls, Intrusion Detection and Prevention systems - IDPS, and Deep Packet Inspection - DPI) for both incoming and outcoming traffic .

However, hardware-based middleboxes have the following disadvantages (Cziva et al., 2015). First, they incur high operational costs (OPEX) due to the management complexity. These middleboxes come from different manufacturers and must be deployed, configured, and managed individually. Second, hardware-based middleboxes incur capital costs (CAPEX). When new network functions are necessary, enterprises must purchase one or more middleboxes due to the inflexibility in proprietary hardware that creates vendor lock-in and limits innovation.

NFV/SDN architectures can be used to deal with these challenges. The primary objectives are to reduce both CAPEX and OPEX and to provide fast delivery of network function, elasticity, and dynamic service chaining. In these works, NFV manages virtual middleboxes, while SDN provides interconnection between VNFs to delivery Service Function Chaining (network services).

In this context, the authors in (Sherry et al., 2012) consider two approaches (see Figure 8) for redirecting the traffic to virtualized middleboxes for further processing, namely Bounce and IP redirections. In the case of Bounce Redirections, a certain enterprise gateway uses tunneling techniques to redirect both ingress and egress traffic to the virtual middleboxes (grouped into service chains), as shown in Figure (a)a. It requires minimal configuration (i.e., few static rules) at the gateway since it only redirects traffic to a cloud provider hosting the middleboxes. However, this approach might increase the end-to-end delay due to these redirections for each packet. On the other hand, to avoid the extra round-trips of the Bounce Redirection (see Figure (a)a), IP Redirection allows routing traffic directly to/from the cloud provider, as shown in Figure (b)b. In this approach, the cloud will be located in the middle of the communication between enterprise and external sites. However, the provider must announce naming and addressing on the company’s behalf (e.g., DNS redirection).

NFV/SDN architectures (Telecom et al., 2014; Batalle et al., 2013; Schulz-Zander et al., 2015; Lin et al., 2015; Sonkoly et al., 2015; Deng et al., 2015; Cziva et al., 2015, 2015; Rossem et al., 2015; Callegati et al., 2015; Cziva and Pezaros, 2017) have been proposed to deal with Middleboxes Virtualization. Some of them will be presented below.

The works of Cziva et al. (2015) proposed an NFV/SDN framework⁵⁵5https://netlab.dcs.gla.ac.uk/projects/glasgow-network-functions, so-called Glasgow Network Functions (GNF), to deploy and manage container-based network services in public (Cziva et al., 2015) and private (Cziva et al., 2015; Cziva and Pezaros, 2017) Cloud environments. This framework aims at overcoming the limited network reconfigurability in these scenarios, delivering network programmability and fast deployment of new network services. As shown in Figure 9, GNF is composed of 4 planes: the infrastructure encompasses all the physical resources of network and computations, where we have only NFV Centralized Cloud Infrastructures in (Cziva et al., 2015), and incorporated with edge devices (e.g., CPEs, virtual routers, and IoT gateways) in (Cziva and Pezaros, 2017). The VIM and Orchestration planes are responsible for Resource Orchestration. For this, the operator must deploy the GNF Agent on all Cloud servers and all edge devices. This feature has two functions: i) local VNF instantiation by using Docker Engine (Inc., 2016) for fast deployments and low resource utilization, and ii) local traffic steering management by using OpenFlow rules (via OVSDB) and virtual switches. Also, GNF uses the OpenDaylight controller for network connectivity in the NFVI. The GNF Manager is responsible for receiving NFV service requests (Service plane) and performing the necessary operations using the OpenDaylight and GNF Agent instances.

Figure 9. The GNF platform (Cziva and Pezaros, 2017).

In addition, Sonkoly et al. (2015) (Sonkoly et al., 2015) extended the UNIFY architecture (see Section 5.1) to create a service function chaining control plane. This solution aims to support SFC in distributed cloud scenarios, where VNFs from the same SFC can run in different NFVI-PoPs. A prototype framework, called Extensible Service Chain Prototyping Environment (ESCAPE), was implemented in Python on top of a POX Controller. This prototype works with two domains in the Infrastructure Layer (IL): Cloud and OpenFlow. The OpenStack Cloud Platform (Rackspace Cloud Computing, 2016) and the OpenDaylight Controller (Foundation, 2016) perform the management of Cloud domains while VNFs are deployed as KVM virtual machines (running a Click process). The OpenFlow domains handle transport networks with Linux nodes running Open vSwitches (OpenFlow support). The POX Controller (NOXRepo.org, 2016) (network management) and the NETCONF/YANG (VNF management) manage these domains while VNFs are deployed as distinct processes (Linux cgroups) and run network functions implemented in Click Modular Router.

Finally, Deng et al. (2015) (Deng et al., 2015) proposed the VNGuard framework that uses NFV to provide fast and dynamic virtual firewalls in a cloud environment, for the protection of Virtual Networks. Aimed at considering VN’s changeable topology (VMs dispersion and migration), this framework uses SDN to provide fast and flexible traffic steering to virtual firewalls. They used OpenStack as a VIM and ClickOS for VNF development. CloudLab⁶⁶6https://www.cloudlab.us/ is a testbed that provides an Infrastructure as a Service (IaaS) for cloud-based experiments.

5.4. Virtualized Customers Premises Equipment (vCPE)

CPE (Customers Premises Equipment) means any equipment (router, modem, etc.) within the customer domain that receives a communication service. CPEs have been a barrier to the current goals of both telecommunications companies and service providers, due to the high cost of maintenance, management difficulties, and the impossibility of remote upgrades. As an alternative, a solution is the CPE virtualization using an NFV architecture, also known as Virtualized CPE (vCPE) (ETSI, 2013).

According to an IHS Markit Survey⁷⁷7NFV Strategies Service Provider Survey - https://technology.ihs.com/572348/nfv-strategies-service-provider-survey-2016, published in 2016, 100% of consulted service providers said they intend to deploy NFV at some point. 81% expect to roll out this deployment by 2017. Most service providers (more than 80%) have a preference for deploying vCPE.

vCPE is a service in which some or all of the functions associated with CPE are virtualized (ETSI, 2013). One of the main problems related to CPEs virtualization is how to instantiate network services in distributed infrastructures (using multiple NFVI-PoPs) (Cerrato et al., 2015). In this type of scenario (see Figure 10), also called Distributed NFV (Gittik, 2014), the VNFs are placed either in the service provider Cloud platform (Cloud CPE) or the on-premise CPE, depending on where they are most efficient regarding latency, available resources, etc. SDN has been the technology adopted to implement the communication management of different scenarios (e.g., Cloud, CPE, and WAN) to provide Distributed NFV.

Figure 10. NFV/SDN architecture for Virtualized Customers Premises Equipment (vCPE).

Cerrato et al. (2015) (Cerrato et al., 2015) proposed a service-oriented NFV/SDN architecture for Telco networks that delivers generic network services selected by telecom operators (DHCP and NAT) or end users (BitTorrent client). The deployment of these network services can occur in a distributed manner either in the telecom data center or the CPE. Figure 11 shows our simplified view of this architecture.

Figure 11. An NFV/SDN architecture design for vCPE.

This solution is based on the UNIFY architecture (see Section 5.1), including its three layers. The Service Layer Application (SLApp) represents the UNIFY SL enabling different players (operators and end users) to select their network services. For this, the SLApp includes an authentication mechanism and provides a high-level data model for defining flexible network services (including traffic steering primitives), called Service Graph (SG).

Also, the Global Orchestrator (GO) represents the UNIFY Orchestration Layer (OL). The GO manipulates the Forwarding Graph (FG) received from SLApp to enable the network service deployment according to the VNF requirements and infrastructure capabilities. To allow distributed NFV, the GO implements multiple Control Adaptors to coordinate different infrastructures and an Orchestrator component responsible for the centralized coordination of multiple Control Adaptors. Then, the GO selects one of the infrastructures to implement all the network service requested. The authors proposed two different infrastructures (UNIFY Infrastructure Layer - IL) to host network services: the integrated node and the OpenStack node.

The integrated node represents the CPE (home gateway). It receives an FG from the GO through the Node Resource Manager (NRM) via REST API. The NRM will instantiate all VNFs using Docker containers, DPDK process or any hypervisor supported by libvirt. For traffic steering, the NRM uses an extensible Data-Path daemon (xDPd) to create an OpenFlow switch (and its correspondent Controller) for each FG. Separately, the OpenStack node represents the telco data center and uses the OpenStack Cloud Platform for network service deployment. In this case, the KVM hypervisor creates the VNFs, and the OpenDaylight and Open vSwitch control the traffic steering.

The works of Soares (Soares et al., 2014, 2015) proposed the Cloud4NFV platform, an NFV/SDN framework for Telco network virtualization. This platform considers multiple NFVI-PoPs and WAN domains when deploying new Service Function Chaining. Cloud4NFV considers a topology with multiples customer sites (NFVI-PoPs). All NFVI-PoPs include an OpenStack distribution working as a Cloud VIM and an OpenDaylight controller to provide VNF connectivity. The VNFs are CPE functions, and they are deployed as VMs in the NFVI-PoP closest to the customer. Further, Cloud4NFV includes a WAN VIM to provide a view of a unified WAN domain connecting the NFVI-PoPs and Telco Data Center.

5.5. Wireless Networks

Due to the increasing popularity of wireless networks, new requirements have arisen, such as mobility support, programmability, fast delivery of network services, performance, and security (Schulz-Zander et al., 2015). However, the management and configuration of today’s large WiFi networks are complex and inflexible, ignoring the application requirements or user needs. We present the problems addressed by NFV/SDN architectures designed for different wireless networks scenarios.

5.5.1. Wireless LAN

Regarding a WiFi network, the studies related to WLANs leverage the Virtual Access Point (VAP) abstraction by moving the MAC layer or middleboxes processing to the Cloud. When associated with the wireless network, each client acquires a VAP that will be dedicated, independent of the client migrating from one access point to another (handover), as shown in Figure 12. In this example, the Physical AP 1 allocates two VAPs to two clients, Bob and Alice. If Bob moves toward another AP, his VAP will also be migrated and deployed to the new AP.

Figure 12. NFV/SDN architecture for WiFi networks.

Shulz-Zander et al. (2015) (Schulz-Zander et al., 2015, 2017) proposed the OpenSDWN, an NFV/SDN approach to implement per client access points and virtual middleboxes. Figure 12 shows our detailed view of this architecture. For the access point case, the authors created an extension to Odin (Suresh et al., 2012), called Light Virtual Access Point (LVAP). An LVAP uses SDN applications to abstract some functionality of the 802.11 Access Point, such as authentication, handoff, and client associations. A physical AP supports multiple LVAPs, one for each client (which receives a unique BSSID). Therefore, a certain LVAP serves as a dedicated link between its client and infrastructure. The authors also implemented virtual middleboxes (e.g., firewall), which can be deployed either on a Middlebox Server or at the access point itself and can integrate them with LVAPs using virtual networks. A service differentiation mechanism (DPI-based) tries to identify and classify flows to redirect traffic to the correct vMiddleboxes (vMB). OpenSDWN Controller performs all the above functionalities, using the Floodlight (Schulz-Zander et al., 2015) and ONOS (Schulz-Zander et al., 2017) as SDN controllers. Such an abstraction allows the seamless mobility with the migration of both LVAPs and vMBs among APs.

In (Vestin and Kassler, 2015), the authors extended CloudMAC framework (Dely et al., 2012) to provide QoS for VAPs. In the proposed solution, the VAP is responsible for the MAC layer management frames (e.g., beacons, probes request/response) and runs as a VM in a Cloud environment. The physical AP redirects these frames to the destination VAP, using OpenFlow rules. The QoS mechanism implements VAP traffic prioritization using different queue management strategies (e.g., Stochastic Fair Queueing) on all Open vSwitches between the APs and VAPs. The OpenDaylight Controller manages both traffic redirection and prioritization. Seamless handovers can be achieved by just changing the SDN forwarding rules.

5.5.2. Wireless Mesh Networks (WMN)

The only study found for WMN proposes the Urban-X, an NFV/SDN solution for dense urban scenarios (Kim, 2015). In this case, the authors create a multi-radio cognitive mechanism to dynamically self-adapt when there are variations in the interference conditions on the WiFi channels. For this, each mesh node includes an OF Agent component that supports OpenFlow and allows the instantiation of VNFs. The VIM component controls all OF Agents. It uses an OpenFlow Controller to establish a path for end-to-end connectivity, including one or more mesh nodes. For this, VIM uses the OF Agents to monitor link status to configure the path with minimum latency. At the same time, the VIM instantiates TCP accelerators as VNFs at each mesh node included in the path. Thus, the Urban-X improves TCP throughput to the mesh clients.

5.6. Wireless and Mobile Networks

A new generation of mobile network technologies appears every ten years. The first generation (1G) came in the mid-1980s with analog cellular networks. The second generation (2G) began in the mid-1990s and started the era of digital mobile phones encompassing technologies such as CDMA, TDMA, GSM, GPRS, and EDGE. The third-generation (3G) emerged in the late 1990s introducing the use of packet switching rather than circuit switching for data transmission. 3G technologies such as the Universal Mobile Telecommunications System (UMTS) achieves high connection speeds (up to 42 Mbit/s downlink), making it possible to use multimedia applications. The fourth generation (4G) with its Long Term Evolution (LTE) appeared in the mid-2010s with the aim of providing speed improvements up to 10-fold over the existing 3G technologies.

It is worth emphasizing that every new generation tries to address service and network requirements not met by its predecessors. One of the current challenges for mobile networks is how to handle the ever-increasing traffic volume. According to Cisco VNI (Paper, 2017), mobile traffic volume grew 63% in 2016, reaching an average of 7.2 exabytes per month. Such research forecasts an increase in this monthly traffic volume by seven times in the future, reaching the mark of 49 exabytes. To address this growth, mobile operators are investing in infrastructure, thus increasing OPEX/CAPEX costs and management complexity. In this context, 5G networks aim at addressing the following demands (Gupta and Jha, 2015): improved data rate, decreased latency, and increased capacity for consistent QoS/QoE.

Currently, the standardization of 5G networks is still a work-in-progress (itu, 2017). The International Telecommunications Union (ITU⁸⁸8ITU website: http://www.itu.int) will be the specialized agency responsible for publishing the final standard in mid-2020’s, which is also referenced as International Mobile Telecommunications (IMT)-2020. The 3rd Generation Partnership Project (3GPP⁹⁹93GPP website: www.3gpp.org/) is the standard body that unites several mobile industries with the objective of elaborating and submitting a proposed specification to the ITU (mid-2018’s) to be part of the IMT-2020 standard.

Both industrial and academic researchers have also put research efforts on the architectural components of 5G networks. For instance, Verizon created a 5G Tech Forum (5GTF) in September 2015 (5gt, 2017). The 5GTF is a vital initiative where major vendors such as Verizon, Cisco, Ericsson, Nokia, and Apple work together to develop early 5G specifications and then contribute to the 3GPP. 5GTF published its first specification release in July 2016.

Also, the European Union funded 5G Public-Private Partnership (5GPPP). 5GPPP is a joint initiative between the European Commission (EC) and the European ICT industry to develop solutions, architectures, and standards to put Europe in the leadership position for the 5G networks (5gp, 2017). The 5GPPP has been supporting different projects in its first¹⁰¹⁰105G PPP Phase I Projects - https://5g-ppp.eu/5g-ppp-phase-1-projects/ and second¹¹¹¹115G PPP Phase II Projects - https://5g-ppp.eu/5g-ppp-phase-2-projects/ phases such as METIS and SELFNET. Those projects focus on research topics ranging from physical infrastructure to overall architecture, virtualization, network management, and software networks. As a result, 5GPPP has published several specifications, including a view on the 5G architecture (Group, 2016a).

A 5G infrastructure must provide features that support different types of vertical business such as Automotive (e.g., car manufacturers), eHealth (e.g., health industry), Energy (e.g., power companies), Factories (e.g., IoT technology providers), Media & Entertainment (e.g., content providers) (Group, 2016a). All of those markets encompass different types of use cases (e.g., automated driving, robotics for remote surgery, on-site live event experience, etc.) that have their characteristics (e.g., data traffic patterns, mobility support, etc.) and requirements (e.g., throughput, latency, etc.). An exhaustive list of case studies for 5G can be found in (Group, 2016b). Furthermore, some performance requirements have been enumerated for such new generation of mobile networks (Gupta and Jha, 2015; Osseiran et al., 2014):

10 to 100 times higher data rate (1 to 10 Gbps);
5 times reduced end-to-end latency (less than 5 milliseconds);
1000 times higher capacity (9 Gigabytes per hour in busy period and 500 Gigabytes per month per subscriber);
10 to 100 times higher massive number of connections (300,000 connections per access point);
10 times extended battery life;
Availability of 99.999%.

To support such heterogeneity in the use cases as well as to meet the performance requirements, new 5G technologies will impact the entire mobile network including mobile devices; radio access, transport, and core networks; and the cloud (local, regional, or global). Figure 13 shows this type of infrastructure. A 5G architecture should enable speed, agility, and cost-efficiency when delivering new services such as those in the context of the Internet of Things (IoT) and Smart Cities. 5G networks should also provide multi-tenancy, multi-service, and multi-domain support. To this end, the infrastructure providers must allocate logical networks (accessible by northbound APIs), so-called network slices (Network Slice layer), or for mobile operators or service providers, who in turn can create their own slices or services (Software Network Service Chain and Service layers). To build logical networks, the infrastructure providers will have to deploy end-to-end resource, infrastructure (Resource Abstraction and Virtualization layer), and service orchestration functions to reserve appropriate computing and network resources from different administrative domains, keeping QoS tailored to user demand (Group, 2016a).

Finally, in the Radio Access Network (RAN), 5G architecture should operate in a broad spectrum range with a diverse variety of characteristics, provide efficient transmission and data processing, support the coexistence of different radio access technologies (5G, LTE, and Wi-Fi) and be energy efficient. In this case, techniques such as Mobile Edge Computing (MEC) can be used as it allows pushing the services to the RAN with the objective of meeting the ultra-low latency and higher-speed requirements (Group, 2016a; Technologies, 2015).

Figure 13. Software network technologies for 5G infrastructures (Group, 2016a).

The key enablers to achieve these functions are virtualization, softwarization, and programmability features, which can deliver the suitable level of flexibility in 5G networks. The use of NFV and SDN technologies will play a significant role in 5G networks, since they allow the network programmability and the fast delivery of new services, enabling network slicing and MEC implementation and orchestration (Group, 2017). In this context, several NFV/SDN architectures have been designed to overcome most of these challenges. We highlight some prospective solutions in the following sub-sections.

5.6.1. Mobile Network Function Virtualization

These architectures use cloud computing to assist mobile network virtualization. The goal is to provide a virtualization and communication platform for mobile network services as a way to deliver a flexible and scalable environment.

Regarding 3G services, (Kyung et al., 2015) proposed the Software Defined Transitional Networking (SDTN), an NFV/SDN architecture to support legacy service integration in 4G networks. The authors assumed LTE as the underlying network (SDTN data plane). 3G functions are VNF instances that replace the following components: Serving General Packet Radio Service (GPRS) Support Node (SGSN), Gateway GPRS Support Node (GGSN), and Home Location Register (HLR). The Edge Controller acts as an SDN Controller mapping the actions performed by the virtualized 3G functions to the physical 4G network (using the 4G forwarding control plane). The Edge Controller coordinates the redirection of network services traffic to VNFs by programming edge switches (using OpenFlow).

When considering 4G services, most of the studies that focus on the mobile core network include the virtualization of Evolved Packet Core (EPC), such as: Serving Gateway (SGW) (Italia et al., 2015; Unicom et al., 2015; Haleplidis et al., 2014; Basta et al., 2014; Nguyen and Kim, 2014; Costa-Requena et al., 2015; Medhat et al., 2015; Ahmad et al., 2016; Tawbeh et al., 2017), Packet Data Network Gateway (PGW) (Italia et al., 2015; Unicom et al., 2015; Haleplidis et al., 2014; An et al., 2014; Basta et al., 2014; Nguyen and Kim, 2014; Costa-Requena et al., 2015; Medhat et al., 2015; Ahmad et al., 2016; Tawbeh et al., 2017), Mobility Management Entity (MME) (Italia et al., 2015; Unicom et al., 2015; Nguyen and Kim, 2014; Costa-Requena et al., 2015; Medhat et al., 2015; Ahmad et al., 2016), Home Subscriber Server (HSS) (Italia et al., 2015; Costa-Requena et al., 2015; Ahmad et al., 2016), and Policy and Charging Rules Function (PCRF) (Costa-Requena et al., 2015; Ahmad et al., 2016).

In (Italia et al., 2015) and (Costa-Requena et al., 2015), the authors proposed a PoC to evaluate the virtualization of EPC components (vSGW, vPGW, and vMME) as VNFs over an NFV/SDN architecture. The testbed comprised eNodeBs (emulator or eNodeB model Flexi Zone from Nokia Networks) interconnected with a Cloud data center through OpenFlow switches with MPLS support (Coriant Oy 8615 Smart Router). All EPC VNFs run in a Cloud environment and were implemented using the following tools: eMME SW module (Aalto University) for vMME, open source nwEPC¹²¹²12https://www.openhub.net/p/nwepc for vSGW and vPGw, and an SQL database fo vHSS. A Ryu Controller coordinates the OpenFlow switches providing NFVI connectivity, eNodeB and VNFs interconnection and QoS support using MPLS tagging.

In addition, several studies proposed NFV/SDN architectures for virtualization of SGi-LAN services (Telefonica et al., 2016; Telecom et al., 2014; Telenor et al., 2016; Grønsund et al., 2015). Serving Gateway interface (SGi) interconnects mobile packet core and external IP networks. In a 4G network, SGi runs between PGW and a Packet Data Network (PDN) being responsible for ensuring the intercommunication performance and reliability. For this, SGi encompasses some services, such as Deep Packet Inspection (DPI), firewall, NAT, TCP optimization, and several caches. Gronsund et al. (2015) (Grønsund et al., 2015) proposed to replace the SGi elements for a physical OpenFlow Switch. The VNFs (TCP and Video Optimizer, firewall, HTTP content filter, etc.) run in a Cloud data center environment with RHEL OpenStack (Red Hat). The OpenStack coordinates the OpenDaylight Helium Controller to create OpenFlow rules in the switch to redirect and load balancing traffic to the VNFs. To keep track of VNF instances (in constant quantity variation for elasticity purposes) the OpenStack uses the LISP mapping service of OpenDaylight.

5.6.2. Network Slicing

Network Slicing refers to the partitioning of a certain physical infrastructure, composed of both network and computational resources, into multiple logical networks, called network slices (Ordonez-Lucena et al., 2017). Figure 14 shows that each slice is a self-contained network with its own virtual resources created on top of the underlying infrastructure. It can be designed and optimized for a particular mobile operator or service provider.

When compared to traditional physical networks, Network Slicing have the following advantages (Ordonez-Lucena et al., 2017): i) customization of logical networks according to service requirements; ii) on-demand provisioning to scale resources up or down as conditions change, and iii) network resource isolation for improved security and reliability.

Figure 14. Conceptual illustration of network slicing.

Network Slicing aims at providing efficient resource sharing, traffic differentiation per slice, and management and protection tools (Group, 2016a). NFV and SDN technologies are capable of providing the flexibility required in this context (Group, 2017).

A use case for the use of NFV/SDN architectures in Network Slicing is for the creation of SDN-enabled Virtual Tenant Networks (VTNs). VTNs are virtual networks deployed to different tenants in an isolated way (independent of underlying physical network resources) to support specific Quality of Service (QoS) and Service Level Agreement (SLA) requirements. There is a trend to use SDN in the creation of virtual networks. By enabling network programmability, SDN renders the abstraction necessary for its use as a network hypervisor. In the case of SDN-Enabled VTNs, one or many SDN controllers create a VTN (called an Infrastructure SDN Controller), while a new SDN controller is instantiated to manage this VTN (called a Tenant SDN Controller).

When an SDN-Enabled VTN deployment takes place, the respective Tenant SDN Controller is manually installed and configured on a dedicated server, which can be a long process. NFV/SDN architectures can be used to virtualize tenant SDN Controllers and provide fast and dynamic VTN provisioning.

The works of Munoz and Vilalta (Munoz et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015a, b, 2016a; Vilalta et al., 2016b) proposed an NFV/SDN solution for fast and dynamic deployment of SDN-Enabled Virtual Tenant Networks over multiple Data Centers and WAN domains. Their solution aims at providing geographically distributed cloud services with specific QoS and SLAs.

In (Munoz et al., 2015), the authors used NFV and Cloud to virtualize tenant SDN Controllers (OpenDaylight or Floodlight) to control the underlying SDN-enabled VTNs and provide fast and dynamic VTN provisioning (see Figure 15). They used OpenStack as VIM for each Data Center and an OpenDaylight controller to interconnect a virtual tenant SDN Controller with its respective VTN.

Figure 15. An NFV/SDN architecture design for SDN-enabled VTNs (Munoz et al., 2015).

To create the VTNs, they used the Multidomain SDN Orchestrator (MSO) mechanism as a Network Operating System (NOS). The MSO creates an abstraction over multiple domains including different transport network technologies thus enabling the composition of end-to-end services over heterogeneous WAN networks. Also, the authors use the Multidomain Network Hypervisor (MNH) to create end-to-end SDN-enabled VTNs, over the abstraction provided by MSO. Using the Global Cloud and Network Orchestrator, a VIM mechanism, this architecture integrates geographically distributed Data Centers and multiple WAN domains, providing a unified cloud and network operating system for the creation of end-to-end NFV services over VTNs.

Several research papers focus on providing Network Slicing (Akyildiz et al., 2015; Mwangama et al., 2015; Nguyen and Kim, 2014; Medhat et al., 2015; Ordonez-Lucena et al., 2017) for the new generation of mobile network. The 3GPP has identified Network Slicing as one of the key technologies to achieve the goals in 5G Networks (Ordonez-Lucena et al., 2017) since it is a potential solution to enable suitable flexibility to address the specific requirements of different use cases. In this scenario, mobile operators could share the same physical network substrate, adding their virtual networks with their services (e.g., 3G, 4G services) and a centralized management plane, creating the so-called Mobile Virtual Network Operators (MVNO). These logical networks must be isolated from each other as a way to maintain privacy between operators. In this case, NFV provides the mobile network services per operator as VNFs and, in turn, SDN creates the slice as well as establishes network functions interconnectivity.

Mwangama et al. (2015) (Mwangama et al., 2015) designed an NFV/SDN architecture to support MVNOs in a federated cloud environment. A prototype was implemented using the non-open source FOKUS OpenSDNCore Orchestrator (Fraunhofer FOKUS, 2016) to coordinate the network services between MVNOs. The Orchestrator uses OpenStack as VIM to create the virtual tenant networks and to instantiate the following VNFs per mobile operator: EPC (non-open source FOKUS OpenEPC¹³¹³13http://www.openepc.com/ platform), IMS – IP Multimedia Subsystem (open source FOKUS OpenIMSCore¹⁴¹⁴14http://www.openimscore.org/), M2M (non-open source FOKUS OpenMTC¹⁵¹⁵15http://www.open-mtc.org/).

Li et al. (2017) (Ordonez-Lucena et al., 2017) proposed a three-layer Network Slicing framework model for 5G networks considering NFV and SDN technologies. The bottom layer is the 5G Software-Defined Infrastructure (5G-SDI) which comprises multiple administrative and physical domains (e.g., RAN, transport and core networks, etc.) with SDN-based control and management. Their SDN-based approach uses hierarchically organized SDN controllers to provide abstraction and distributed dynamic allocation of resources. Furthermore, RAN and MEC can be deployed to enable a cloud-based infrastructure. The Virtual Resource layer creates network slices with virtual resources (radio, computing, and network) and VNFs that are customized to meet the requirements of different types of services. The Application and Service layer includes the per-tenant services (e.g., connected vehicles, virtual reality, etc.) that will use these slices to perform their functionalities. Also, the life cycle of network slices is managed and orchestrated by the Slicing MANO that acts as VIM, VNF Manager, and Slice Orchestrator.

Recently Munoz and Vilalta (2016 and 2017) (Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016) have adapted the previously defined NFV/SDN architecture (Munoz et al., 2015) to 5G scenarios, including the entire mobile network (e.g., radio access network) for fast and dynamic deployment of MVNOs. Besides, in (Muñoz et al., 2017; Vilalta et al., 2016) the authors proposed the ADRENALINE Testbed for 5G and IoT services on top of an NFV/SDN platform.

5.6.3. Mobile Edge Computing (MEC)

Mobile Edge Computing (MEC) or Multi-access Edge Computing has been a trend in mobile networks. Like NFV, the MEC architecture has been standardized by ETSI through Group Specification (GS) MEC (ETSI, 2016a, b) since 2016. MEC provides IT and Cloud Computing capabilities within the Radio Access Network (RAN). For this, a set of computer and storage resources (e.g., data centers, clusters, etc.) are deployed at the edges of a mobile operator’s network to assist the core data center in supporting computing and communication (see Figure 16) (Roman et al., 2016). MEC focuses on delivering the services closest to the user, as a way to meet certain critical application (e.g., video analytics, Internet-of-Things, augmented reality, and data caching) requirements that are not supported only by Cloud Computing, such as high bandwidth, low latency and jitter, context awareness, and mobility support.

Figure 16. Mobile Edge Computing scenario.

According to 5G-PPP, MEC is vital technological component to enable 5G networks (Group, 2015). NFV/SDN architectures are in line with current trends for MEC solutions. Because it is a new technology, only a few studies have been found in this SLR. As an example, the EU H2020 SELFNET project (EU SELFNET Project, 2016)

proposes the design and implementation of an Autonomic Management Framework for 5G networks, using technologies such as SDN, NFV, Self-Organizing Network (SON), Cloud Computing, and Artificial Intelligence. This framework aims at reducing OPEX and at improving QoE of the end users, addressing the following self-organizing capabilities: i) self-protection against distributed cyber-attacks, ii) self-healing against network failures, and iii) self-optimization of the network traffic. In this context, Neves et al. proposed a SELFNET approach to support SFC in MEC scenarios

(Neves et al., 2016, 2017), to meet 5G requirements defined by the 5G-PPP initiative (5gp, 2017). They considered a federated cloud infrastructure (i.e., multiples edge NFVI-PoP and a core NFVI-PoP) to provide IT and network resources to execute VNFs that support some management elements and network services. The WAN Infrastructure Management (WIN) uses SDN Controllers to provide connectivity between edge NFVI-PoPs and the core NFVI-PoP through the creation of virtual tenant networks.

6. Taxonomy of Nfv/sdn Architectures Design

This section provides support to answer the third research question, by describing a taxonomy to organize the various decision-making levels for the design of NFV/SDN architectures.

Figure 17 depicts our proposed taxonomy that provides a conventional architectural design for using SDN in an NFV framework. It was derived from architectures and implementations found in the selected studies and published NFV/SDN reference architectures (ETSI, 2015; Verizon, 2016). This taxonomy is useful as a guide for simplifying the work of researchers when studying NFV/SDN architectures or providing new solutions.

Figure 17. A taxonomy to classify NFV/SDN architectures design.

In this taxonomy, the NFV/SDN architectures design was divided into two sides: NFV-side and SDN-side. In the NFV-side, we must decide whether or not to use two features inherent to the architecture design. We describe these features as follows.

Distributed NFV (D-NFV)::: In D-NFV, the MANO framework places Virtual Network Functions (VNFs) where they could be most efficiently and economically be deployed, such as in data centers, forwarding devices, or the CPEs (Gittik, 2014).
Multiple VIMs::: A designer could place Multiple VIMs in different NFVI-PoPs to support the multi-domain administration or in the same NFVI-PoP to provide scalability and performance (ETSI, 2015).

As far as we are concerned with NFV, it is important to identify the NFV Management and Orchestration (MANO) tools. Tools such as OpenMANO (Telefonica I+D, 2016) and OpenBaton (Fraunhofer FOKUS, 2016) can provide complete solutions for MANO. On the other hand, the OpenStack enables VIM implementation to provide support for existing or new VNF Managers and NFV Orchestrators.

On the SDN-side, a first step is the placement of SDN elements in the NFV Framework (ETSI, 2015). These elements are described below:

SDN Resources::: Comprise of both physical and virtual switches and routers;
SDN Controllers::: Responsible for controlling the SDN resources, determining the behavior of network traffic;
SDN Applications::: Interfaces with one or multiple SDN controllers to enforce high-level network policy, such as firewall, network address translation, QoS, and network management.

We list some possible locations for the placement of SDN Resources in the NFV Framework, as follows (ETSI, 2015):

Physical switch or router;
Virtual switch or router;
E-switch, software-based SDN-enabled switch in a server NIC;
Switch or router as a VNF.

There are also some possible locations for the placement of SDN Controllers in NFV Framework (ETSI, 2015). They are the following:

Merged with the Virtualized Infrastructure Manager (VIM);
Virtualized as a VNF;
As part of the NFVI and not as a VNF;
As part of the OSS/BSS;
As a Physical Network Function (PNF).

Finally, some locations for the placement of SDN Applications in NFV Framework are listed below (ETSI, 2015):

As part of a PNF;
As part of the VIM;
Virtualized as a VNF;
As part of an EMS;
As part of the OSS/BSS.

There are also some architectural decisions to be made when one considers the SDN Controller, as follows:

Does the solution implement multiple SDN Controllers? If so, what is the main objective? Multiple SDN Controllers are hierarchically distributed to provide performance, scalability, reliability, administrative domains interaction, or Network as a Service (NaaS) management in an NFV Framework.
What is the function of SDN Controller in the NFV Framework? Network Connectivity in the NFVI, Control of Virtual Networks, Interconnecting VNFCs, and Interconnecting VNFs are some of these functions (extracted from the studies selected in this SLR).

Finally, we must identify the SDN Controller tools, including its underlying software and the used bound interfaces (at South, North, West, and East). As SDN Controllers we can cite: OpenDaylight (Foundation, 2016), Floodlight (Networks, 2016), ONOS ((ONF), 2017), Ryu (Telegraph and (NTT), 2016), and POX (NOXRepo.org, 2016).

The next Sections (7 and 8) aim to answer research question 3, using this taxonomy to organize the description and the differences between the NFV/SDN solutions extracted from articles selected in SLR.

7. Nfv-Side Design

This section uses the NFV-side of the taxonomy described in Section 6 to organize the NFV/SDN solutions extracted from articles selected in the SLR.

We organized the studies according to how they implemented the components of MANO (i.e., the NFV Orchestrator, the VNF Manager, and the Virtualized Infrastructure Manager). For each component, the research studies are classified as follows:

Real Implementation: The study proposes and implements its own component;
Theoretical: The study proposes its own component, but it is not implemented;
Vendor-specific: The study uses a proprietary tool to implement the component.

The majority of the articles were classified as Real Implementation (see Figure 18). These studies adopt modern tools to assist in the implementation of solutions, mainly the VIM component (e,g., OpenStack (Rackspace Cloud Computing, 2016)). However, it is worth mentioning that most of them implement the orchestration functions without the support of existing NFV MANO frameworks, such as OpenStack Tacker, OpenMANO, OpenBaton, Open-O, ECOMP, Hurtle, and the like. On the other hand, three papers (Medhat et al., 2015; Carella et al., 2015; Mwangama et al., 2015) developed their solutions on top of the OpenSDNCore Orchestrator (Fraunhofer FOKUS, 2016), from Fraunhofer FOKUS Institute. Last, we highlight that the Vendor-specific solutions are only used in some PoCs from ETSI NFV ISG.

Figure 18. Number of studies per Type of Analysis/Year.

Furthermore, some studies have provided architectures that deal with Distributed NFV (D-NFV) and Multiple VIMs. With D-NFV we could place VNFs wherever they may be most effective (performance and scalability) and least expensive. On the other hand, Multiple VIMs are often used to perform management of several administrative domains or NFVI-PoPs. In this scenario, VIMs are hierarchically distributed. So-called secondary VIMs are responsible for managing NFVI-PoPs. The primary VIM controls the secondary VIMs to create an abstraction layer on all NFVI-PoPs and performs a centralized management. Such a hierarchy enables the creation of end-to-end network services, involving multiple domains (e.g., Cloud and WAN).

Implementation Studies Distributed NFV (Telenor et al., 2016; Shen et al., 2015; Cerrato et al., 2015; Soares et al., 2015, 2014; Vilalta et al., 2015a, 2015, 2016a; Mohammadkhan et al., 2015; Muñoz et al., 2015; Munoz et al., 2015; Vilalta et al., 2015b; Carella et al., 2015; Sonkoly et al., 2015; Neves et al., 2016; Lombardo et al., 2015; Basta et al., 2014; Kyung et al., 2015; Mamatas et al., 2015; Rossem et al., 2015; Ordonez-Lucena et al., 2017; Cziva and Pezaros, 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016; Duan et al., 2016; Neves et al., 2017) Multiple VIMs (AT&T et al., 2015; Telenor et al., 2016; Basta et al., 2014; Munoz et al., 2015; Szabó et al., 2015; Neves et al., 2016; Cerrato et al., 2015; Soares et al., 2015, 2014; Mohammadkhan et al., 2015; Shen et al., 2015; Batalle et al., 2013; Carella et al., 2015; Kyung et al., 2015; Vilalta et al., 2015a, 2015, b, 2016a; Muñoz et al., 2015; Sonkoly et al., 2015; Rossem et al., 2015; Ordonez-Lucena et al., 2017; Cziva and Pezaros, 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016; Duan et al., 2016; Neves et al., 2017)

Table 4. List of studies addressing Distributed NFV and/or Multiple VIMs.

Table 4 lists the studies that implement Distributed NFV or Multiple VIMs. It is worth mentioning that most of the studies implemented both designs (see Figure 19). As an example, (Shen et al., 2015) proposed the vConductor, a Cloud CPE (see Section 5.4

) solution for automation of multi-tenant virtual network provisioning. vConductor deploys all enterprise network functions as VNFs in a Cloud domain comprised of multiple data centers. By using a User Portal, the customers can acquire new network functions and define how their VNFs must be chained. A virtual tenant network (VTN) is established connecting the enterprise CPE and the Cloud infrastructure through an OpenFlow-enabled WAN domain. Each data center in a Cloud domain uses an OpenStack as a management platform (secondary VIM). Further, an OpenDaylight Controller (secondary VIM) updates the OpenFlow rules required for VTN management in WAN domain. vConductor acts as NFVO, VNFM, and primary VIM, controlling the multiple secondary VIMs. Finally, vConductor includes the Virtual Network Life Cycle Manager (VNLM) to creates a D-NFV scenario. VNLM implements a multi-objective resource scheduling algorithm (MORSA) that uses a genetic algorithm to provides near-optimal placement of VNFs over different data centers.

Figure 19. Number of studies addressing Distributed NFV and/or Multiple VIMs.

However, there are also works that implement only one of these scenarios. As an example of D-NFV scenarios without multiple VIMs, (Lombardo et al., 2015) proposed an NFV/SDN architecture, called NetFATE (Network Functions At an Edge), aimed at allocating VNFs at the CPE nodes (multiple NFVI-PoPs) to minimize end-to-end latency. For this, a general-purpose computer replaces the old proprietary hardware-based CPE. The new CPE operating system is the CentOS 6.4 running a Xen Hypervisor (Foundation, 2016) for network function instantiation as VNFs, and an Open vSwitch (OVS) for VNF interconnection. NetFATE works as the MANO framework, using the NFV Coordinator (C++ software) to manage the VNF life cycle and a POX controller to control the OVSs. Finally, the Orchestration Engine determines how to distribute the VNFs and compose the network services.

When we consider the use of multiple VIMs without distributed NFV, we usually have a scenario where there are two secondary VIMs, one to manage a data center for virtualization purposes (Cloud domain) and another to manage a transport network (WAN domain) for end-to-end network services provisioning. As an example, PoC 16 (AT&T et al., 2015) proposed a multi-domain NFV/SDN architecture intended to provide enterprise services (firewall, IPS/IDS, and load balancer) to remote users across an MPLS-based transport network. This PoC uses an OpenStack (secondary VIM) as cloud orchestrator for VNF instantiation while ensuring end-to-end connectivity and SLAs over the WAN by using OpenFlow with Ryu controller (secondary VIM). The NFV Orchestrator acts as both NFVO and primary VIM, controlling the secondary VIMs to instantiate the end-to-end network services.

8. Sdn-Side Design

This section uses the SDN-side of the taxonomy described in Section 6 to organize the NFV/SDN solutions regarding the SDN.

Position Studies
NFVI (Munoz et al., 2015; Szabó et al., 2015; Deng et al., 2015; Neves et al., 2016; Cziva et al., 2015; Sonkoly et al., 2015; Cerrato et al., 2015; Soares et al., 2015; Cziva et al., 2015; Grønsund et al., 2015; Soares et al., 2014; Vilalta et al., 2015a, 2015, 2016a; Ahmad et al., 2016; Lucrezia et al., 2015; Mohammadkhan et al., 2015; Akyildiz et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015b; Nguyen and Kim, 2014; Giotis et al., 2015; Medhat et al., 2015; Ding et al., 2015; Rossem et al., 2015; Lin et al., 2015; Callegati et al., 2015; Batalle et al., 2013; Costa-Requena et al., 2015; Carella et al., 2015; Lombardo et al., 2015; Lai et al., 2015; Wang et al., 2015; Mwangama et al., 2015; Xia et al., 2015; Vestin and Kassler, 2015; Schulz-Zander et al., 2015; Mamatas et al., 2015; Telefonica et al., 2014; NTT et al., 2014; Telekom et al., 2014; Telefonica et al., 2016; AT&T et al., 2015; Telecom et al., 2014; Unicom et al., 2015; DT et al., 2015; Telenor et al., 2016; Telstra et al., 2016; Ordonez-Lucena et al., 2017; Schulz-Zander et al., 2017; Cziva and Pezaros, 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016; Callegati et al., 2017; Neves et al., 2017)
VNF (Haleplidis et al., 2014; Haleplidis et al., 2014; Neves et al., 2016; Kim, 2015; Telefonica et al., 2016; Neves et al., 2017)

Table 5. The position of virtual switches in NFV Framework.

8.1. Placement of SDN Elements in the NFV Framework

Table 5 shows how the studies place virtual switches in the given NFV Framework. The NFVI is the most used as a location for SDN resources. This scenario is a common approach to providing network programmability and flexibility for connectivity and traffic steering among VNFs.

However, some works also include virtual switches as VNFs. These works intend to provide an SDN-enabled virtual network to different customers. In (Haleplidis et al., 2014; Haleplidis et al., 2014), this placement is possible because they work with the Forwarding and Control Element Separation (ForCES) protocol (Doria et al., 2010) as SDN Southbound API and consider the Logical Functional Blocks (LFBs) as VNFs (detailed in Section 9.5.2). On the other hand, Neves et al. (Neves et al., 2016, 2017) created an abstraction for deployment of network services through the instantiation of Virtual Network Elements (VNE). VNEs are VNFs running a virtual switch process that perform packet processing (networking services) over the network traffic. VNEs can be distributed in the core or the edge NFVI-PoPs (see Section 5.6.3).

Table 6 shows how the studies positioned the SDN Controllers in the given NFV Framework. According to the (ETSI, 2015), an SDN Controller can run in five (5) places: NFVI, VIM, VNF, OSS/BSS, and can be a Physical Network Function (PNF). In this work, we did not find references for the last 2 (two) placements. Table 7 shows how the studies positioned the SDN Applications in the NFV Framework. According to the (ETSI, 2015), SDN Applications can run in five (5) points: as part of a PNF, as part of the VIM, virtualized as a VNF, as part of an Element Manager (EM), and as part of the OSS/BSS. In this work, we did not find references to SDN Applications as part of a PNF or an EM.

The VIM is the most used as a position for both SDN Controllers and Applications (see Figure 20). The VIM is the best place for these elements because it offers a global view of both NFVI physical and virtual infrastructures and the VNFs. This property allows the implementation of different functionalities, such as VNFs or VNFCs interconnections, network connectivity in the NFVI, and the control of virtual networks as shown in Table 8.

However, the NFVI have also been widely adopted as SDN Controller placement. According to (ETSI, 2015), this scenario is a classic case of the SDN controller providing network connectivity in the NFVI. In this work, we consider the SDN controller as a NFVI component when the VIM and its functions are distinguishable, as in the case of OpenStack controlling OpenDaylight.

Figure 20. Number of studies addressing the position of SDN Controllers in the NFV Framework.

Position Studies
NFVI (Medhat et al., 2015; Lucrezia et al., 2015; Carella et al., 2015; Rossem et al., 2015; Callegati et al., 2015; Cerrato et al., 2015; Soares et al., 2015, 2014; Munoz et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015a, b, 2015, 2016a; Szabó et al., 2015; Deng et al., 2015; Sonkoly et al., 2015; Grønsund et al., 2015; Ahmad et al., 2016; Lin et al., 2015; Telekom et al., 2014; BT et al., 2015; DT et al., 2015; Telenor et al., 2016; Telstra et al., 2016; Ordonez-Lucena et al., 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016; Duan et al., 2016)
VIM (Haleplidis et al., 2014; Szabó et al., 2015; Matias et al., 2015; Neves et al., 2016; Cziva et al., 2015; Sonkoly et al., 2015; Cziva et al., 2015; Cheng et al., 2015; Mohammadkhan et al., 2015; Shen et al., 2015; Akyildiz et al., 2015; Haleplidis et al., 2014; Nguyen and Kim, 2014; Giotis et al., 2015; Ding et al., 2015; Rossem et al., 2015; Batalle et al., 2013; Carella et al., 2015; Lombardo et al., 2015; Lai et al., 2015; Wang et al., 2015; Mwangama et al., 2015; Xia et al., 2015; Vestin and Kassler, 2015; Saridis et al., 2016; Basta et al., 2014; An et al., 2014; Kyung et al., 2015; Schulz-Zander et al., 2015; Mamatas et al., 2015; Kim, 2015; Telefonica et al., 2014; NTT et al., 2014; Telecom et al., 2014; Unicom et al., 2015; Schulz-Zander et al., 2017; Cziva and Pezaros, 2017; Callegati et al., 2017; Neves et al., 2017)
VNF (Munoz et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015a, b, 2016a; Neves et al., 2016; Nguyen and Kim, 2014; Rossem et al., 2015; Costa-Requena et al., 2015; Telefonica et al., 2016; AT&T et al., 2015; Italia et al., 2015; Unicom et al., 2015; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016; Tawbeh et al., 2017; Neves et al., 2017)

Table 6. The position of SDN Controllers in the NFV Framework.

A good example to illustrate SDN Controllers and Applications placement is the work of Rossem et al. (2015) (Rossem et al., 2015). In that work, the authors have used ESCAPE (see Section 5.3) environment to implement a NFV/SDN solution for elastic virtual router provisioning, needed in a VPN service. The main goal is to increase the throughput by load balancing (using Valliant Load Balancing) traffic among multiple virtual switches. In this architecture, the Service Layer receives the VPN requests and define the required VNFs to be instantiated by Orchestration Layer (OL) in an optimal way. There are three VNF types: Ctrl App, OF Ctrl, and SDN-enabled virtual switches. The Ctrl App and OF Ctrl are deployed as VMs in OpenStack (Cloud domain), while SDN switches are deployed in an Mininet¹⁶¹⁶16http://mininet.org/ emulator (representing an OpenFlow domain). On the data plane, an elastic router comprises one or more SDN switches. On the control plane, the SDN Ctrl manages the topology creation on the top of SDN switches. Moreover, the SDN application Ctrl App monitors the SDN flow statistics and triggers topology changes (if needed), adding more or less SDN switches.

In (Rossem et al., 2015), a hybrid solution for the positioning of SDN controllers was proposed. In this case, the POX Controller (NOXRepo.org, 2016) was placed on VIM to support the creation and management of network services in OpenFlow domains. The OpenDaylight (Foundation, 2016) was placed on NFVI and is used by the OpenStack (Rackspace Cloud Computing, 2016) to provide connectivity in the Cloud domain. Finally, the SDN Ctrl is a Ryu Controller created as a VNF to coordinate the SDN-enabled virtual network.

Position Studies
VIM (Carella et al., 2015; Callegati et al., 2015; Neves et al., 2016; Deng et al., 2015; Munoz et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015a, b, 2015, 2016a; Szabó et al., 2015; Cziva et al., 2015; Sonkoly et al., 2015; Cerrato et al., 2015; Soares et al., 2015; Cziva et al., 2015; Soares et al., 2014; Lucrezia et al., 2015; Mohammadkhan et al., 2015; Shen et al., 2015; Nguyen and Kim, 2014; Medhat et al., 2015; Ding et al., 2015; Rossem et al., 2015; Lin et al., 2015; Lombardo et al., 2015; Xia et al., 2015; Vestin and Kassler, 2015; Schulz-Zander et al., 2015; Mamatas et al., 2015; Kim, 2015; Telefonica et al., 2014; NTT et al., 2014; Telekom et al., 2014; AT&T et al., 2015; Unicom et al., 2015; DT et al., 2015; Telenor et al., 2016; Telstra et al., 2016; Ordonez-Lucena et al., 2017; Schulz-Zander et al., 2017; Cziva and Pezaros, 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016; Duan et al., 2016; Neves et al., 2017; Callegati et al., 2017)
VNF (Grønsund et al., 2015; Haleplidis et al., 2014; Matias et al., 2015; Neves et al., 2016; Ahmad et al., 2016; Akyildiz et al., 2015; Nguyen and Kim, 2014; Giotis et al., 2015; Rossem et al., 2015; Batalle et al., 2013; Costa-Requena et al., 2015; Lai et al., 2015; Wang et al., 2015; An et al., 2014; Kyung et al., 2015; Telefonica et al., 2016; BT et al., 2015; Telecom et al., 2014; Italia et al., 2015; Unicom et al., 2015; Tawbeh et al., 2017; Neves et al., 2017)
OSS/BSS (Munoz et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015a, b, 2015, 2016a; Szabó et al., 2015; Cheng et al., 2015; Haleplidis et al., 2014; Mwangama et al., 2015; Xia et al., 2015; Saridis et al., 2016; Basta et al., 2014; Schulz-Zander et al., 2015; Mamatas et al., 2015; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016)

Table 7. The position of SDN Applications in the NFV Framework.

Regarding SDN Applications placement, the SDN App also runs as a VNF on top of SDN Ctrl (Ryu Controller). For OpenFlow domains, the SDN applications run as a VIM component, on top of POX. Finally, for Cloud domains, the Neutron service (VIM) performs the control of OpenDaylight instance.

The Operations support systems (OSS) and Business Support Systems (BSS) have also been widely adopted as SDN Applications placement. Application at this level enables multiple tenants to control dedicated SDN networks to provide their own services. This scenario is common in works that propose solutions for 5G Cellular Networks (Neves et al., 2016, 2017; Mwangama et al., 2015). Examples of SDN application placement as OSS/BSS management task are the works of Munoz and Vilalta (Munoz et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015a, b, 2016a; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016). In those works, a tenant SDN Controller runs as a VNF to control an underlying VTN. The end-users or service provider operators (components of OSS/BSS element) have direct access to this controller and can implement customized applications for VTN control and management.

8.2. SDN Controller Functions in the NFV Framework

Table 8 shows the possible functions for SDN Controllers when applied in the NFV Framework. Below, we describe these functions:

Interconnecting VNFs/VNFCs::: The SDN Controller might be used to connect and manage the traffic between VNFs and/or VNFCs to enable Network Services, by creating Service Function Chaining (SFC). As seen in Section 2.1, a VNF might be composed of several VNFCs.
Network Connectivity in the NFVI::: The SDN Controller is used to provide L2/L3 connectivity among end-to-end devices;
Control of Virtual Networks::: The SDN Controller might be responsible for creating and managing virtual networks for different customers.

Figure 21. Number of studies addressing SDN Controller functions.

Function Studies
Interconnecting VNFs/VNFCs (Carella et al., 2015; Haleplidis et al., 2014; Szabó et al., 2015; Neves et al., 2016; Cziva et al., 2015; Sonkoly et al., 2015; Cerrato et al., 2015; Soares et al., 2015; Cziva et al., 2015; Grønsund et al., 2015; Soares et al., 2014; Cheng et al., 2015; Ahmad et al., 2016; Lucrezia et al., 2015; Mohammadkhan et al., 2015; Haleplidis et al., 2014; Nguyen and Kim, 2014; Medhat et al., 2015; Ding et al., 2015; Rossem et al., 2015; Lin et al., 2015; Callegati et al., 2015; Lombardo et al., 2015; Lai et al., 2015; Wang et al., 2015; Mwangama et al., 2015; Xia et al., 2015; Basta et al., 2014; An et al., 2014; Kyung et al., 2015; Schulz-Zander et al., 2015; Mamatas et al., 2015; Kim, 2015; Telefonica et al., 2014; NTT et al., 2014; Telekom et al., 2014; Telefonica et al., 2016; AT&T et al., 2015; BT et al., 2015; Telecom et al., 2014; Italia et al., 2015; Unicom et al., 2015; DT et al., 2015; Telenor et al., 2016; Telstra et al., 2016; Ordonez-Lucena et al., 2017; Schulz-Zander et al., 2017; Cziva and Pezaros, 2017; Callegati et al., 2017; Tawbeh et al., 2017; Duan et al., 2016; Neves et al., 2017)
Network Connectivity in the NFVI (Grønsund et al., 2015; Vestin and Kassler, 2015; Lucrezia et al., 2015; Callegati et al., 2015; Rossem et al., 2015; Cerrato et al., 2015; Soares et al., 2015, 2014; Deng et al., 2015; Munoz et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015a, b, 2015, 2016a; Szabó et al., 2015; Matias et al., 2015; Neves et al., 2016; Sonkoly et al., 2015; Ahmad et al., 2016; Shen et al., 2015; Akyildiz et al., 2015; Giotis et al., 2015; Medhat et al., 2015; Batalle et al., 2013; Costa-Requena et al., 2015; Carella et al., 2015; Lombardo et al., 2015; Saridis et al., 2016; Basta et al., 2014; Kyung et al., 2015; Schulz-Zander et al., 2015; Kim, 2015; Telefonica et al., 2014; Telekom et al., 2014; AT&T et al., 2015; BT et al., 2015; Italia et al., 2015; Unicom et al., 2015; Telenor et al., 2016; Telstra et al., 2016; Ordonez-Lucena et al., 2017; Schulz-Zander et al., 2017; Cziva and Pezaros, 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016; Tawbeh et al., 2017; Duan et al., 2016; Neves et al., 2017)
Control of Virtual Networks (Vestin and Kassler, 2015; Neves et al., 2016; Munoz et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015a, b, 2016a; Akyildiz et al., 2015; Nguyen and Kim, 2014; Mwangama et al., 2015; Saridis et al., 2016; Kyung et al., 2015; Schulz-Zander et al., 2015; Telefonica et al., 2014; Telstra et al., 2016; Schulz-Zander et al., 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016; Neves et al., 2017)

Table 8. List of possible functions for SDN Controllers when applied in NFV Framework.

It is worth emphasizing that interconnecting VNFs is the main objective for using SDN in the NFV Framework (see Figure 21), mainly for automation and monitoring of SFC deployments. The works of Cziva el al. (Cziva et al., 2015, 2015; Cziva and Pezaros, 2017) used an OpenDaylight Controller to create OpenFlow rules in Open vSwitch instances and OpenFlow switches, intending to provide interconnectivity and traffic steering for network services (set of linked VNFs). According to (Mijumbi et al., 2016), there are some problems regarding Interconnecting VNFs, such as network function placement, survivability of VNFs, and dynamic service function chaining (elasticity).

As an example of SDN Controllers performing Network Connectivity in the NFVI and Control of Virtual Networks, Vestin et al. (2015) (Vestin and Kassler, 2015) used OpenDaylight Controller as VIM to manage the communication among physical APs and Virtual Access Points (VAP) (see Section 5.5.1). In this case, the controller provides Network connectivity in the NFVI by connecting physical AP and Cloud infrastructure, and Control of Virtual Networks by using OpenFlow to connect a mobile client with its respective VAP.

8.3. The Use of Multiple SDN Controllers

Table 9 shows the main objectives for implementing Multiple SDN Controllers in the NFV Framework. As well as the use of Multiple VIMs (see Section 7), multiple SDN controllers are organized hierarchically to provide the following objectives.

Distributed Performance::: When the VNFs have to be distributed to the chosen location, they are interconnected by location-specific SDN Controllers;
Scalability::: Multiple controllers manage an NFVI-PoP infrastructure;
Reliability::: Fault tolerance, disaster recovery, and full isolation management.
Administrative Domains Interaction::: Communication management of different scenarios (e.g., Cloud and WAN) using different SDN Controllers integrated hierarchically in a unique platform;
Network as a Service (NaaS) Management::: The concept of NaaS is related to the provision of virtualized network services to customers with different requirements (Mustafa et al., 2015). This function includes network virtualization (Network Slicing).

Objective Studies
Distributed Performance (Telenor et al., 2016)
Scalability (Callegati et al., 2015)
Reliability (Munoz et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015a, b, 2016a; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016)
Administrative Domains Interaction (Rossem et al., 2015; Neves et al., 2016; Munoz et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015a, b, 2015, 2016a; Szabó et al., 2015; Sonkoly et al., 2015; Batalle et al., 2013; Carella et al., 2015; Kyung et al., 2015; Telefonica et al., 2014; AT&T et al., 2015; Ordonez-Lucena et al., 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016; Duan et al., 2016; Neves et al., 2017)
NaaS Management (Neves et al., 2016; Munoz et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015a, b, 2016a; Szabó et al., 2015; Cerrato et al., 2015; Akyildiz et al., 2015; Rossem et al., 2015; Telefonica et al., 2014; AT&T et al., 2015; Ordonez-Lucena et al., 2017; Cziva et al., 2015, 2015; Cziva and Pezaros, 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016; Neves et al., 2017)

Table 9. List of objectives for use Multiple SDN Controllers.

In PoC 34 (Telenor et al., 2016), the developers implement Distributed Performance by putting EPC Virtual Functions to different NFVI-PoPs, each NFVI-PoP managed by its own SDN Controller (OpenDaylight). These controllers communicate with each other to allow the programming of GPRS Tunneling Protocol (GTP) tunnels interconnecting VNFs placed in different locations.

Scalability is handled in Callegati et al. (2015) (Callegati et al., 2015). In this work, the authors used two SDN controllers to manage the virtual networks in an OpenStack-based Cloud environment. They create OpenFlow rules using a Neutron Open vSwitch Agent for connectivity in NFVI and interconnection of VNFs, and monitor the throughput of OpenFlow rules using a POX (NOXRepo.org, 2016) controller for traffic steering mechanisms.

Regarding Reliability, the works of Munoz et al. (2015) (Munoz et al., 2015; Muñoz et al., 2015) used NFV and Cloud to virtualize tenant SDN Controllers to control the underlying VTNs. The authors pointed out that a clear advantage of using cloud virtualization for SDN controllers is the reliability achieved with the lack of hardware maintenance downtime and the decreasing recovery time.

However, it is worth noting that Administrative Domains Interaction and NaaS Management are the main objectives for using multiple SDN controllers in the NFV Framework. As an example, Rossem et al. (2015) (Rossem et al., 2015) used three SDN controllers in their NFV/SDN architecture to provide elastic virtual router provisioning in a multi-domain scenario (described in Section 8.1). For Administrative Domains Interaction, the authors used OpenDaylight and POX controllers to provide an SDN-enabled virtual network on top of a Cloud (OpenStack-based) and a WAN domain (OpenFlow-based), respectively. Finally, for NaaS Management, the authors instantiate SDN Controllers (Ryu) as VNFs to control the underlying virtual networks.

9. Auxiliary Tools

This section presents the tools used in the studies carried out to implement NFV/SDN architectures. The primary focus is on free or open-source tools, which are thus freely available for future implementations.

9.1. VNF Instantiation

The following open source tools have been used in NFVI to enable virtualization of network functions.

Kernel Virtual Machine (KVM) (Project, 2016)::: KVM enables Linux Kernel as hardware-assisted virtualization hypervisor on x86 and x86-64 systems equipped with virtualization extensions (Intel VT or AMD-V). KVM allows the running of multiple virtual machines with unmodified Unix-based systems (e.g., Linux or NetBSD) and Windows images. All studies that applied OpenStack as VIM (see 9.2) used KVM because this is the OpenStack default hypervisor.
Xen Hypervisor (Foundation, 2016)::: Xen is a hypervisor that allows the creation of multiple virtual machines, using paravirtualization capabilities in x86, x86-64, ARM, and PowerPC architectures. By working with paravirtualization, Xen uses modified images (e.g., Linux or Windows) to provide fast execution of virtual machines. As seen in Section 7, Lombardo et al. (2015) used Xen to enable the rapid deployment of new VNFs on vCPE nodes (Lombardo et al., 2015).
ClickOS (Martins et al., 2014)::: ClickOS is a high-performance Xen-based software platform that enables VNF development using a Click modular router software running on top of MiniOS. It allows the creation of small VMs (5MB) with a fast bootloader (30 milliseconds). As seen in Section 5.3, Deng et al. (2015) used ClickOS for its virtual firewall development (Deng et al., 2015).
Docker Engine (Inc., 2016)::: Docker uses container-based virtualization to create multiple isolated containers that run natively on Windows, Linux and MacOS systems. A container is just a package that contains a set of libraries needed to run the hosted applications. Unlike virtual machines, containers do not have a dedicated operating system, sharing the features of the host operating system. As a consequence, containers are faster and consume less computational resources, enabling improvements in VNF provisioning time (including up/down/update), runtime performance (e.g., throughput), and in the number of VNFs (hundreds) that a commodity computing devices can host (natarajan.sriram@gmail.com et al., 2016). As seen in Section 5.3, Cziva et al. (Cziva et al., 2015, 2015; Cziva and Pezaros, 2017) used Docker Engine to provide fast deployment of new virtual middleboxes with less resource utilization.
Data Plane Development Kit (DPDK) (Foundation, 2016)::: DPDK allows high-performance packet processing through a set of data libraries and drivers for network interface controlling. The VNF developer can use such libraries to create network functions with high throughput. These functions run as an isolated process and communicate through xDPd OpenFlow Switches (detailed in Section 9.3). (Cerrato et al., 2015) used DPDK processes in its integrated node to create VNFs with fast packet processing in equipment with limited resources.

For more information on paravirtualization and hardware-assisted virtualization, we refer the reader to the following white paper from VMWare¹⁷¹⁷17http://www.vmware.com/techpapers/2007/understanding-full-virtualization-paravirtualizat-1008.html: “Understanding Full Virtualization, Paravirtualization, and Hardware Assist”. For more information on container-based virtualization, please read the Docker definition¹⁸¹⁸18https://www.docker.com/what-container.

9.2. OpenStack Cloud Platform

Table 10 lists all studies that used OpenStack as part of their solutions. All of them used OpenStack as VIM for cloud domain management, seeking to provide a platform for VNF virtualization.

Considered one of the most significant open source projects, the OpenStack is a Cloud Platform that emerged in 2010 through an initiative of Rackspace Hosting¹⁹¹⁹19Rackspace website: https://www.rackspace.com/ and NASA. Its structure was based on the NASA platform Nebula and the Rackspace cloud file system. Since 2012, the project has been managed by OpenStack Foundation.

Tool Studies OpenStack (Munoz et al., 2015; Muñoz et al., 2015; Deng et al., 2015; Cerrato et al., 2015; Soares et al., 2015; Grønsund et al., 2015; Soares et al., 2014; Vilalta et al., 2015a, 2015, 2016a; Lucrezia et al., 2015; Shen et al., 2015; Vilalta et al., 2015b; Medhat et al., 2015; Rossem et al., 2015; Callegati et al., 2015; Costa-Requena et al., 2015; Carella et al., 2015; Mwangama et al., 2015; Telefonica et al., 2014; Telekom et al., 2014; Telefonica et al., 2016; DT et al., 2015; Telenor et al., 2016; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016)

Table 10. List of works using OpenStack.

The OpenStack Cloud Platform provides an Infrastructure as a Service (IaaS) solution through a set of related services, written in Python. Below we describe the main services.

KeyStone::: Represents the Identity Service. It provides functionalities such as authentication, authorization, and service catalog.
Nova::: Represents the Computing Service. It is responsible for managing the lifecycle of virtual machine instances. Currently, it provides support for different hypervisors: Xen, XenServer/XCP, QEMU, KVM, UML, VMware, vSphere, and Hyper-V.
Neutron::: Represents the Networking Service. This service provides connectivity to the interfaces of the VMs. It has a flexible API that allows the construction of complex networks: flat and shared networks, VLANs, VXLAN, GRE, DHCP, IPv6, and SDN, for example.
Glance::: Represents the Image Service. It stores and retrieves disk images of virtual machines. It also stores metadata of these images.
Swift::: Represents the Object Storage Service. It stores and retrieves unstructured data objects. It works with data replication to provide a highly tolerant and scalable architecture.
Cinder::: Represents the Block Storage Service. It provides a persistent storage block for running instances.

9.3. Virtual Switches

The Open vSwitch (OVS) (Open Source Community, 2016) was the most adopted virtual switch. Currently maintained by the Linux Foundation²⁰²⁰20https://www.linuxfoundation.org/, OVS aims to automate network tasks. Therefore, it supports many protocols, such as OpenFlow (versions 1.0 (ONF, 2009) and 1.3 (ONF, 2015)), OVSDB (Pfaff and Davie, 2013), NetFlow, sFlow, IPFIX, and the like. It also provides many additional features, such as VLAN isolation, traffic filtering, traffic queuing, and traffic shaping. Furthermore, the OVS has also been integrated into popular cloud platforms including oVirt, OpenNebula, and OpenStack.

GNF (Glasgow Network Functions) (Cziva et al., 2015, 2015; Cziva and Pezaros, 2017) proposes an NFV Framework for public clouds and uses OVS to provide the data plane required to interconnect network functions (Docker container-based) and connect them to arbitrary services. In Vestin and Kassler (2015), an extended OVS was installed in 802.11 Access Points to provide Virtual Access Points with QoS enforcement. The NetFATE (Network Functions At the Edge) platform (Lombardo et al., 2015) uses the OVS to implement NFV services at the edge of a Telco network, providing data plane in CPE nodes.

Carella et al. (2015) (Carella et al., 2015) adopted the OpenSDNCore Switches (OSCS) (Fraunhofer FOKUS, 2016) to provide resource reservation through the creation of flow entries and priority queues in a network layer. The OSCS is a software implementation (in C language) of an extended OpenFlow 1.4 switch with specific telecom oriented extensions supporting many features, such as OpenFlow 1.4, GPRS Tunneling Protocol (GTP), Generic Routing Encapsulation (GRE), asynchronous metrics (statistics), traffic shaping, and topology learning.

Finally, the eXtensible Datapath daemon (xDPd) (Foundation, 2016) provides a framework for building multiples high-performance OpenFlow datapath elements, called Logical Switch Instances (LSIs). xDPd is written in C/C++ and supports the following platforms: GNU/Linux amd64/x86 user-space, GNU/Linux Intel’s DPDK accelerated driver, NetFGPA-10G (netfpga10g), Broadcom Triumph2 (bcm), and Octeon network processors. xDPs support multiple OpenFlow versions (1.0, 1.2, and 1.3). Cerrato et al. (2015) used xDPd to create multiples LSIs (one for each customer) in the integrated node equipment with limited resources that represents a CPE (Cerrato et al., 2015) (as described in Section 5.4).

9.4. SDN Controllers

SDN Controller Studies
OpenDaylight (Munoz et al., 2015; Cziva et al., 2015; Sonkoly et al., 2015; Cerrato et al., 2015; Soares et al., 2015; Cziva et al., 2015; Grønsund et al., 2015; Soares et al., 2014; Vilalta et al., 2015a, 2015, 2016a; Lucrezia et al., 2015; Shen et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015b; Medhat et al., 2015; Vestin and Kassler, 2015; Saridis et al., 2016; Telenor et al., 2016; Cziva and Pezaros, 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016) ONOS (Schulz-Zander et al., 2017; Callegati et al., 2017)
Floodlight (Munoz et al., 2015; Cheng et al., 2015; Vilalta et al., 2015a; Muñoz et al., 2015; Vilalta et al., 2015b; Batalle et al., 2013; Schulz-Zander et al., 2015; BT et al., 2015)
Ryu (Rossem et al., 2015; Lin et al., 2015; Costa-Requena et al., 2015; Carella et al., 2015; Lai et al., 2015; NTT et al., 2014; AT&T et al., 2015; Italia et al., 2015)
POX (Sonkoly et al., 2015; Rossem et al., 2015; Callegati et al., 2015; Lombardo et al., 2015)

Table 11. List of works grouped by the used SDN Controllers.

Table 11 lists all SDN controllers used in implementations of NFV/SDN architectures. As seen in Section 8.1, such controllers have been used as NFVI, VIM or VNF components. Below, we present a brief description of them.

OpenDaylight (ODL) (Foundation, 2016)::: The ODL is a modular SDN open source platform maintained by The Linux Foundation²¹²¹21The Linux Foundation website: www.linuxfoundation.org/. Written in Java, the ODL aims to accelerate the development of solutions for SDN and NFV in production environments. ODL offers plugins that support different SDN Southbound API, such as OpenFlow (1.0 and 1.3), LISP, NETCONF, and OVSDB. Currently, the newest version of the ODL is the Boron, released in December 2016.
Open Network Operating System (ONOS) ((ONF), 2017)::: ONOS comprises an open source SDN Controller focused on the construction of NFV/SDN solutions. Like ODL, ONOS was developed in Java on top of the Apache Karaf OSGi container and provides the following features: i) a GUI for the view the network state, support different SDN Southbound API, such as OpenFlow, NETCONF, and OpenConfig; ii) northbound abstractions to simplify the creation of intent-based virtualized networks; iii) high availability and scalability support (e.g., cluster of ONOS instances).
Floodlight (Networks, 2016)::: Floodlight is a SDN Controller written in Java that supports the following OpenFlow versions: 1.0, 1.1, 1.2, 1.3, and 1.4. It is Apache-licensed and supported by engineers and developers from Big Switch Networks. In addition to OpenFlow controller, Floodlight provides a set of internal SDN applications (e.g., firewall and load balancing) and a REST API for development of external applications.
Ryu (Telegraph and (NTT), 2016)::: Ryu is an open source framework (Apache 2.0 licensed) created by NTT and written in Python. Ryu supports several southbound interfaces, such as OpenFlow, OF-config, and NETCONF. Regarding OpenFlow, Ryu supports the following versions: 1.0, 1.2, 1.3, 1.4, and 1.5. A REST API is available to be used for external SDN applications. Currently, Ryu is fully integrated into Neutron (OpenStack Networking Service).
POX (NOXRepo.org, 2016)::: Developed at Stanford University, POX was one of the first open source developed SDN controllers. Written in Python, POX only supports OpenFlow version 1.0. It is currently a discontinued project.

The adoption of both ODL and OpenStack to compose NFV/SDN solutions is emerging, mainly due to the soft integration of these tools. The Neutron service uses the Module Layer 2 (ML2) Plugin (Rackspace Cloud Computing, 2016) to provide networking services in a Cloud. The ML2 might control an ODL instance using the Neutron API, a REST API provided by ODL. It is worth mentioning that some new NFV/SDN architectures (Schulz-Zander et al., 2017; Callegati et al., 2017) with the focus only on SDN solutions (Kim et al., 2016; Muqaddas et al., 2017) have adopted the ONOS SDN Controller. This controller has gained momentum, and it is currently the main competitor of ODL. ONOS is the official distribution of the Open Network Foundation (ONF) along with the Open Networking Lab (ON.Lab). Furthermore, essential projects started as an ONOS use case, such as Central Office Re-architected as a Datacenter (CORD). CORD combines NFV and SDN technologies to create a general-purpose platform that is capable of delivering a broad range of innovative services targeting network operators, from access services (e.g., Fiber-to-the-Home) to general cloud services (SaaS) ((ONF), 2017). In addition to ONOS, CORD supports other open source tools such as OpenStack, Docker, etc. Major players such as AT&T, Google, Cisco, NEC, Nokia, Fujitsu, Intel, SK Telecom, Verizon, China Unicom and NTT Communications are already supporting CORD.

9.5. SDN Southbound Interfaces

Table 12 lists all the interfaces used as SDN Southbound API in the works. Such APIs are part of two SDN standards: OpenFlow and ForCES. These standards, as well as their interfaces to access network devices, are described below.

SDN Southbound API Studies ForCES Protocol (Haleplidis et al., 2014; Haleplidis et al., 2014)
OpenFlow Protocol (Cziva et al., 2015, 2015; Deng et al., 2015; Matias et al., 2015; Cheng et al., 2015; Ahmad et al., 2016; Mohammadkhan et al., 2015; Akyildiz et al., 2015; Nguyen and Kim, 2014; Giotis et al., 2015; Batalle et al., 2013; Costa-Requena et al., 2015; Carella et al., 2015; Lai et al., 2015; Xia et al., 2015; An et al., 2014; Kyung et al., 2015; NTT et al., 2014; BT et al., 2015; Telecom et al., 2014; Unicom et al., 2015; Telstra et al., 2016; Sonkoly et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015a, b, 2016a; Rossem et al., 2015; Callegati et al., 2015; Lombardo et al., 2015; Saridis et al., 2016; Ding et al., 2015; Lin et al., 2015; Vestin and Kassler, 2015; Schulz-Zander et al., 2015; Telekom et al., 2014; Telefonica et al., 2016; AT&T et al., 2015; Italia et al., 2015; DT et al., 2015; Schulz-Zander et al., 2017; Cziva and Pezaros, 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016; Callegati et al., 2017)
OVSDB Management Protocol (Cziva et al., 2015, 2015; Munoz et al., 2015; Muñoz et al., 2015; Cerrato et al., 2015; Soares et al., 2015; Grønsund et al., 2015; Soares et al., 2014; Vilalta et al., 2015a, 2015, b, 2016a; Lucrezia et al., 2015; Shen et al., 2015; Medhat et al., 2015; Telefonica et al., 2014; Callegati et al., 2015; Telenor et al., 2016; Cziva and Pezaros, 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016)

Table 12. List of works grouped by SDN Southbound API used.

9.5.1. OpenFlow

The development of OpenFlow began in 2007, and its evolution has received contributions from both academia and industry. Designed originally by researchers at Stanford University and the University of California at Berkeley, this standard has been maintained by the Open Networking Foundation (ONF) (Kreutz et al., 2015).

The ONF²²²²22Open Networking Foundation website: https://www.opennetworking.org/index.php is an organization committed to the development and dissemination of SDN. For this, ONF is responsible for the creation of open standards for SDN, such as the OpenFlow specifications. OpenFlow is frequently updated, thus adding new features from version 1.0 (ONF, 2009) to 1.3 (ONF, 2015). Multiple flow tables, group and meter tables, and MPLS support are some of the advances since version 1.1.

The OpenFlow comprises three main elements, namely switches, controllers, and protocols (Southbound API). The OpenFlow Switches are responsible for the data packet forwarding (i.e., data plane) according to the rules created and maintained by the Controller (ONF, 2015). Multiple Flow Tables store these rules. The controller is the main component of OpenFlow. In architectural terms, the controller supports the network applications, determining the rules to be stored and applied by switches. There are several OpenFlow controllers available, such as OpenDaylight, Floodlight, ONOS, Ryu, POX, and NOX.

The controller can use two types of interfaces to create OpenFlow rules on switches: the OpenFlow Protocol and the Open vSwitch Database (OVSDB) Management Protocol. Both interfaces are described below.

OpenFlow Protocol::: Defined in OpenFlow Specifications (ONF, 2015), this protocol uses a secure and encrypted channel (TCP/TLS) for performing the management of switches. It includes a set of messages for different situations: establishment and configuration of the management channel (e.g., Hello, Echo Request/Reply, Features Request/Reply, Set-Config), receiving (Packet-in) and redirecting (Packet-out) data packets, and OpenFlow rules management (Flow-mod). The OpenFlow protocol can coordinate all OpenFlow-enabled switches.
OVSDB Management Protocol::: Defined in RFC 7047 (Pfaff and Davie, 2013), this protocol uses a set of operations available in Open vSwitch (programmatic extension) to manage OpenFlow rules. Such operations allow insertion, updating, and deletion of forwarding rules directly into the Open vSwitch Database. As a consequence, the OVSDB Management Protocol is limited to use in virtual switches based on Open vSwitch.

As an example of using these two protocols, Callegati et al. (2015) (Callegati et al., 2015) proposed a solution to deploy multi-tenant service function chaining of edge network functions, using OpenStack. For this, the authors used an OVSDB management protocol (via Neutron Open vSwitch Agent) to provide connectivity to VNFs and an OpenFlow protocol (via POX controller) to monitor the throughput of OpenFlow rules.

9.5.2. Forwarding and Control Element Separation (ForCES)

ForCES is an SDN standard defined by the Internet Engineering Task Force (IETF) (Doria et al., 2010). In ForCES, the plane separation occurs by dividing the Network Elements (NE) into two entities: Forwarding Elements (FE) and the Control Elements (CE).

FEs represent the Data Plane and comprise both physical and virtual switches. ForCES models FEs by defining one or many Logical Functional Blocks (LFBs) classes, realized by an XML-based modeling language. An LFB comprises input and output ports and acts as a packet processing resource performing different functions, such as filtering, classification, and measurement. Multiple LFB instances in the same FE can be connected in a directed graph to create a network service.

Each LFB provides operational parameters, capabilities, and events to a CE that acts as an SDN Controller. CE uses the ForCES protocol as a Southbound API to perform the per-LFB controlling.

The works of Haleplidis et al. (2014) (Haleplidis et al., 2014; Haleplidis et al., 2014) proposed an NFV/SDN architecture using a ForCES standard. In this solution, the NFVI contains an LFB hypervisor that allows the creation of FE/LFBs as VNFs and CEs as EMS entities. The NFVI also provides LFBs acting as virtual switches to provide interconnection between these VNFs. In (Haleplidis et al., 2014), a PoC has been proposed to evaluate this architecture when applied for virtualization of 4G Evolved Packet Core (EPC) components.

9.6. SDN Northbound Interfaces

Regarding Northbound Interface (NBI), all studies that used this type of interface (see Table 13) chose to work with REST API (Chinnici et al., 2007). According to (Li et al., 2016), the REST API has become a prevalent choice for the NBI in SDN, because it is highly extensible and maintainable for managing services from both data and control planes. HTTP (Fielding et al., 1999) is usually adopted as a protocol for the communication between REST services (called web services). Procera (Voellmy et al., 2012) and Frenetic (Foster et al., 2011) would be alternatives for NBIs, but they run on top of a single OpenFlow Controller, and they are not so extensible as RESTful interfaces (Li et al., 2016).

SDN Northbound API Studies REST API (Munoz et al., 2015; Cerrato et al., 2015; Neves et al., 2016; Cziva et al., 2015; Soares et al., 2015; Cziva et al., 2015; Grønsund et al., 2015; Soares et al., 2014; Vilalta et al., 2015a, 2015, 2016a; Lucrezia et al., 2015; Shen et al., 2015; Muñoz et al., 2015; Vilalta et al., 2015b; Nguyen and Kim, 2014; Medhat et al., 2015; Batalle et al., 2013; Carella et al., 2015; Saridis et al., 2016; Telefonica et al., 2014; NTT et al., 2014; Telekom et al., 2014; AT&T et al., 2015; Italia et al., 2015; Telenor et al., 2016; Cziva and Pezaros, 2017; Vilalta et al., 2016b; Casellas et al., 2016; Vilalta et al., 2016; Mayoral et al., 2016; Martínez et al., 2017; Muñoz et al., 2017; Vilalta et al., 2016; Callegati et al., 2017; Neves et al., 2017)

Table 13. List of works using REST API as NBI.

RESTful interfaces are available in the main SDN Controllers, such as OpenDaylight, Floodlight, ONOS, and Ryu.

9.7. Vendor-specific Tools

It is also important to note that the Vendor-specific solutions are only used in some PoCs from ETSI NFV ISG, for all NFV components.

Some of the features used are: HP (Telecom et al., 2014; Unicom et al., 2015; Telenor et al., 2016; Telstra et al., 2016) and Huawei (DT et al., 2015) NFV Orchestrators (NFVO); ZTE (Unicom et al., 2015), Riverbed (DT et al., 2015), Samsung (Telecom et al., 2014), Telcoware (Telecom et al., 2014), HP (Telenor et al., 2016) and F5 (Telstra et al., 2016) VNF Managers (VNFM); and Samsung (Telecom et al., 2014), Telcoware (Telecom et al., 2014), HP (Unicom et al., 2015; Telstra et al., 2016) and ZTE (Unicom et al., 2015) Virtual Infrastructure Managers (VIM).

10. Lessons Learned

This section summarizes our view from the literature review described in the previous sections (Sections 5, 7, 8, and 9) and points out some trends for the design and implementation of NFV/SDN architectures.

Cloud Computing is the dominant scenario for implementing NFV/SDN solutions (72% of the studies found). According to (Mijumbi et al., 2016), Cloud Computing and Software-Defined Networking (SDN) are two concepts closely related to NFV. Most of the proposed NFV solutions have been implemented and tested in cloud-based environments. It has been the primary choice for the creation of NFV infrastructures (NFVI) mainly due to its flexibility, rapid deployment of new services, and inherent elasticity. The VNFs of a specific SFC are deployed as functions in dedicated Virtual Machines (VMs), which can be instantiated on devices placed in different geographic locations. Cloud Computing allows NFV/SDN solutions to provide better services for users by simplifying the provision of network services and enabling the quick deployment, management, and optimization of physical infrastructure dynamically, using resource virtualization mechanisms.

At the SDN-side, the SDN elements have been placed in different points of the NFV framework. It is clear that SDN Switches are most present in NFVI, whereas SDN Controllers are deployed in VIM and at the NFVI. Also, SDN Applications are usually placed in VIMs. The VIM is most used as a position for both SDN Controllers and Applications. The VIM seems to be the best place for these elements because it offers a global view of both NFVI physical and virtual infrastructures and the VNFs. It allows the implementation of different functionalities, such as VNFs or VNFCs interconnections, network connectivity in the NFVI, and the control of virtual networks.

Another important observation is that the most used elements in the VIM component are the OpenStack (Rackspace Cloud Computing, 2016) and the SDN Controller OpenDaylight (ODL) (Foundation, 2016) due to their soft integration. As described in Section 9, the Neutron service uses the Module Layer 2 (ML2) Plugin (Rackspace Cloud Computing, 2016) to provide networking services in a Cloud. The ML2 might control an ODL instance using the Neutron API, a REST API provided by ODL. However, the ONOS SDN Controller has gained space in both academia and industry and is currently the leading competitor of ODL. ONOS is the official distribution of the Open Network Foundation (ONF). Some industrial use case projects have been used ONOS, such as the Central Office Re-architected as a Datacenter (CORD), an NFV/SDN platform supported by major service providers (e.g., Google and Verizon).

Figure 22. Statistics related to the Problem Areas versus NFV Design.

The number of studies using OpenFlow 1.0 in NFV/SDN architectures has been steadily declining over the years (there is only one study in 2016) (Saridis et al., 2016). As expected, there is a clear trend to use the most current OpenFlow version in recent studies (AT&T et al., 2015; Italia et al., 2015; DT et al., 2015; Telenor et al., 2016; Schulz-Zander et al., 2017; Cziva and Pezaros, 2017). ForCES does not seem to have attracted the interest of the research community since only a few studies have used it as the southbound protocol (Haleplidis et al., 2014; Haleplidis et al., 2014).

In Section 7, the reader can observe that most solutions rely on both Distributed NFV and Multiple VIMs (see Figure 22), except for Wireless LAN and Wireless Mesh Networks. Particularly, 80% of the studies for Network Slicing have used this type of NFV design, mainly because they are deployed under multiple administrative domains (see Figure 24) including different types of network infrastructures (e.g., RAN, transport and core networks).

Figure 23. Statistics related to the Problem Areas versus SDN Controller Function.

As far as we are concerned to the SDN Controller Functions (see Figure 23), the primary focus for all areas was Interconnecting VNFs/VNFCs, which can be considered indispensable for any NFV/SDN architectures. As SDN can deliver intelligent traffic steering, service chains can indeed benefit from this integration. Also, although the Control of Virtual Networks function has been widely adopted in areas (e.g., Network Slicing with 75% of studies), we have not identified its implementation in other scenarios such as On-demand and Application-specific Traffic Steering, Dynamic Service Chaining, and vCPE. We argue that the absence of this functionality is not an impediment to the implementation of solutions for traffic steering and dynamic SFC and it should be a functional requirement for vCPE. Please recall that vCPE must provide multi-tenant services, leaving the client responsible for the selection and configuration of their VNFs.

As shown in Figure 24, the use of multiple SDN controllers has not been explored in Wireless LAN and Wireless Mesh Networks solutions, which may be a gap to be addressed for new NFV/SDN architectures with multiple controllers. Furthermore, we have identified a few studies dealing with Scalability (elasticity mechanisms) and Reliability (fault tolerance mechanisms) problems (see section 7 and subsection 8.3), which require orchestrators to manage environments with multiple VIMs and SDN controllers in order to provide support to perform D-NFV management in several administrative domains or NFVI-PoPs. According to the 5GPPP, such characteristics are essential for the 5G network, since novel 5G technologies (e.g., Network Slicing) might impact the entire mobile network including mobile devices; radio access, transport, and core networks; and the cloud (local, regional, or global).

Figure 24. Statistics related to the Problem Areas versus Objectives of Multiple SDN Controller.

Finally, the use of NFV/SDN architectures has been a growing trend for providing fast delivery of network services in a flexible and automated way for 5G networks. A certain NFV/SDN architecture creates an abstraction layer that unifies both computer and network resources and enables dynamic and application-specific traffic steering. Most studies have demonstrated a growing attention to the cost problem where they tried to reduce both CAPEX and OPEX costs. On the other hand, the use of NFV/SDN architectures for the Mobile Edge Computing (MEC) is incipient. Current solutions are still in their infancy, and better schemes are needed to provide distributed and dynamic SFC as well as to meet flow requirements. The problem of how to integrate MEC and network slicing in a unique NFV/SDN architecture has been slightly addressed (Akyildiz et al., 2015; Mwangama et al., 2015; Nguyen and Kim, 2014; Medhat et al., 2015; Neves et al., 2016).

11. Future Research Directions: Opportunities and Challenges

We have analyzed the selected studies and the surveys to have a clear view of the directions for future research efforts. We have identified some challenges in the design and implementation of NFV/SDN architectures. They are described in the following subsections.

11.1. Deployment of Network Services

Service Function Chaining (SFC) provides an ordered list for service processing of traffic flows (Li and Chen, 2015). The fast SFC deployment and provisioning of NFV and the centralized and flexible control of SDN have enabled new opportunities regarding this topic. However, solutions that provide better performance and optimal resource utilization in function deployment are still needed. Below, we describe some challenges related to NFV/SDN solutions regarding the deployment of network functions.

VNF Performance::

Virtualized network functions should meet the user’s performance requirements, especially when the SDN Controller is a VNF. Current hypervisors must be optimized for fast packet processing in standard servers as a way to obtain high I/O speed, short transmission delays, and so on. Some initiatives are DPDK (Foundation, 2016), NetVM (Hwang et al., 2015), and ClickOS (Martins et al., 2014). Further, the use of container-based virtualization, such as Docker containers, is of paramount importance in environments that require high-performance with low resource consumption such as edge devices (e.g., vCPEs, MEC, etc.) since they have relatively low capabilities compared to traditional NFV servers. However, there are some challenges to be considered when choosing the containers technologies for deploying VNFs in NFV/SDN architectures (Cziva and Pezaros, 2017; ETSI, 2016):

Orchestration: all containers share the same kernel as well as its services and configurations. This characteristic increases the complexity of orchestration and management platforms since the allocation process must take into account whether a VNF has unique needs in kernel, such as a particular module or configuration;
Security: container-based virtualization has a broader attack surface than other virtualization techniques since its interface is more sophisticated than hardware emulation interfaces. Besides, it provides weaker functional isolation among instances since containers may require different kernel configurations that can conflict. It also offers more vulnerable performance isolation since containers in the same host can be placed under resource pressure (e.g., memory or CPU overconsumption) by external attacks or new instances.

VNFs Scheduling and Placement::

The scheduling and placement of VNFs impact the performance of Service Chaining significantly. For better performance, the physical resources should be used efficiently. Also, energy-efficient hardware and energy-aware network service placement remain some of the main challenges in NFV and solutions are still limited (Mijumbi et al., 2016)

. Therefore, optimization and machine learning techniques are necessary to achieve optimal, automatic, and dynamic resource reservation, allocation, and migration of VNFs, considering a global view of the resources and the customer requirements. Integer programming and heuristic approaches can be used for VNFs Scheduling

(Shen et al., 2015) and Placement (Deng et al., 2015), considering resource constraints. Tools such as Google’s Borg (Verma et al., 2015), Omega (Schwarzkopf et al., 2013), and Apache Mesos (Hindman et al., 2011) may be considered for scheduling of VNFs.

High-level Policies::

The definition of high-level policies is necessary to simplify the configuration of NFV Orchestrator operations, such as resource allocation and optimization mechanisms, and to meet the customers’ requirements (interfaces to OSS/BSS). In this case, OpenStack’s HOT (Heat Orchestration Template) and TOSCA (Topology and Orchestration Specification for Cloud Applications) template languages could be used (Rackspace Cloud Computing, 2016).

Traffic Steering::

In NFV/SDN solutions, traffic steering and network function deployment should be optimized jointly, providing a network-aware scheduling mechanism (Cerrato et al., 2015; Lucrezia et al., 2015) that deploys VNFs considering both the paths expressed in the forwarding graph and the network behavior (available bandwidth, latency, jitter, etc.). As a consequence, more variables are introduced, and heuristic algorithms should be created to reduce computing complexity.

Elastic Network Function::

The dynamic service scaling at runtime provides better resource utilization, reducing both CAPEX and OPEX, and maintains service level requirements (Szabó et al., 2015). It is necessary that NFV/SDN solutions can scale (in/out or up/down) networking services and monitor both servers’ and networks’ resources to offer elastic, pay-as-used services.

Orchestration::

Orchestration services are necessary for elastic, adaptable, and autonomic network function deployment, provisioning, and management. Tools such as OpenMANO (Telefonica I+D, 2016) and OpenBaton (Fraunhofer FOKUS, 2016) might be used as a solution for NFV MANO (Management and Orchestration).

11.2. Improving the Programmability

SDN and NFV are the critical enablers for realizing some of the expected features in 5G networks, such as network programmability, flexibility (e.g., network abstraction, infrastructure sharing, and reconfigurability), adaptability (e.g., self-healing, self-configuration, self-protection, and self-optimization) and capabilities (e.g., network slicing and MEC) (Group, 2016a). However, some improvements should be provided so that the existing SDN standards such as OpenFlow can be applied in this type of scenario.

OpenFlow is the most used protocol for the Southbound API in NFV/SDN solutions, as described in Section 8. However, currently, it does not support application layer packet processing. The application layer inspection and classification is necessary to provide fine-grained flow distribution for different network services, and thus to provide intelligent service chaining.

Finally, OpenFlow is not suitable for Wireless Networks (e.g., WiFi, LTE, etc.) since flow tables just include rules for Ethernet-based switches. Wireless communication is more complex, as wireless links are time-varying and vulnerable to interference. For this, extensions must be implemented to allow WiFi programming rules, enabling the matching and monitoring of wireless frames (Schulz-Zander et al., 2015). Besides, SDN must provide support to Radio Access Network (RAN) virtualization infrastructures (Foukas et al., 2017). In this case, SDN approaches must support both legacies (e.g., 3G and 4G) and new radio access technologies (e.g., 5G and narrowband Internet of Things, NB-IoT), ensuring radio resource isolation.

11.3. Multi-Tenant, Multi-Service and Multi-Domain Support

An NFV/SDN architecture that supports multiple domains or NFVI-PoPs is necessary for the provision of quality of service (QoS) and SLA enforcement in multi-tenant environments with end-to-end services. However, it remains a challenge since the orchestration functions must support the following features (Group, 2016a):

Multi-domain orchestration of diverse programmable infrastructure technologies (e.g., RAN, transport and core networks, data centers, etc.), possibly belonging to different operators;
Northbound interface for Network Slicing management, providing multi-tenancy and multi-service support;
End-to-end network slices that are flexible to the dynamic requirements of different services (e.g., IoT, smart cities, etc.) and mobile operators, providing a multi-service and context-aware adaptation of network functions;
Advanced autonomic network management platforms to address complexity in such scenarios.

Furthermore, studies are still being carried out to evaluate the impact of end-to-end slices on the RAN design. RAN Virtualization is currently under investigation and is one of the major obstacles to creating NFV/SDN architectures for 5G networks (Foukas et al., 2017).

11.4. Multiple SDN Controllers

NFV/SDN solutions could be used for the control and management of heterogeneous network resources (Optical, MPLS, IP, etc.), distributed in different geographical locations. Therefore, hierarchical and federated SDN Controllers must be used to meet scalability, availability, reliability, and end-to-end (multi-domain) provisioning requirements. Tools such as FlowVisor as well as North/East/WestBound interfaces from popular SDN Controllers (OpenDayLight (Foundation, 2016) and ONOS ((ONF), 2017)) can be used to provide such solutions.

11.5. Security

In addition to current security problems that are unique to each technology (Mijumbi et al., 2016; Kreutz et al., 2015), the NFV/SDN solutions also have security challenges related to the integration process, such as the lack of authentication and authorization mechanisms in the communication interfaces between SDN and NFV modules. Besides, by exploring network programmability, security services should be developed to deal with malfunctioning software (e.g., detecting and preventing exploits) or attacks caused by malicious adversaries (e.g., intrusion detection and prevention systems) such as Distributed Denial of Service (DDoS) (Roman et al., 2016). Such services should take into account attack surfaces at all levels of the infrastructure, including network, edge and core data centers, virtualization, and user devices.

Furthermore, regarding 5G networks, security in network slicing is a complex task since there is resource sharing among slices and they may have different security policy requirements. This problem gets worse when we consider multi-domain scenarios. In this context, security solutions in the NFV/SDN architecture should provide mechanisms for resource isolation between slices, considering their impact on the entire infrastructure and providing security policy coordination among different domain infrastructures (Li et al., 2017).

11.6. Extensibility and the Expressiveness of NFV/SDN Models

It is important to use a single model (framework) to address both NFV and SDN issues, instead of a combination that focuses on one problem at a time. This type of model eases implementation and the learning curve as well as reduces interdependencies (plugins to interconnect different frameworks) (Haleplidis et al., 2014).

11.7. Standardization

Even with the existence of reference architectures defined by industry (Verizon, 2016), an effort should be made towards standardization of an architecture that integrates the NFV and SDN technologies to simplify the work of researchers when providing new NFV/SDN solutions. This reference architecture must include standardized interfaces and resource catalogs (Shen et al., 2015) so that new VNFs can be rapidly integrated and deployed into the system.

12. Threats to Validity

This section describes several threats to the validity of this study, to evaluate the quality of this research. The potential validity threats to our study and the strategies for overcoming them are listed below:

Error-prone analysis::: The process of selection and extraction of studies were carried out by only one researcher (M. Bonfim), which may lead to subjectivity in the selection and inconsistencies in data extraction of the articles. To mitigate this threat, we have created a well-defined and extensive process to the studies selection and data extraction through the SLR protocol. Besides, two experienced researchers (K. Dias and S. Fernandes) checked and validated all the stages of the SLR (protocol, identification, and execution) and the summarization of results, to ensure the robustness and expressiveness of this work;
Data sources::: The primary studies were obtained from different web search engines and comprised of only academic studies. Even considering the Proof of Concepts of ETSI NFV ISG, this work is limited largely to academic expertise. It is important to conduct further research using industry data sources, such as the websites of companies, given that large companies (e.g., CISCO, Verizon, Juniper, HP, etc.) have invested heavily in NFV/SDN solutions. Furthermore, it is noteworthy that the search was done in April 2016, and the authors subsequently added some other bibliographies published after this period (until 2017), to provide a level of updating that allowed them to delimit some trends. Therefore, the volume of articles published in 2016 and 2017 may not reflect the current state nor indicate any trend for decrease in the interest of the research community;
Meta-analysis::: The large number of selected studies in this SLR leads to a large variation in the reporting of NFV/SDN solutions regarding architecture and experimental techniques and dataset. This scenario has made our synthesis of the data largely qualitative because it has not been possible to carry out a meta-analysis to strengthen the differences among relevant studies. Conducting a statistical analysis of the data extracted from selected studies will need to be undertaken in the future.

13. Conclusion

Even with different purposes, NFV and SDN are complementary paradigms and technologies capable of providing one consolidated solution that offers the best of both technologies. NFV/SDN architectures are of paramount importance for a passage from the static design of conventional networks to an intelligent, open network environment. Therefore, this work proposed a Systematic Literature Review (SLR) for NFV/SDN architectures, intending to provide a profound understanding of such integrated designs. We aimed to identify the current trend in this field. For this, a total of 74 articles have been studied in-depth according to our predefined SLR protocol. Through comprehensive analysis and interpretation of the collected data, this SLR achieved three goals. First, we described the main characteristics (target environment and problems to solve) of integrated NFV/SDN solutions practices. Second, we compared their architecture designs (NFV framework design and tools, SDN APIs and place of SDN elements) and classified them against the presented taxonomy. Then, we discussed some opportunities and challenges for research work in the next generation of NFV/SDN architectures.

References

(1)
ETSI (2012) ETSI. 2012. Network Functions Virtualisation - An Introduction, Benefits, Enablers, Challenges and Call for Action. White Paper (Out. 2012).
ETSI (2015) ETSI. 2015. Network Functions Virtualisation (NFV) - Network Operator Perspectives on Industry Progress. White Paper (Jan. 2015).
Szabó et al. (2015) Róbert Szabó, Mario Kind, Fritz-joachim Westphal, Hagen Woesner, Dávid Jocha, and András Császar. 2015. Elastic Network Functions: Opportunities and Challenges. IEEE Network June (2015), 15–21. DOI:http://dx.doi.org/10.1109/MNET.2015.7113220
McKeown et al. (2008) Nick McKeown, Tom Anderson, Hari Balakrishnan, Guru Parulkar, Larry Peterson, Jennifer Rexford, Scott Shenker, and Jonathan Turner. 2008. OpenFlow: Enabling Innovation in Campus Networks. SIGCOMM Comput. Commun. Rev. 38, 2 (March 2008), 69–74. DOI:http://dx.doi.org/10.1145/1355734.1355746
Kreutz et al. (2015) D. Kreutz, F. M. V. Ramos, P. E. Veríssimo, C. E. Rothenberg, S. Azodolmolky, and S. Uhlig. 2015. Software-Defined Networking: A Comprehensive Survey. Proc. IEEE 103, 1 (Jan 2015), 14–76. DOI:http://dx.doi.org/10.1109/JPROC.2014.2371999
Mijumbi et al. (2016) R. Mijumbi, J. Serrat, J. L. Gorricho, N. Bouten, F. De Turck, and R. Boutaba. 2016. Network Function Virtualization: State-of-the-Art and Research Challenges. IEEE Communications Surveys Tutorials 18, 1 (Firstquarter 2016), 236–262. DOI:http://dx.doi.org/10.1109/COMST.2015.2477041
Batista et al. (2015) Daniel M. Batista, Gordon Blair, Fabio Kon, Raouf Boutaba, David Hutchison, Raj Jain, Ramachandran Ramjee, and Christian Esteve Rothenberg. 2015. Perspectives on software-defined networks: interviews with five leading scientists from the networking community. Journal of Internet Services and Applications 6, 1 (2015), 1–10. DOI:http://dx.doi.org/10.1186/s13174-015-0035-3
d. J. Gil Herrera and Vega (2016) J. d. J. Gil Herrera and J. F. Botero Vega. 2016. Network Functions Virtualization: A Survey. IEEE Latin America Transactions 14, 2 (Feb 2016), 983–997. DOI:http://dx.doi.org/10.1109/TLA.2016.7437249
Li and Chen (2015) Y. Li and M. Chen. 2015. Software-Defined Network Function Virtualization: A Survey. IEEE Access 3 (2015), 2542–2553. DOI:http://dx.doi.org/10.1109/ACCESS.2015.2499271
López et al. (2015) L. I. Barona López, Á L. Valdivieso Caraguay, L. J. García Villalba, and D. López. 2015. Trends on virtualisation with software defined networking and network function virtualisation. IET Networks 4, 5 (2015), 255–263. DOI:http://dx.doi.org/10.1049/iet-net.2014.0117
Kitchenham et al. (2009) Barbara Kitchenham, O. Pearl Brereton, David Budgen, Mark Turner, John Bailey, and Stephen Linkman. 2009. Systematic literature reviews in software engineering – A systematic literature review. Information and Software Technology 51, 1 (2009), 7 – 15. DOI:http://dx.doi.org/10.1016/j.infsof.2008.09.009 Special Section - Most Cited Articles in 2002 and Regular Research Papers.
Kitchenham (2004) Barbara Kitchenham. 2004. Procedures for performing systematic reviews. Keele, UK, Keele University 33, 2004 (2004), 1–26.
ETSI (2014) ETSI. 2014. Network Functions Virtualisation (NFV) - Architectural Framework. ETSI GS NFV 002 V1.2.1 (Dez. 2014). http://www.etsi.org/deliver/etsi_gs/nfv/001_099/002/01.02.01_60/gs_nfv002v010201p.pdf
ONF (2015) ONF. 2015. OpenFlow Switch Specification - Version 1.3.5. (Mar. 2015). https://3vf60mmveq1g8vzn48q2o71a-wpengine.netdna-ssl.com/wp-content/uploads/2014/10/openflow-switch-v1.3.5.pdf.
ETSI (2015) ETSI. 2015. Network Functions Virtualisation (NFV) - Ecosystem - Report on SDN Usage in NFV Architectural Framework. ETSI GS NFV-EVE 005 V1.1.1 (Dec. 2015). http://www.etsi.org/deliver/etsi_gs/NFV-EVE/001_099/005/01.01.01_60/gs_nfv-eve005v010101p.pdf
Telefonica et al. (2014) Telefonica, Sprint, 6WIND, Dell, EnterpriseWeb, Mellanox, Metaswitch, Overture Networks, Qosmos, and Aeroflex. 2014. PoC#1 - CloudNFV Open NFV Framework Project. Technical Report. ETSI - the European Telecommunications Standards Institute. http://docbox.etsi.org/ISG/NFV/Closed_WGs/PER/05-CONTRIBUTIONS/2014/NFVPER(14)000064_CloudNFV_Open_NFV_Framework_POC_1_Interim_Report.docx
NTT et al. (2014) NTT, Cisco, HP, and Juniper Networks. 2014. PoC#2 - Service Chaining for NW Function Selection in Carrier Networks. Technical Report. ETSI - the European Telecommunications Standards Institute. http://nfvwiki.etsi.org/images/NFVPER%2814%29000004r2_NFV_ISG_PoC_Proposal_Service_Chaining_for_NW_Function_Select.pdf
Telekom et al. (2014) Deutsche Telekom, Ericsson, x-ion GmbH, and Deutsche Telekom Innovation Laboratories. 2014. PoC#8 - Automated Network Orchestration. Technical Report. ETSI - the European Telecommunications Standards Institute. http://docbox.etsi.org/ISG/NFV/Closed_WGs/PER/05-CONTRIBUTIONS/2014//NFVPER(14)000080__NFV_ISG_PoC_Report__Automated_Network_Orchestration.docx
Telefonica et al. (2016) Telefonica, Vodafone, Radware, HP, and Melanox. 2016. PoC#13 - SteerFlow: Multi-Layered Traffic Steering for Gi-LAN. Technical Report. ETSI - the European Telecommunications Standards Institute. https://docbox.etsi.org/ISG/NFV/TST/05-CONTRIBUTIONS/2016//NFVTST(16)000019_Final_Report_for_PoC_13__Multi-Layered_Traffic_Steering_for_.docx
AT&T et al. (2015) AT&T, Telecom Italia, Netronome, Intel, ServiceMesh, PLUMgrid, and Cisco Systems. 2015. PoC#16 - NFVIaaS with Secure, SDN-controlled WAN Gateway. Technical Report. ETSI - the European Telecommunications Standards Institute. http://docbox.etsi.org/ISG/NFV/Closed_WGs/PER/05-CONTRIBUTIONS/2014/NFVPER%2814%29000108_PoC16_Report-NFVIaaS_with_Secure__SDN-controlled_WAN_Gateway.doc
BT et al. (2015) BT, Huawei, EZChip, AMD, Tilera, Altera, Broadcom, EANTC, and Ixia. 2015. PoC#21 - Network Intensive and Compute Intensive Hardware Acceleration. Technical Report. ETSI - the European Telecommunications Standards Institute. https://docbox.etsi.org/ISG/NFV/TST/05-CONTRIBUTIONS/2015//NFVTST(15)000111r1_PoC_21__Network_Intensive_and_Compute_Intensive_Hardware_Acc.docx
Telecom et al. (2014) SK Telecom, Hewlett Packard, Samsung, and Telcoware. 2014. PoC#23 - E2E orchestration of virtualized LTE core-network functions and SDN-based dynamic service chaining of VNFs using VNF FG. Technical Report. ETSI - the European Telecommunications Standards Institute. http://docbox.etsi.org/ISG/NFV/Closed_WGs/PER/05-CONTRIBUTIONS/2014/NFVPER(14)000113_Proof-of-Concept__23_Final_Report.docx
Italia et al. (2015) Telecom Italia, Nokia Networks, EXFO, Coriant, and Aalto University. 2015. PoC#26 - Virtual EPC with SDN Function in Mobile Backhaul Networks. Technical Report. ETSI - the European Telecommunications Standards Institute. http://docbox.etsi.org/ISG/NFV/TST/05-CONTRIBUTIONS/2015/NFVTST(15)000116r3_PoC_26__final_report.docx
Unicom et al. (2015) China Unicom, ZTE Corporation, and Hewlett-Packard. 2015. PoC#27 - VoLTE Service based on vEPC and vIMS Architecture. Technical Report. ETSI - the European Telecommunications Standards Institute. http://nfvwiki.etsi.org/images/NFVTST%2815%29000137_Final_Report_PoC_27_VoLTE_Service_based_on_vEPC_and_vIMS_Arc.pdf
DT et al. (2015) DT, Vodafone, Huawei, Freescale, Qosmos, Netronome, MRV, Corsa, Riverbed, BlueCoat, Ixia, and ONF. 2015. PoC#28 - SDN Controlled VNF Forwarding Graph. Technical Report. ETSI - the European Telecommunications Standards Institute. http://nfvwiki.etsi.org/images/PoC_28_report.pdf
Telenor et al. (2016) Telenor, Vodafone, Hewlett Packard Enterprise, ImVision Tech, Mavenir, Redhat, and Altiostar. 2016. PoC#34 - SDN Enabled Virtual EPC Gateway. Technical Report. ETSI - the European Telecommunications Standards Institute. http://nfvwiki.etsi.org/images/NFVTST%2815%29000006_NFV_ISG_PoC_Proposal_SDN_Enabled_EPC_Gwy_r2_was_PER114.pdf
Telstra et al. (2016) Telstra, Hewlett-Packard, Alcatel Lucent, and F5 Networks. 2016. PoC#38 - Full ISO 7-layer stack fulfilment, activation and orchestration of VNFs in carrier networks. Technical Report. ETSI - the European Telecommunications Standards Institute. http://nfvwiki.etsi.org/images/NFV_ISG_PoC38_Report.pdf
Ding et al. (2015) W. Ding, W. Qi, J. Wang, and B. Chen. 2015. OpenSCaaS: an open service chain as a service platform toward the integration of SDN and NFV. IEEE Network 29, 3 (2015), 30–35. DOI:http://dx.doi.org/10.1109/MNET.2015.7113222
ETSI (2014) ETSI. 2014. Network Functions Virtualisation (NFV) - Management and Orchestration. ETSI GS NFV-MAN 001 V1.1.1 (Dez. 2014). http://www.etsi.org/deliver/etsi_gs/NFV-MAN/001_099/001/01.01.01_60/gs_NFV-MAN001v010101p.pdf
Császár et al. (2013) A. Császár, W. John, M. Kind, C. Meirosu, G. Pongrácz, D. Staessens, A. Takács, and F. J. Westphal. 2013. Unifying Cloud and Carrier Network: EU FP7 Project UNIFY. In 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing. 452–457. DOI:http://dx.doi.org/10.1109/UCC.2013.89
Sköldström et al. (2014) P. Sköldström, B. Sonkoly, A. Gulyás, F. Németh, M. Kind, F. J. Westphal, W. John, J. Garay, E. Jacob, D. Jocha, J. Elek, R. Szabó, W. Tavernier, G. Agapiou, A. Manzalini, M. Rost, N. Sarrar, and S. Schmid. 2014. Towards Unified Programmability of Cloud and Carrier Infrastructure. In 2014 Third European Workshop on Software Defined Networks. 55–60. DOI:http://dx.doi.org/10.1109/EWSDN.2014.18
T-NOVA Project (2015) T-NOVA Project. 2015. T-NOVA: Network Functions-as-a-Service (NFaaS) over Virtualized Infrastructures. (2015). http://www.t-nova.eu, Accessed: 2016-07-25.
Rackspace Cloud Computing (2016) Rackspace Cloud Computing. 2016. OpenStack Open Source Cloud Computing Software. https://www.openstack.org/. (2016). Accessed: 2016-07-25.
Foundation (2016) Linux Foundation. 2016. The OpenDaylight Plataform. http://www.opendaylight.org. (2016). Accessed: 2016-07-25.
Sonkoly et al. (2015) B. Sonkoly, R. Szabo, D. Jocha, J. Czentye, M. Kind, and F. J. Westphal. 2015. UNIFYing Cloud and Carrier Network Resources: An Architectural View. In 2015 IEEE Global Communications Conference (GLOBECOM). 1–7. DOI:http://dx.doi.org/10.1109/GLOCOM.2015.7417869
Carella et al. (2015) G. Carella, J. Yamada, N. Blum, C. Lück, N. Kanamaru, N. Uchida, and T. Magedanz. 2015. Cross-layer service to network orchestration. In 2015 IEEE International Conference on Communications (ICC). 6829–6835. DOI:http://dx.doi.org/10.1109/ICC.2015.7249414
Fraunhofer FOKUS (2016) Fraunhofer FOKUS. 2016. OpenSDNCore - Reasearch and testbed for the carrier-grade nfv/sdn environment. http://www.opensdncore.org/. (2016). Accessed: 2016-07-25.
Sherry et al. (2012) Justine Sherry, Shaddi Hasan, Colin Scott, Arvind Krishnamurthy, Sylvia Ratnasamy, and Vyas Sekar. 2012. Making Middleboxes Someone else’s Problem: Network Processing As a Cloud Service. In Proceedings of the ACM SIGCOMM 2012 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM ’12). ACM, New York, NY, USA, 13–24. DOI:http://dx.doi.org/10.1145/2342356.2342359
Cziva et al. (2015) R. Cziva, S. Jouet, K. J. S. White, and D. P. Pezaros. 2015. Container-based network function virtualization for software-defined networks. In 2015 IEEE Symposium on Computers and Communication (ISCC). 415–420. DOI:http://dx.doi.org/10.1109/ISCC.2015.7405550
Batalle et al. (2013) J. Batalle, J. Ferrer Riera, E. Escalona, and J. A. Garcia-Espin. 2013. On the Implementation of NFV over an OpenFlow Infrastructure: Routing Function Virtualization. In Future Networks and Services (SDN4FNS), 2013 IEEE SDN for. 1–6. DOI:http://dx.doi.org/10.1109/SDN4FNS.2013.6702546
Schulz-Zander et al. (2015) Julius Schulz-Zander, Carlos Mayer, Bogdan Ciobotaru, Stefan Schmid, and Anja Feldmann. 2015. OpenSDWN: Programmatic Control over Home and Enterprise WiFi. In Proceedings of the 1st ACM SIGCOMM Symposium on Software Defined Networking Research (SOSR ’15). ACM, New York, NY, USA, 16:1–16:12. DOI:http://dx.doi.org/10.1145/2774993.2775002
Lin et al. (2015) Y. D. Lin, P. C. Lin, C. H. Yeh, Y. C. Wang, and Y. C. Lai. 2015. An extended SDN architecture for network function virtualization with a case study on intrusion prevention. IEEE Network 29, 3 (2015), 48–53. DOI:http://dx.doi.org/10.1109/MNET.2015.7113225
Deng et al. (2015) J. Deng, H. Hu, H. Li, Z. Pan, K. C. Wang, G. J. Ahn, J. Bi, and Y. Park. 2015. VNGuard: An NFV/SDN combination framework for provisioning and managing virtual firewalls. In 2015 IEEE Conference on Network Function Virtualization and Software Defined Network (NFV-SDN). 107–114. DOI:http://dx.doi.org/10.1109/NFV-SDN.2015.7387414
Cziva et al. (2015) R. Cziva, S. Jouet, and D. P. Pezaros. 2015. GNFC: Towards network function cloudification. In 2015 IEEE Conference on Network Function Virtualization and Software Defined Network (NFV-SDN). 142–148. DOI:http://dx.doi.org/10.1109/NFV-SDN.2015.7387419
Rossem et al. (2015) S. Van Rossem, W. Tavernier, B. Sonkoly, D. Colle, J. Czentye, M. Pickavet, and P. Demeester. 2015. Deploying elastic routing capability in an SDN/NFV-enabled environment. In 2015 IEEE Conference on Network Function Virtualization and Software Defined Network (NFV-SDN). 22–24. DOI:http://dx.doi.org/10.1109/NFV-SDN.2015.7387398
Callegati et al. (2015) F. Callegati, W. Cerroni, C. Contoli, and G. Santandrea. 2015. Implementing dynamic chaining of Virtual Network Functions in OpenStack platform. In 2015 17th International Conference on Transparent Optical Networks (ICTON). 1–4. DOI:http://dx.doi.org/10.1109/ICTON.2015.7193561
Cziva and Pezaros (2017) R. Cziva and D. P. Pezaros. 2017. Container Network Functions: Bringing NFV to the Network Edge. IEEE Communications Magazine 55, 6 (2017), 24–31. DOI:http://dx.doi.org/10.1109/MCOM.2017.1601039
Inc. (2016) Docker Inc. 2016. Docker Documentation. https://docs.docker.com/. (2016). Accessed: 2016-07-25.
NOXRepo.org (2016) NOXRepo.org. 2016. The POX Controller. http://www.noxrepo.org/pox/about-pox/. (2016). Accessed: 2016-07-25.
ETSI (2013) ETSI. 2013. Network Functions Virtualisation (NFV) - Use Cases. ETSI GS NFV 001 V1.1.1 (Out. 2013). http://www.etsi.org/deliver/etsi_gs/nfv/001_099/001/01.01.01_60/gs_nfv001v010101p.pdf
Cerrato et al. (2015) Ivano Cerrato, Alex Palesandro, Fulvio Risso, Marc Suñé, Vinicio Vercellone, and Hagen Woesner. 2015. Toward dynamic virtualized network services in telecom operator networks. Computer Networks 92, Part 2 (2015), 380–395. DOI:http://dx.doi.org/10.1016/j.comnet.2015.09.028
Gittik (2014) Yuri Gittik. 2014. White Paper - Distributed Network Functions Virtualization (RAD). (Mar. 2014). http://www.rad.com/template.MEDIA
Soares et al. (2014) J. Soares, M. Dias, J. Carapinha, B. Parreira, and S. Sargento. 2014. Cloud4NFV: A platform for Virtual Network Functions. In 2014 IEEE 3rd International Conference on Cloud Networking (CloudNet). 288–293. DOI:http://dx.doi.org/10.1109/CloudNet.2014.6969010
Soares et al. (2015) J. Soares, C. Gonçalves, B. Parreira, P. Tavares, J. Carapinha, J. P. Barraca, R. L. Aguiar, and S. Sargento. 2015. Toward a telco cloud environment for service functions. IEEE Communications Magazine 53, 2 (2015), 98–106. DOI:http://dx.doi.org/10.1109/MCOM.2015.7045397
Schulz-Zander et al. (2017) J. Schulz-Zander, C. Mayer, B. Ciobotaru, S. Schmid, and A. Feldmann. 2017. Unified Programmability of Virtualized Network Functions and Software-Defined Wireless Networks. IEEE Transactions on Network and Service Management PP, 99 (2017), 1–1. DOI:http://dx.doi.org/10.1109/TNSM.2017.2744807
Suresh et al. (2012) Lalith Suresh, Julius Schulz-Zander, Ruben Merz, Anja Feldmann, and Teresa Vazao. 2012. Towards Programmable Enterprise WLANS with Odin. In Proceedings of the First Workshop on Hot Topics in Software Defined Networks (HotSDN ’12). ACM, New York, NY, USA, 115–120. DOI:http://dx.doi.org/10.1145/2342441.2342465
Vestin and Kassler (2015) J. Vestin and A. Kassler. 2015. QoS enabled WiFi MAC layer processing as an example of a NFV service. In 2015 1st IEEE Conference on Network Softwarization (NetSoft). 1–9. DOI:http://dx.doi.org/10.1109/NETSOFT.2015.7116164
Dely et al. (2012) P. Dely, J. Vestin, A. Kassler, N. Bayer, H. Einsiedler, and C. Peylo. 2012. CloudMAC: An OpenFlow based architecture for 802.11 MAC layer processing in the cloud. In 2012 IEEE Globecom Workshops. 186–191. DOI:http://dx.doi.org/10.1109/GLOCOMW.2012.6477567
Kim (2015) Wooseong Kim. 2015. Toward network function virtualization for cognitive wireless mesh networks: a TCP case study. EURASIP Journal on Wireless Communications and Networking 2015, 1 (Oct. 2015), 1–16. DOI:http://dx.doi.org/10.1186/s13638-015-0450-y
Paper (2017) Cisco White Paper. 2017. Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2016-2021. (March 2017). http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/mobile-white-paper-c11-520862.pdf
Gupta and Jha (2015) A. Gupta and R. K. Jha. 2015. A Survey of 5G Network: Architecture and Emerging Technologies. IEEE Access 3 (2015), 1206–1232. DOI:http://dx.doi.org/10.1109/ACCESS.2015.2461602
itu (2017) 2017. ITU towards “IMT for 2020 and beyond”. (2017). http://www.itu.int/en/ITU-R/study-groups/rsg5/rwp5d/imt-2020/Pages/default.aspx, Accessed: 2017-09-28.
5gt (2017) 2017. Verizon 5G Technical Forum. (2017). http://www.5gtf.org/, Accessed: 2017-09-28.
5gp (2017) 2017. 5G Infrastructure Public Private Partnership–5G PPP. (2017). https://5g-ppp.eu/, Accessed: 2017-09-28.
Group (2016a) 5G PPP Architecture Working Group. 2016a. View on 5G Architecture. Technical Report. https://5g-ppp.eu/wp-content/uploads/2014/02/5G-PPP-5G-Architecture-WP-July-2016.pdf
Group (2016b) 5G PPP Architecture Working Group. 2016b. 5G PPP use cases and performance evaluation models. Technical Report. https://5g-ppp.eu/wp-content/uploads/2014/02/5G-PPP-use-cases-and-performance-evaluation-modeling_v1.0.pdf
Osseiran et al. (2014) A. Osseiran, F. Boccardi, V. Braun, K. Kusume, P. Marsch, M. Maternia, O. Queseth, M. Schellmann, H. Schotten, H. Taoka, H. Tullberg, M. A. Uusitalo, B. Timus, and M. Fallgren. 2014. Scenarios for 5G mobile and wireless communications: the vision of the METIS project. IEEE Communications Magazine 52, 5 (May 2014), 26–35. DOI:http://dx.doi.org/10.1109/MCOM.2014.6815890
Technologies (2015) Huawei Technologies. 2015. 5G Network Architecture-A High Level View. Technical Report. http://carrier.huawei.com/~/media/CNBG/Downloads/Program/5g_nework_architecture_whitepaper_en.pdf
Group (2017) 5G PPP Architecture Working Group. 2017. Vision on Software Networks and 5G. Technical Report. https://5g-ppp.eu/wp-content/uploads/2014/02/5G-PPP_SoftNets_WG_whitepaper_v20.pdf
Kyung et al. (2015) Y. Kyung, T. M. Nguyen, K. Hong, J. Park, and J. Park. 2015. Software defined service migration through legacy service integration into 4G networks and future evolutions. IEEE Communications Magazine 53, 9 (Sept. 2015), 108–114. DOI:http://dx.doi.org/10.1109/MCOM.2015.7263353
Haleplidis et al. (2014) E. Haleplidis, D. Joachimpillai, J. H. Salim, D. Lopez, J. Martin, K. Pentikousis, S. Denazis, and O. Koufopavlou. 2014. ForCES Applicability to SDN-Enhanced NFV. In 2014 Third European Workshop on Software Defined Networks. 43–48. DOI:http://dx.doi.org/10.1109/EWSDN.2014.27
Basta et al. (2014) Arsany Basta, Andreas Blenk, Marco Hoffmann, Hans Jochen Morper, Klaus Hoffmann, and Wolfgang Kellerer. 2014. SDN and NFV Dynamic Operation of LTE EPC Gateways for Time-Varying Traffic Patterns. In Mobile Networks and Management, Ramón Agüero, Thomas Zinner, Rossitza Goleva, Andreas Timm-Giel, and Phuoc Tran-Gia (Eds.). Number 141 in Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. Springer International Publishing, 63–76. http://link.springer.com/chapter/10.1007/978-3-319-16292-8_5 DOI: 10.1007/978-3-319-16292-8_5.
Nguyen and Kim (2014) V. G. Nguyen and Y. H. Kim. 2014. Slicing the next mobile packet core network. In 2014 11th International Symposium on Wireless Communications Systems (ISWCS). 901–904. DOI:http://dx.doi.org/10.1109/ISWCS.2014.6933481
Costa-Requena et al. (2015) J. Costa-Requena, J. L. Santos, V. F. Guasch, K. Ahokas, G. Premsankar, S. Luukkainen, O. L. Pérez, M. U. Itzazelaia, I. Ahmad, M. Liyanage, M. Ylianttila, and E. M. de Oca. 2015. SDN and NFV integration in generalized mobile network architecture. In 2015 European Conference on Networks and Communications (EuCNC). 154–158. DOI:http://dx.doi.org/10.1109/EuCNC.2015.7194059
Medhat et al. (2015) A. M. Medhat, G. Carella, J. Mwangama, and N. Ventura. 2015. Multi-tenancy for Virtualized Network Functions. In 2015 1st IEEE Conference on Network Softwarization (NetSoft). 1–6. DOI:http://dx.doi.org/10.1109/NETSOFT.2015.7116177
Ahmad et al. (2016) I. Ahmad, M. Liyanage, S. Namal, M. Ylianttila, A. Gurtov, M. Eckert, T. Bauschert, Z. Faigl, L. Bokor, E. Saygun, O. L. Akyildiz, H. A. and, M. U. Itzazelaia, B. Ozbek, and A. Ulas. 2016. New concepts for traffic, resource and mobility management in software-defined mobile networks. In 2016 12th Annual Conference on Wireless On-demand Network Systems and Services (WONS). 1–8.
Tawbeh et al. (2017) A. Tawbeh, H. Safa, and A. R. Dhaini. 2017. A hybrid SDN/NFV architecture for future LTE networks. In 2017 IEEE International Conference on Communications (ICC). 1–6. DOI:http://dx.doi.org/10.1109/ICC.2017.7997391
An et al. (2014) X. An, W. Kiess, and D. Perez-Caparros. 2014. Virtualization of cellular network EPC gateways based on a scalable SDN architecture. In 2014 IEEE Global Communications Conference. 2295–2301. DOI:http://dx.doi.org/10.1109/GLOCOM.2014.7037150
Grønsund et al. (2015) P. Grønsund, K. Mahmood, G. Millstein, A. Noy, G. Solomon, and A. Sahai. 2015. A solution for SGi-LAN services virtualization using NFV and SDN. In 2015 European Conference on Networks and Communications (EuCNC). 408–412. DOI:http://dx.doi.org/10.1109/EuCNC.2015.7194108
Ordonez-Lucena et al. (2017) J. Ordonez-Lucena, P. Ameigeiras, D. Lopez, J. J. Ramos-Munoz, J. Lorca, and J. Folgueira. 2017. Network Slicing for 5G with SDN/NFV: Concepts, Architectures, and Challenges. IEEE Communications Magazine 55, 5 (May 2017), 80–87. DOI:http://dx.doi.org/10.1109/MCOM.2017.1600935
Munoz et al. (2015) R. Munoz, R. Vilalta, R. Casellas, R. Martinez, T. Szyrkowiec, A. Autenrieth, V. Lopez, and D. Lopez. 2015. Integrated SDN/NFV management and orchestration architecture for dynamic deployment of virtual SDN control instances for virtual tenant networks [invited]. IEEE/OSA Journal of Optical Communications and Networking 7, 11 (Nov. 2015), B62–B70. DOI:http://dx.doi.org/10.1364/JOCN.7.000B62
Muñoz et al. (2015) R. Muñoz, R. Vilalta, R. Casellas, R. Martínez, T. Szyrkowiec, A. Autenrieth, V. López, and D. López. 2015. SDN/NFV orchestration for dynamic deployment of virtual SDN controllers as VNF for multi-tenant optical networks. In Optical Fiber Communications Conference and Exhibition (OFC), 2015. 1–3.
Vilalta et al. (2015a) R. Vilalta, A. Mayoral, R. Muñoz, R. Casellas, and R. Martínez. 2015a. The SDN/NFV Cloud Computing platform and transport network of the ADRENALINE testbed. In 2015 1st IEEE Conference on Network Softwarization (NetSoft). 1–5. DOI:http://dx.doi.org/10.1109/NETSOFT.2015.7116150
Vilalta et al. (2015b) R. Vilalta, A. Mayoral, R. Muñoz, R. Casellas, and R. Martínez. 2015b. Multi-tenant transport networks with SDN/NFV. In 2015 European Conference on Optical Communication (ECOC). 1–3. DOI:http://dx.doi.org/10.1109/ECOC.2015.7341931
Vilalta et al. (2016a) R. Vilalta, A. Mayoral, R. Muñoz, R. Casellas, and R. Martínez. 2016a. Multitenant Transport Networks With SDN/NFV. Journal of Lightwave Technology 34, 6 (March 2016), 1509–1515. DOI:http://dx.doi.org/10.1109/JLT.2015.2508044
Vilalta et al. (2016b) R. Vilalta, A. Mayoral, V. Lopez, V. Uceda, R. Casellas, R. Martinez, R. Munoz, A. Aguado, J. Marhuenda, R. Nejabati, D. Simeonidou, N. Yoshikane, T. Tsuritani, I. Morita, T. Szyrkowiec, and A. Autenrieth. 2016b. Peer SDN Orchestration: End-to-End Connectivity Service Provisioning Through Multiple Administrative Domains. In ECOC 2016; 42nd European Conference on Optical Communication. 1–3.
Akyildiz et al. (2015) Ian F. Akyildiz, Shih-Chun Lin, and Pu Wang. 2015. Wireless software-defined networks (W-SDNs) and network function virtualization (NFV) for 5G cellular systems: An overview and qualitative evaluation. Computer Networks 93, Part 1 (2015), 66–79. DOI:http://dx.doi.org/10.1016/j.comnet.2015.10.013
Mwangama et al. (2015) J. Mwangama, N. Ventura, A. Willner, Y. Al-Hazmi, G. Carella, and T. Magedanz. 2015. Towards Mobile Federated Network Operators. In 2015 1st IEEE Conference on Network Softwarization (NetSoft). 1–6. DOI:http://dx.doi.org/10.1109/NETSOFT.2015.7116187
Casellas et al. (2016) R. Casellas, R. Muñoz, R. Vilalta, and R. Martínez. 2016. Orchestration of IT/cloud and networks: From Inter-DC interconnection to SDN/NFV 5G services. In 2016 International Conference on Optical Network Design and Modeling (ONDM). 1–6. DOI:http://dx.doi.org/10.1109/ONDM.2016.7494060
Vilalta et al. (2016) R. Vilalta, A. Mayoral, R. Casellas, R. Martínez, and R. Muñoz. 2016. SDN/NFV orchestration of multi-technology and multi-domain networks in cloud/fog architectures for 5g services. In 2016 21st OptoElectronics and Communications Conference (OECC) held jointly with 2016 International Conference on Photonics in Switching (PS). 1–3.
Mayoral et al. (2016) A. Mayoral, R. Vilalta, R. Casellas, R. Martinez, and R. Munoz. 2016. Multi-tenant 5G Network Slicing Architecture with Dynamic Deployment of Virtualized Tenant Management and Orchestration (MANO) Instances. In ECOC 2016; 42nd European Conference on Optical Communication. 1–3.
Martínez et al. (2017) R. Martínez, A. Mayoral, R. Vilalta, R. Casellas, R. Muñoz, S. Pachnicke, T. Szyrkowiec, and A. Autenrieth. 2017. Integrated SDN/NFV orchestration for the dynamic deployment of mobile virtual backhaul networks over a multilayer (packet/optical) aggregation infrastructure. IEEE/OSA Journal of Optical Communications and Networking 9, 2 (Feb 2017), A135–A142. DOI:http://dx.doi.org/10.1364/JOCN.9.00A135
Muñoz et al. (2017) R. Muñoz, L. Nadal, R. Casellas, M. S. Moreolo, R. Vilalta, J. M. Fàbrega, R. Martínez, A. Mayoral, and F. J. Vílchez. 2017. The ADRENALINE testbed: An SDN/NFV packet/optical transport network and edge/core cloud platform for end-to-end 5G and IoT services. In 2017 European Conference on Networks and Communications (EuCNC). 1–5. DOI:http://dx.doi.org/10.1109/EuCNC.2017.7980775
Vilalta et al. (2016) R. Vilalta, A. Mayoral, R. Casellas, R. Martínez, and R. Muñoz. 2016. Experimental demonstration of distributed multi-tenant cloud/fog and heterogeneous SDN/NFV orchestration for 5G services. In 2016 European Conference on Networks and Communications (EuCNC). 52–56. DOI:http://dx.doi.org/10.1109/EuCNC.2016.7561003
ETSI (2016a) ETSI. 2016a. Mobile Edge Computing (MEC): Technical Requirements. ETSI GS MEC 002 v1.1.1 (Mar. 2016). http://www.etsi.org/deliver/etsi_gs/MEC/001_099/002/01.01.01_60/gs_MEC002v010101p.pdf
ETSI (2016b) ETSI. 2016b. Mobile Edge Computing (MEC): Framework and Reference Architecture. ETSI GS MEC 003 v1.1.1 (Mar. 2016). http://www.etsi.org/deliver/etsi_gs/MEC/001_099/003/01.01.01_60/gs_MEC003v010101p.pdf
Roman et al. (2016) Rodrigo Roman, Javier Lopez, and Masahiro Mambo. 2016. Mobile Edge Computing, Fog et al.: A Survey and Analysis of Security Threats and Challenges. CoRR abs/1602.00484 (2016). http://arxiv.org/abs/1602.00484
Group (2015) 5G PPP Architecture Working Group. 2015. 5G Vision. Technical Report. https://5g-ppp.eu/wp-content/uploads/2015/02/5G-Vision-Brochure-v1.pdf
EU SELFNET Project (2016) EU SELFNET Project. 2016. Framework for Self-Organized Network Management in Virtualized and Software Defined Networks, Project reference: ICT-2014-2/671672. Funded under H2020. (2016). http://www.selfnet-5g.eu/, Accessed: 2016-07-25.
Neves et al. (2016) Pedro Neves, Rui Calé, Mário Rui Costa, Carlos Parada, Bruno Parreira, Jose Alcaraz-Calero, Qi Wang, James Nightingale, Enrique Chirivella-Perez, Wei Jiang, Hans Dieter Schotten, Konstantinos Koutsopoulos, Anastasius Gavras, and Maria João Barros. 2016. The SELFNET Approach for Autonomic Management in an NFV/SDN Networking Paradigm. Int. J. Distrib. Sen. Netw. 2016, Article 2 (Jan. 2016), 1 pages. DOI:http://dx.doi.org/10.1155/2016/2897479
Neves et al. (2017) Pedro Neves, Rui Calé, Mário Costa, Gonçalo Gaspar, Jose Alcaraz-Calero, Qi Wang, James Nightingale, Giacomo Bernini, Gino Carrozzo, Ángel Valdivieso, Luis Javier García Villalba, Maria Barros, Anastasius Gravas, José Santos, Ricardo Maia, and Ricardo Preto. 2017. Future mode of operations for 5G – The SELFNET approach enabled by SDN/NFV. Computer Standards & Interfaces 54, Part 4 (2017), 229 – 246. DOI:http://dx.doi.org/10.1016/j.csi.2016.12.008 SI: Standardization SDN & NFV.
Verizon (2016) Verizon. 2016. SDN-NFV Reference Architecture. Verizon Network Infrastructure Planning (Feb. 2016). http://innovation.verizon.com/content/dam/vic/PDF/Verizon_SDN-NFV_Reference_Architecture.pdf
Telefonica I+D (2016) Telefonica I+D. 2016. OpenMANO - A ETSI NFV compliant Management and Orchestration (MANO). https://github.com/nfvlabs/openmano. (2016). Accessed: 2016-07-25.
Fraunhofer FOKUS (2016) Fraunhofer FOKUS. 2016. OpenBaton - A ETSI NFV compliant Network Function Virtualization Orchestrator (NFVO). http://openbaton.github.io/. (2016). Accessed: 2016-07-25.
Networks (2016) Big Switch Networks. 2016. The Floodlight Project. http://www.projectfloodlight.org/floodlight/. (2016). Accessed: 2016-07-25.
(ONF) (2017) Open Network Foundation (ONF). 2017. The ONOS Project. http://onosproject.org/. (2017). Accessed: 2017-10-02.
Telegraph and (NTT) (2016) Nippon Telegraph and Telephone (NTT). 2016. Ryu SDN Framework. https://osrg.github.io/ryu/. (2016). Accessed: 2016-07-25.
Shen et al. (2015) W. Shen, M. Yoshida, K. Minato, and W. Imajuku. 2015. vConductor: An enabler for achieving virtual network integration as a service. IEEE Communications Magazine 53, 2 (2015), 116–124. DOI:http://dx.doi.org/10.1109/MCOM.2015.7045399
Vilalta et al. (2015) R. Vilalta, R. Muñoz, A. Mayoral, R. Casellas, R. Martínez, V. López, and D. López. 2015. Transport Network Function Virtualization. Journal of Lightwave Technology 33, 8 (April 2015), 1557–1564. DOI:http://dx.doi.org/10.1109/JLT.2015.2390655
Mohammadkhan et al. (2015) A. Mohammadkhan, G. Liu, W. Zhang, K. K. Ramakrishnan, and T. Woodv. 2015. Protocols to support autonomy and control for NFV in software defined networks. In 2015 IEEE Conference on Network Function Virtualization and Software Defined Network (NFV-SDN). 163–169. DOI:http://dx.doi.org/10.1109/NFV-SDN.2015.7387422
Lombardo et al. (2015) A. Lombardo, A. Manzalini, G. Schembra, G. Faraci, C. Rametta, and V. Riccobene. 2015. An open framework to enable NetFATE (Network Functions at the edge). In 2015 1st IEEE Conference on Network Softwarization (NetSoft). 1–6. DOI:http://dx.doi.org/10.1109/NETSOFT.2015.7116179
Mamatas et al. (2015) L. Mamatas, S. Clayman, and A. Galis. 2015. A service-aware virtualized software-defined infrastructure. IEEE Communications Magazine 53, 4 (April 2015), 166–174. DOI:http://dx.doi.org/10.1109/MCOM.2015.7081091
Duan et al. (2016) Q. Duan, N. Ansari, and M. Toy. 2016. Software-defined network virtualization: an architectural framework for integrating SDN and NFV for service provisioning in future networks. IEEE Network 30, 5 (September 2016), 10–16. DOI:http://dx.doi.org/10.1109/MNET.2016.7579021
Foundation (2016) Linux Foundation. 2016. The Xen Project. https://www.xenproject.org/. (2016). Accessed: 2016-07-25.
Lucrezia et al. (2015) F. Lucrezia, G. Marchetto, F. Risso, and V. Vercellone. 2015. Introducing network-aware scheduling capabilities in OpenStack. In 2015 1st IEEE Conference on Network Softwarization (NetSoft). 1–5. DOI:http://dx.doi.org/10.1109/NETSOFT.2015.7116155
Giotis et al. (2015) K. Giotis, Y. Kryftis, and V. Maglaris. 2015. Policy-based orchestration of NFV services in Software-Defined Networks. In 2015 1st IEEE Conference on Network Softwarization (NetSoft). 1–5. DOI:http://dx.doi.org/10.1109/NETSOFT.2015.7116145
Lai et al. (2015) J. Lai, Q. Fu, and T. Moors. 2015. Rapid IP Rerouting with SDN and NFV. In 2015 IEEE Global Communications Conference (GLOBECOM). 1–7. DOI:http://dx.doi.org/10.1109/GLOCOM.2015.7417318
Wang et al. (2015) H. Wang, S. Chen, H. Xu, M. Ai, and Y. Shi. 2015. SoftNet: A software defined decentralized mobile network architecture toward 5G. IEEE Network 29, 2 (March 2015), 16–22. DOI:http://dx.doi.org/10.1109/MNET.2015.7064898
Xia et al. (2015) M. Xia, M. Shirazipour, Y. Zhang, H. Green, and A. Takacs. 2015. Optical service chaining for network function virtualization. IEEE Communications Magazine 53, 4 (April 2015), 152–158. DOI:http://dx.doi.org/10.1109/MCOM.2015.7081089
Callegati et al. (2017) F. Callegati, W. Cerroni, C. Contoli, and F. Foresta. 2017. Performance of intent-based virtualized network infrastructure management. In 2017 IEEE International Conference on Communications (ICC). 1–6. DOI:http://dx.doi.org/10.1109/ICC.2017.7997431
Haleplidis et al. (2014) Evangelos Haleplidis, Jamal Hadi Salim, Spyros Denazis, and Odysseas Koufopavlou. 2014. Towards a Network Abstraction Model for SDN. Journal of Network and Systems Management 23, 2 (July 2014), 309–327. DOI:http://dx.doi.org/10.1007/s10922-014-9319-3
Doria et al. (2010) A. Doria, J. Hadi Salim, R. Haas, H. Khosravi, W. Wang, L. Dong, R. Gopal, and J. Halpern. 2010. Forwarding and Control Element Separation (ForCES) Protocol Specification. RFC 5810. RFC Editor. http://www.rfc-editor.org/rfc/rfc5810.txt http://www.rfc-editor.org/rfc/rfc5810.txt.
Matias et al. (2015) Jon Matias, Jokin Garay, Nerea Toledo, Juanjo Unzilla, and Eduardo Jacob. 2015. Toward an SDN-enabled NFV architecture. IEEE Communications Magazine 53, 4 (2015), 187–193. DOI:http://dx.doi.org/10.1109/MCOM.2015.7081093
Cheng et al. (2015) Guozhen Cheng, Hongchang Chen, Hongchao Hu, Zhiming Wang, and Julong Lan. 2015. Enabling network function combination via service chain instantiation. Computer Networks 92, Part 2 (2015), 396–407. DOI:http://dx.doi.org/10.1016/j.comnet.2015.09.015
Saridis et al. (2016) G. M. Saridis, S. Peng, Y. Yan, A. Aguado, B. Guo, M. Arslan, C. Jackson, W. Miao, N. Calabretta, F. Agraz, S. Spadaro, G. Bernini, N. Ciulli, G. Zervas, R. Nejabati, and D. Simeonidou. 2016. Lightness: A Function-Virtualizable Software Defined Data Center Network With All-Optical Circuit/Packet Switching. Journal of Lightwave Technology 34, 7 (April 2016), 1618–1627. DOI:http://dx.doi.org/10.1109/JLT.2015.2509476
Mustafa et al. (2015) H.D. Mustafa, B.M. Baveja, S. Vijayan, S.N. Merchant, and U.B. Desai. 2015. Replicating the geographical cloud: Provisioning omnipresence, omniscience and omnipotence. Future Generation Computer Systems 47 (2015), 1 – 15. DOI:http://dx.doi.org/10.1016/j.future.2014.12.004 Special Section: Advanced Architectures for the Future Generation of Software-Intensive Systems.
Project (2016) KVM Project. 2016. The Kernel-based Virtual Machine Project. https://www.linux-kvm.org/page/Main_Page. (2016). Accessed: 2016-07-25.
Martins et al. (2014) Joao Martins, Mohamed Ahmed, Costin Raiciu, Vladimir Olteanu, Michio Honda, Roberto Bifulco, and Felipe Huici. 2014. ClickOS and the Art of Network Function Virtualization. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (NSDI’14). USENIX Association, Berkeley, CA, USA, 459–473. http://dl.acm.org/citation.cfm?id=2616448.2616491
natarajan.sriram@gmail.com et al. (2016) natarajan.sriram@gmail.com, Ram (Ramki) Krishnan, Anoop Ghanwani, Dilip Krishnaswamy, Peter Willis, Ashay Chaudhary, and Felipe Huici. 2016. An Analysis of Lightweight Virtualization Technologies for NFV. Internet-Draft draft-natarajan-nfvrg-containers-for-nfv-03. Internet Engineering Task Force. https://datatracker.ietf.org/doc/html/draft-natarajan-nfvrg-containers-for-nfv-03 Work in Progress.
Foundation (2016) Linux Foundation. 2016. Data Plane Development Kit (DPDK) Documentation. http://dpdk.org/doc. (2016). Accessed: 2016-07-25.
Open Source Community (2016) Open Source Community. 2016. Open vSwitch. http://openvswitch.org/. (2016). Accessed: 2016-07-25.
ONF (2009) ONF. 2009. OpenFlow Switch Specification - Version 1.0.0. (Dec. 2009). https://3vf60mmveq1g8vzn48q2o71a-wpengine.netdna-ssl.com/wp-content/uploads/2013/04/openflow-spec-v1.0.0.pdf.
Pfaff and Davie (2013) Ben Pfaff and Bruce Davie. 2013. The Open vSwitch Database Management Protocol. RFC 7047. (16 Dec. 2013). DOI:http://dx.doi.org/10.17487/rfc7047
Kim et al. (2016) W. Kim, J. Li, J. W. K. Hong, and Y. J. Suh. 2016. OFMon: OpenFlow monitoring system in ONOS controllers. In 2016 IEEE NetSoft Conference and Workshops (NetSoft). 397–402. DOI:http://dx.doi.org/10.1109/NETSOFT.2016.7502474
Muqaddas et al. (2017) A. S. Muqaddas, P. Giaccone, A. Bianco, and G. Maier. 2017. Inter-controller Traffic to Support Consistency in ONOS Clusters. IEEE Transactions on Network and Service Management PP, 99 (2017), 1–1. DOI:http://dx.doi.org/10.1109/TNSM.2017.2723477
(ONF) (2017) Open Network Foundation (ONF). 2017. The CORD Project. http://opencord.org/. (2017). Accessed: 2017-10-02.
Chinnici et al. (2007) Roberto Chinnici, Jean-Jacques Moreau, Arthur Ryman, and Sanjiva Weerawarana. 2007. Web Services Description Language (WSDL) Version 2.0 Part 1: Core Language. World Wide Web Consortium, Recommendation REC-wsdl20-20070626. (June 2007).
Li et al. (2016) L. Li, W. Chou, W. Zhou, and M. Luo. 2016. Design Patterns and Extensibility of REST API for Networking Applications. IEEE Transactions on Network and Service Management 13, 1 (March 2016), 154–167. DOI:http://dx.doi.org/10.1109/TNSM.2016.2516946
Fielding et al. (1999) R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, and T. Berners-Lee. 1999. Hypertext Transfer Protocol – HTTP/1.1. (1999).
Voellmy et al. (2012) Andreas Voellmy, Hyojoon Kim, and Nick Feamster. 2012. Procera: A Language for High-level Reactive Network Control. In Proceedings of the First Workshop on Hot Topics in Software Defined Networks (HotSDN ’12). ACM, New York, NY, USA, 43–48. DOI:http://dx.doi.org/10.1145/2342441.2342451
Foster et al. (2011) Nate Foster, Rob Harrison, Michael J. Freedman, Christopher Monsanto, Jennifer Rexford, Alec Story, and David Walker. 2011. Frenetic: A Network Programming Language. In Proceedings of the 16th ACM SIGPLAN International Conference on Functional Programming (ICFP ’11). ACM, New York, NY, USA, 279–291. DOI:http://dx.doi.org/10.1145/2034773.2034812
Hwang et al. (2015) J. Hwang, K. K. Ramakrishnan, and T. Wood. 2015. NetVM: High Performance and Flexible Networking Using Virtualization on Commodity Platforms. IEEE Transactions on Network and Service Management 12, 1 (March 2015), 34–47. DOI:http://dx.doi.org/10.1109/TNSM.2015.2401568
ETSI (2016) ETSI. 2016. Network Functions Virtualisation (NFV) - Virtualisation Technologies - Report on the application of Different Virtualisation Technologies in the NFV Framework. ETSI GS NFV-EVE 004 V1.1.1 (Mar. 2016). http://www.etsi.org/deliver/etsi_gs/NFV-EVE/001_099/004/01.01.01_60/gs_NFV-EVE004v010101p.pdf
Verma et al. (2015) Abhishek Verma, Luis Pedrosa, Madhukar R. Korupolu, David Oppenheimer, Eric Tune, and John Wilkes. 2015. Large-scale cluster management at Google with Borg. In Proceedings of the European Conference on Computer Systems (EuroSys). Bordeaux, France.
Schwarzkopf et al. (2013) Malte Schwarzkopf, Andy Konwinski, Michael Abd-El-Malek, and John Wilkes. 2013. Omega: flexible, scalable schedulers for large compute clusters. In SIGOPS European Conference on Computer Systems (EuroSys). Prague, Czech Republic, 351–364. http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf
Hindman et al. (2011) Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, and Ion Stoica. 2011. Mesos: A Platform for Fine-grained Resource Sharing in the Data Center. In Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation (NSDI’11). USENIX Association, Berkeley, CA, USA, 295–308. http://dl.acm.org/citation.cfm?id=1972457.1972488
Foukas et al. (2017) X. Foukas, G. Patounas, A. Elmokashfi, and M. K. Marina. 2017. Network Slicing in 5G: Survey and Challenges. IEEE Communications Magazine 55, 5 (May 2017), 94–100. DOI:http://dx.doi.org/10.1109/MCOM.2017.1600951
Li et al. (2017) X. Li, M. Samaka, H. A. Chan, D. Bhamare, L. Gupta, C. Guo, and R. Jain. 2017. Network Slicing for 5G: Challenges and Opportunities. IEEE Internet Computing 21, 5 (2017), 20–27. DOI:http://dx.doi.org/10.1109/MIC.2017.3481355