Design of NFV Platforms: A Survey

02/25/2020
by   Tianzhu Zhang, et al.
0

Due to the intrinsically inefficient service provisioning in traditional networks, Network Function Virtualization (NFV) keeps gaining attention from both industry and academia. By replacing the purpose-built, expensive, proprietary network equipments with software network functions consolidated on commodity hardware, NFV envisions a shift towards a more agile and open service provisioning paradigm with much lower capital expenditure (CapEx) and operational expenditure (OpEx). Nonetheless, just like any complex system, NFV platforms commonly consist of abounding software and hardware components, and usually incorporate disparate design choices based on distinct motivations or use cases. This broad collection of convoluted alternatives makes it extremely arduous for network operators to make proper choices. Although numerous efforts have been devoted to investigating different aspects of NFV, none of them specifically focused on NFV platforms or attempted to explore the design space. In this paper, we present a comprehensive survey on NFV platform design. Our study solely targets existing NFV platform implementations. We begin with a top-down architectural view of the standard reference NFV platform and present our taxonomy of existing NFV platforms based on principal purpose of design. Then we thoroughly explore the design space and elaborate the implementation choices each platform opts for. We believe that our study gives a detailed guideline for network operators or service providers to choose the most appropriate NFV platform based on their respective requirements.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 4

page 19

02/25/2020

NFV Platform Design: A Survey

Due to the intrinsically inefficient service provisioning in traditional...
04/01/2020

FaaSten Your Decisions: Classification Framework and Technology Review of Function-as-a-Service Platforms

Function-as-a-Service (FaaS) is a cloud service model enabling developer...
09/05/2018

A Threat Modeling Framework for Evaluating Computing Platforms Against Architectural Attacks

software component misuse a privileged relationship with the hardware to...
07/27/2021

Two Basic Queueing Models of Service Platforms in Digital Sharing Economy

This paper describes two basic queueing models of service platforms in d...
02/17/2021

SimFaaS: A Performance Simulator for Serverless Computing Platforms

Developing accurate and extendable performance models for serverless pla...
04/07/2020

Modularis: Modular Data Analytics for Hardware, Software, and Platform Heterogeneity

Today's data analytics displays an overwhelming diversity along many dim...
06/04/2021

Analyzing Open-Source Serverless Platforms: Characteristics and Performance

Serverless computing is increasingly popular because of its lower cost a...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Traditionally, network services are provisioned by means of purpose-built, proprietary hardware appliances (or middleboxes). Middleboxes embody a large variety of specialized functions to forward, classify, or transform traffic based on packet content. Examples of moddleboxes include but not limited to L2 switching, L3 Routing, Network Address Translation (NAT), Firewall (FW), Deep Packet Inspection (DPI), Intrusion Detection System (IDS), Load Balancer (LB), WAN optimizer, and stateful proxy. Nowadays, middleboxes are ubiquitous in enterprise networks 

[107]. However, with the increasingly diversified user requirements as well as the rapid growth of Internet traffic, in terms of both volume and heterogeneity [19], hardware middleboxes begin to exhibit several fundamental disadvantages. First off, middleboxes are generally expensive to acquire and usually require domain-specific knowledge to manage them, resulting in large capital expenditure (CapEx) and operational expenditure (OpEx). In addition, adding customized functionalities is extremely time-consuming if not impossible, and it sometimes takes an entire purchase cycle (e.g, four years) to bring in equipments with new features [72]. Such tight coupling with hardware production cycle considerably hampers network innovation and prolongs time-to-market. Deploying new network services (NSs) is also a tedious process, as technicians are required to visit specific sites and place the middleboxes in a pre-defined order to form the correct service function chains (SFCs). Service instantiation might even take days. Worse still, service maintenance usually involves constant repetition of the same process. Furthermore, because of the inherent inflexibility, it is not trivial for hardware middleboxes to elastically scale in and out based on the shifting demand or other system dynamics. Consequently, network operators usually resort to peak-load provisioning, which in turn leads to ineffective resource utilization and extravagant energy consumption.

Figure 1: Traditional vs. NFV paradigm

To improve service provisioning and get rid of network ossification, telecommunication operators began to pursue new solutions that can guarantee both cost-effectiveness and flexibility. The advent of Software Defined Networking (SDN) [74] and Network Function Virtualization (NFV) [78] provides alternative approaches for network management and service provisioning. SDN decouples the control plane from data plane and leverages a logically centralized controller to configure the programmable switches based on a global view, while NFV replaces specialized middleboxes with software Virtual Network Functions (VNFs) consolidated on Commodity Off-The-Shelf (COTS) hardware. The key to their success lies in separating the evolution timeline of software network functions and specialized hardware, completely unleashing the potential of the former. An illustrative example contrasting NFV paradigm with traditional network is shown in Fig. 1. Compared to the traditional service provisioning paradigm based on hardware middleboxes, NFV manages to achieve cost-effectiveness by consolidating multiple instances of VNFs on high-volume yet inexpensive servers, routers, or storage. Service provisioning in NFV is also highly simplified as the previously troublesome tasks, such as middlebox deployment, monitoring, migration, and scaling, can be optimally automated through software control mechanisms. It is thus convenient for NFV solutions to exploit available resources and management tools of the cloud infrastructure or network edge. In addition, NFV remarkably promotes network innovation and accelerates the time-to-market process as network function development is cut down to writing software programs using standard application programming interfaces (APIs).

Thanks to these indispensable merits, NFV keeps gaining momentum from both industry and academia. The first concerted effort towards NFV standardization began in 2012, with the appointment of European Telecommunications Standards Institute (ETSI) [35] as Industry Specification Group. Currently, ETSI consists of more than 500 members across the world, including major telecommunication operators, service providers, manufacturers, as well as universities. In the meanwhile, the continuous advancement of COTS hardware capabilities and the emergence of high-speed packet processing techniques adequately close the previously huge performance gap between software network functions and specialized middleboxes. Resources of other hardware components, such as Graphics Processing Unit (GPU) and in-path programmable network devices, can also be exploited to share the workload and alleviate the burden of CPU. These technical impetuses immeasurably stimulate the growth of NFV. During the last six years, a large assemblage of NFV platforms have been implemented to spur the innovation and evolution of NFV.

However, just like any complex systems, NFV platforms usually encompass profuse software and hardware components, and embrace divergent design choices driven by their respective motivations or use cases. The design space of these platforms can be very expansive, with design choices ranging from high-level VNF development, such as VNF execution models, state management schemes, or genres of APIs, to low-level infrastructure details, like packet I/O frameworks, VNF interconnect methods, or virtualization techniques. They also opt for various datapath acceleration techniques including compute batching, zero-copy packet transfer, data prefetching, and computation offloading. Such broad range of platform implementations coupled with even more extensive design space makes it extremely difficult (if not impossible) for network operators to choose the most suitable solution. The tradeoffs and caveats between different design choices are also unclear, making the implementation of new platforms laborious and error-prone. Several existing works have investigated some aspects of NFV, including VNF placement [64], resource allocation [45], service function chaining [11], and security [120], but none of them specifically focused on the design of NFV platforms, nor did they attempt to explore the design space or review different implementation choices. In [26, 79, 122], the authors investigated a small set of industrial NFV projects, which are complementary to our work.

In this paper, we focus on existing NFV platforms and present a comprehensive survey on their design. The contribution of this paper can be summarized as follows:

  • we classify existing NFV platforms based on main purposes and overview their internals.

  • we explore the NFV design space and discuss the various design choices adopted by existing platforms.

This paper is organized as follows: in Sec. 2, we give an architectural overview of the components of NFV platforms. Then we present our taxonomy on existing platforms in Sec. 3. In Sec. 4, we propose a collection of critical design choices and survey the solutions adopted by different platforms. We envision future directions and draw conclusion in Sec. LABEL:sec:challenge and 5, respectively.

2 NFV platform: an architectural overview

We devote this section to presenting a general architectural overview of a typical NFV platform and reviewing the key components in depth. Although a reference architecture has been defined by the ETSI specification [36], most of the existing NFV platforms did not strictly follow it. As a result, we seek to combine the ETSI reference architecture with those of the existing platforms and present a generic view, as illustrated in Fig. 2. An NFV platform generally consists of three primary components, namely the NFV Management and Orchestration (MANO) plane, the service plane, and the NFV Infrastructure (NFVI). The MANO plane provides centralized control over service provisioning and management. The NFVI contains a collection of computation, storage, and network resources that are distributed across different infrastructural nodes. MANO plane components systematically monitor and schedule the resources to build the virtualized environment and accommodate different network services. The service plane contains a diversified collection of VNFs that are ordered in the form of service chains to fulfill the promised network services. These service chains are also carefully monitored and adjusted by the MANO plane components to efficiently multiplex the NFVI resources. In general, the service place is enabled through the concerted operations from both MANO plane and NFVI.

Figure 2: The architecture of a general NFV platform

2.1 MANO plane

NFV Management and Orchestration (MANO) is the central point for service provisioning in NFV. A MANO system typically consists of three sub-systems: NFV Orchestrator (NFVO), Virtual infrastructure Manager (VIM), and VNF Manager (VNFM). As shown in Fig. 2, NFVO is responsible for the instantiation, management, and termination of network services. At present, an NFVO commonly encompasses different modules to apply different MANO operations. On the right part of Fig 2, we illustrate four example modules. The placement module is in charge of rendering the best deployment, possibly in an incremental fashion. When new services need to be deployed, the placement module analyzes the service descriptions or requirements specified by network operators, constructs an aggregated service representation (e.g. service processing graph), performs necessary optimizations (e.g. function merging, redundant elimination), and calculates the best possible placement strategy by determining the PoPs to deploy the related VNFs and their chaining order. The monitoring module is responsible for collecting statistics and events from both the service plane and the infrastructure, and provides runtime feedback to other NFVO modules. Based on the traffic condition collected on-the-fly, the placement module can recalculate a new placement to achieve better performance. The scheduling module can dynamically make fine-grained scheduling decisions to attain resource efficiency. The scaling/failover module can also collaborate with the placement module to scale in/out particular VNFs or service chains to cater traffic fluctuation, or instantiate new VNF replicas upon failure. Based on the decisions made by the aforementioned modules, the NFVO closely interacts with the other two MANO plane components to realize the intended service configurations and resource allocations.

VIM is designed to configure infrastructure components to accommodate the heterogeneous VNFs or service chains instantiated in the service plane. In specific, it directs the provision/release/upgrade of NFVI resources and manages the mapping between virtual and physical resources. It also manages the data path for network services by creating/deleting/updating virtual interfaces and logical links, and collects the NFVI software and hardware status on behalf of the NFVO monitoring module. Note that an instance of VIM might control all the resources of the whole NFVI or that of multiple NFVI-PoPs. In some cases, a VIM might also just control a specific type of resource.

On the other hand, VNFM interacts with the service plane and takes care of the instantiation, scaling, upgrade, and termination of individual VNFs and service chains. It also needs to synchronize with VIM to allocate or release the related infrastructural resources. According to ETSI specification, the MANO system might also maintain several data stores to hold configuration information such as network service descriptions, VNF templates, NFVI resources, etc.

2.2 NFV Infrastructure

NFV Infrastructure (NFVI) contains all the essential hardware and software components to compose virtual network services. The infrastructure might belong to Internet service providers, cloud/edge operators, or simply infrastructure providers. It usually embodies a large variety of computing nodes and network equipments. Each computing node or network equipment is commonly referred as NFVI-PoP. Network equipments in NFVI can be traditional purpose-built switches/routers or the emerging programmable switches that can be remotely orchestrated with SDN or P4 [13] semantics. The most typical form of computing node in NFVI are the COTS servers. These servers normally contain several critical hardware components, including the physical Network Interface Controllers (NICs), multicore CPUs, and main memory, and they are interconnected through PCI buses. The physical NICs are capable of operating at Gigabyte level with multiple queues promoting parallelization. High-speed packet I/O techniques are also integrated by the NICs to transport packets to the service plane. Inside the server, multicore CPUs are distributed across non-uniform memory access (NUMA) nodes to speed up traffic processing. Aside from CPU, other computing units such as smartNIC and GPU are also widely utilized by existing NFV platforms to further boost performance. The virtualization layer in COTS server provides the environment to accommodate network functions. The virtualization can be at hardware level relying on bare-metal hypervisors or at OS level using container engines. Some platforms even execute network functions as ordinary processes, which is addressed as Physical Network Functions (PNFs) in some works. In this paper, we universally refer to them as VNFs for the sake of simplicity. To ensure efficient communication between the VNFs and the external network, virtual interconnects need to be precisely established. This is typically accomplished using state-of-the-art software virtual switches or customized forwarding tables. Note that we consider physical links between COTS servers and network equipments part of NFVI as well.

2.3 Service plane

The service plane is populated with a variety of Virtual Network Functions (VNFs) implementing different processing to constitute various network services. The distribution of VNFs inside virtual environments is quite flexible. For instance, a VNF or a whole service chain can be mapped to a single VM for execution, a VNF can also be split into finer-grained processing elements and deployed across multiple NFV PoPs. In addition, VNFs are usually constructed using different programming abstractions and operate in different runtime execution models.

3 Taxonomy of NFV platforms

We devote this section to reviewing existing NFV platforms and classifying them according to their primary design purpose. Based on our literature review and the reference architecture, we classify existing NFV platforms based on three general design purposes, namely integrated NFV platform, MANO system, and NFVI optimization. An integrated NFV platform contains both MANO plane and NFVI to support either end-to-end service provisioning or VNF development. A MANO system deals with all or a subset of the management issues such as instantiation/termination, placement, dynamic scaling, monitoring, and resilience of network services. An NFVI optimization platform strives for guaranteeing an efficient infrastructural packet data path by optimizing the related procedures and eliminating redundant processing.

3.1 Integrated NFV platform

Integrated NFV platforms can be further classified into two categories. The first category is for network operators or service providers to efficiently provision end-to-end services for their clients or subscribers. The second category is for network developers to make VNF implementation less time-consuming and error-prone.

End-to-end service provisioning

Several platforms were proposed for functionality implementation. UNIFY [23] and Cloud4NFV [110] are two earliest projects for integrated NFV platforms. UNIFY introduces a general framework to automate service provisioning. It employs a layered graph abstraction to automatically map user-specified services into actual SFCs deployed to the underlying NFVI PoPs. Cloud4NFV provides a SFC model to allow for fine-grained traffic classification and steering, and relies on cloud management tools to perform NFV MANO operations. CloudBand [20] is an integrated NFV platform conposed of Cloudband node and a management system. The CloudBand node supplies resources to accommodate network services. The management system performs MANO operations across different service domains. GNF [24] brings NFV to the network edge. It exposes a graphical user interface to specify service intent and display system events, and uses a manager to perform MANO operations. An agent is embedded into each edge device to manage the containerized VNFs. Considering the resource constraint of edge devices, it runs VNFs in lightweight Linux containers instead of VMs. NetFATE [69] also aims at deploying network service on the edge and its architecture is similar to GNF. SONATA [28] brings DevOps concept into NFV by providing a service development toolchain integrated with a service platform and orchestration system. The toolchain comprises of a service programming abstract with support tools to allow developers to implement, monitor, and optimize VNFs or SFCs. The service platform encompasses a customizable MANO framework to deploy and manage network services. It also supports platform recursion and network slicing. Eden [6] is another platform purposed for provisioning network functions at end-hosts in a single administrative domain. It is composed of a controller, stages, and enclaves at the end-hosts. The controller provides centralized VNF coordination based on its global network view. Stages reside in the end-host stack to associate application semantics to particular traffic classes. The per-host Eden enclave maintains a set of Match/Action tables to decide the destination VNF for each packet based on its traffic class. VNFs in Eden are written in F# language and are compiled the into executable byte-code, which is subsequently interpreted inside the enclaves.

Other integrated NFV platforms concentrate on performance with high traffic load (e.g. 40/100 Gbps). OpenBox [14] allows developers to implement VNF logic through the northbound API of the OpenBox controller, which in turn deploys the logic to the data plane and realizes the intended processing sequence through the OpenBox protocol. The OpenBox controller additionally merges the core control logics of multiple VNFs to avoid duplicated processing and spare NFVI-PoP resources for other tasks. The OpenBox data plane is extensible with specialized hardware or pure software. By default, OpenBox contains over 40 processing blocks that can be chained to realize various VNFs, and it can further seamlessly integrate custom blocks at runtime. Slick [4]

allows developers to write network functions in high-level programming language and specify intended network service for different traffic flows or classes. The Slick controller performs the placement and traffic steering using several heuristic algorithms. In particular, the Slick runtime parses the specified policy and decides the optimal servers to place the elements, and installs forwarding routes to the in-path switches to realize the intended processing sequence. A shim is configured on each server to provide up-to-date system status for Slick runtime to make incremental optimizations.

Elastic Edge (E2) [86] is meant for a general NFV platform that frees developers from common deployment and management issues, which are instead delegated to the E2 manager. E2 allows network operators to express network policies in terms of individual pipelets, each of which consists of a subset of input traffic (or traffic class) and a processing graph. Similar to OpenBox, E2 manager merges the pipelets into a single graph, and instructs local agents to place the corresponding network functions across the server cluster and interconnect them through the high-speed E2 data plane. E2 also provides hooks for the VNFs and the data plane to detect system overload, and dynamically scales NF instances with flow affinity guaranteed. SDNFV [130] combines SDN and NFV to realize a flexible, hierarchical control framework over VNFs. It comprises of three hierarchies: SDNFV application, SDN controller, NFV orchestrator, and NF manager. SDNFV application utilizes a graph abstraction to represent the intended network services for different traffic flows. Then it proposes a heuristic algorithm to jointly deploy the VNFs to COTS servers and configure traffic routes across them, through the SDN controller and NFV orchestrator. An instance of NF manager is installed on each COTS server to manage the local VNFs and traffic routing. Each manager maintains an extended OpenFlow (OF) table based on host-level status. This table can also be configured by the remote SDN controller (for default routing) and the local VNFs (based on their internal states), realizing a more flexible control paradigm beyond SDN. MicroNF [76] deals with the consolidation, placement, scaling, and scheduling for modularized SFCs altogether. It consists of a centralized controller and a high-speed infrastructure. The MicroNF controller exploits a graph constructor to analyze inter-element dependency and reconstruct the service graph to reuse the redundant elements. Then it uses a placer to optimally place and consolidate modularized VNFs with the goal of reducing inter-VM data transfer. At runtime, it dynamically collects element statistics and applies two resource scaling algorithms for SFC scaling with minimal inter-element latency. In addition, the MicroNF infrastructure ensures high-speed, consistent packet forwarding and fair VNF scheduling. NF [17] advocates designing network functions as the microservices, since the self-contained, loosely-coupled design supports fine-grained resource allocation and VNF scaling. NF is designed for building VNFs and SFCs using disaggregated, reusable network processing components. It consists of a centralized orchestrator and a cluster of per-server agents. Similar to other integrated NFV platforms, it allows network operators to specify service requests as directed graphs, which are converted to equivalent forwarding graphs by the orchestrator. Then agents on the related COTS servers are instructed to deploy and interconnect the corresponding VNFs according to the graph specifications. Metron [53] utilizes the available resources of both underlying hardware and programmable network to accomplish high-speed processing. The Metron controller parses the traffic classes associated with a service chain and generates a synthesized processing graph. Then it decomposes the graph into a bunch of stateless operations offloaded to the in-path programmable network equipments, while mapping the remaining stateful operations to COTS servers. To reduce the overhead of software-based traffic dispatching, Metron leverages the in-path programmable switches to tag packets, which are matched by the COTS NIC to dispatch packets to the correct cores for “run-to-completion” processing. For management, Metron additionally deploys an agent on each server to monitor the runtime statistics and scale overloaded VNF instances accordingly. OPNFV [91]

is a joint open-source project to promote NFV deployment and innovation. It involves a large compilation of tasks including continuous integration of components from upstream projects, function verification, performance benchmarking, and service automation. As an integrated platform, it provides full-fledged features including VNF management, dynamic service provisioning, prompt fault detection and recovery, and vendor-/operator-agnostic deployment.

NNF [12] explores capabilities of resource-constrained devices to deploy VNFs. The authors post some preliminary results to prove the feasibility of implementing NNF over a variety of servers with heterogeneous hardware specifications and accelerators. CoNFV [119] combines cloud and end-hosts to reduce service deployment cost and processing latency. An abstraction is proposed to divide SFC processing logics between cloud infrastructure and end-hosts. An intuitive API is also designed for developers to port existing VNFs into CoNFV.

VNF development

There is a collection of works dedicated to facilitating the development of VNFs. They aim at supplying high-level APIs to facilitate the VNF development process. Most of them are equipped with runtimes to guarantee efficient VNF execution. The spirit of these works is to free developers from reinventing the wheels for common management tasks and let them focus on VNF control logic implementation. xOMB [3] is the earliest endeavor for building scalable, programmable, and performant middleboxes on COTS servers. It arranges a set of programmable modules into a general pipeline to implement the expected network function. The xOMB control plane monitors the execution pipeline to make timely adjustment upon instance failure. CoMb [103] advocates the consolidation of network functions at both execution and management level. The centralized CoMb controller takes as input the service policies and infrastructure specifications and solves an optimization model to decide the optimal deployment strategy, which is mapped to the distributed CoMb middleboxes by allocating the required resources. FlowOS [10] is a kernel-based programmable platform purposed for middlebox development. It supports per-flow processing with a high-level programming model to conceal low-level complexities. NetBricks [87] facilitates the VNF development process by implementing a small set of core processing elements that are highly optimized and customizable through user-defined functions. Instead of replying on VMs or containers, NetBricks employs safe language, an efficient runtime library, and unique types to ensure similar level of VNF isolation with much lower overhead. Scylla [97] is a declarative language for flow-level VNF development. It provides several high-level abstractions to allow developers express their intents such as SFCs, VNF monitoring, and state management. The scylla runtime is in charge of fulfilling these intents to the infrastructure. libVNF [81] implements a generic library to assist the development of VNFs ranging from L2/L3 middleboxes to transport-/application-layer endpoints, with the support seamless integration of the kernel and third-party network stacks. A request object abstraction is porposed to maintain application states across multiple non-blocking, event-driven callbacks. The libVNF API is also capable of interacting with multi-level data stores for state management across threads of a single VNF or multiple VNF replicas. Flick [1] brings application-specific semantics into VNF development on multi-core COTS servers. The authors implement a domain-specific language named flick that offers high-level abstractions and common primitives to assist VNF development. The compiler automatically translates the flick programs into parallel task graphs with bounded runtime resource usage. Multiple graphs can execute simultaneously without interference through cooperative scheduling. NFMorph [2] proposes to decouple network function logic from packet processing optimization. VNF programs are expressed in a domain-specific language with primitives that are coherent with packet processing pipelines. Then the NFMorph runtime dynamically profiles and optimizes the VNF execution. Polycube [77] is purposed to build VNFs that can be dynamically adjusted at runtime. Its kernel fast path leverages extended Berkeley Packet Filter (eBPF) [33] to sustain high-speed packet processing. It also exposes a set of abstractions to ease SFC development. The following platforms adopt different means to avoid the complexities introduced by callback-based asynchronous programming model. NetStar [31] implements a flow-based asynchronous interface combined with the future/promise C++ library for VNF development. Instead of spreading control logic across multiple callback functions, NetStar mimics sequential execution by chaining multiple future objects and continuation functions over a single function call. ClickNF [38] augments the Click modular router [59] with a modular, configurable, and extensible TCP stack to build L2-L7 network functions. It further incorporates DPDK for kernel-bypassing packet I/O. A blocking I/O primitive is also proposed to alleviate development difficulty imposed by the traditional asynchronous non-blocking I/O paradigm. S6 [116] extends distributed state object (DSO) with a programming model to build elastically scalable VNFs. The S6 runtime manages the shared VNF states distributed in the DSO space. To meet performance requirements, S6 employs a set of optimizations including micro-threaded scheduling, DSO space reorganization. StatelessNF [52] embraces the separation of concerns design by decoupling the VNF states from processing, so that developers only need to concentrate on VNF-specific logic, while StatelessNF arranges for state replication and management tasks. The VNF states are maintained in a distributed key-value data store that guarantees low-latency access and data resilience. The orchestration plane dynamically monitors VNF/infrastructure status and makes adjustments.

3.2 Management and orchestration

A lot of platforms are purposed to provide NFV MANO solutions. Some of them strive for full-fledged, holistic MANO systems, while others tackle only a facet of the MANO issues, such as the scheduling, monitoring, scaling, load-balancing, failover, and manage of VNFs and SFCs. In this section, we discuss these two categories of solutions respectively.

Holistic MANO system

ETSO [75] is an ETSI-compliant NFV MANO platform for end-to-end SFC provisioning over heterogeneous cloud environments. It addresses key NFV orchestration challenges with a shared service abstraction, optimal VNF/SFC placement algorithms over heterogeneous technologies related to NFV, SDN, and cloud computing. OpenMANO [70] aims at implementing the ETSI NFV MANO framework with guaranteed levels of performance and portability. It consists of a MANO system (openmano), a virtual infrastructure manager (openvim), and a graphical user interface (GUI). The openvim directly manages with the NFVI resources. It also interacts with an SDN controller to create the intended traffic datapath, and relies on a REST northbound API to communicate with the openmano, where concerned MANO tasks are conducted. Open Baton [84] is another ETSI MANO-compliant platform with major objective of developing an extensible and customizable framework for service orchestration across heterogeneous NFV Infrastructures. It manages a diverse set of VNFs running on a multi-site NFVI with heterogeneous virtualization technologies. It also features network slicing using SDN technologies to multiplex the infrastructural resources across multiple VNF instances or network services. vConductor [104] supports completely automated virtual network deployment by simplifying the provisioning procedure. It also adopts a multi-objective resource scheduling algorithm to meet individual business requirements and uses enhanced inventory management for fault isolation. It further employs a plug-in architecture with modular design to enhance the interoperability with various NFV management entities. T-NOVA [118] leverages SDN controllers and cloud management tools to design and implement a software NFV MANO stack to automate the deployment and management of Network Functions-as-a-Service (NFaaS) over virtualized network infrastructures. TeNOR [96] is an NFV orchestrator based on microservice architecture. It proposes two approaches to address resource and service mapping.

Scaling and failover

One attractive aspect of NFV is the flexibility to cope with traffic and system dynamics. When input traffic swells, the MANO plane can dynamically launch new instances to rebalance the workload. When a VNF instance crashes, new replicas can also be promptly initiated to avoid service disruption. Correspondingly several platforms endeavor to guarantee efficient scaling and failover especially for stateful VNFs. Split/Merge [92] exposes a programming abstraction that promises transparent, load-balanced VNF scaling with guaranteed per-flow state consistency. In its prototype system, a centralized orchestrator along with SDN controller is employed to direct VNF scaling and flow migration. An instance of VMM agent is deployed on each server to create or remove VNFs on demand. By integrating the Split/Merge API, per-flow VNF states are Split or Merged across multiple replicas. The system then migrates the relevant states and configures the network to direct flows to the correct replicas. TFM [115] also aims at achieving safe, transparent, and efficient flow migration. Similar to Split/Merge, it instructs the migration process through a centralized controller. The controller decouples flow and state migration processes with three modules: a state manager, a flow manager and a forwarding manager. The state manager conducts state migration through southbound APIs. The forwarding manager interacts with SDN controller to traffic steering rules. The flow manager manages the TFM boxes, which perform packet classification and buffering during flow migration. OpenNF [41] is a non-intrusive control framework with guaranteed SLAs, correct packet processing, and efficient resource usage. It implements a controller that consists of an event-driven model to capture relevant packets, a southbound API to request the import/export of VNF states at different granularities (i.e. single/multiple/all flows), and a northbound API for control applications to instruct state migration or synchronization. In particular, state migration is carefully crafted to avoid packet losses or out-of-order processing, while state synchronization can be performed with either strong or eventual consistency. DiST [60] and U-HAUL [68] follow similar procedures for state and flow migration, but does not require the involvement of the centralized controller. For instance, U-HAUL aims at implementing an efficient state migration framework. Based on the observation that mice flows are usually short-lived and their state/flow migrations incur huge overhead, it thus only identifies and migrates the execution states for elephant flows while keeps states of mice flows in the original VNF instance until expiration. By reducing the number of migrated flow states, the migration process becomes more efficient. FTMB [106] adopts the rollback recovery scheme to conduct failover operation of network functions. CHC [56] adopts a set of state management and optimization techniques to ensure service correctness without degrading performance. In particular, it offloads VNF states to the distributed data store and employs state caching and update algorithms to ensure high performance. It additionally leverages metadata to guarantee the a set of correctness properties during traffic redistribution and instance/component failures. NFVactor [32] employs the distributed actor model to support per-flow abstraction and provides APIs for implementing resilient VNFs. SFC-Checker [114] is a diagnosis framework to verify if correctness of SFC forwarding behaviors. It consists of an abstract model to manage VNF internal states and an algorithm to validate the forwarding policy under different traffic conditions. It also uses another algorithm to combine equivalent function/flow states. StateAlyzr [57] is a non-intrusive framework that automatically handles state clone and migration based on program analysis.

Scheduling

Some NFV platforms are specialized for VNF scheduling to achieve performance isolation or resource efficiency. NFVnice [61]

is a NFV framework providing fair scheduling and efficient chaining. In particular, It adopts rate-cost proportional fairness by adjusting the CPU weight of each VNF based on its estimated traffic arriving rate and service time. The scheduling is performed by tuning the OS scheduler via Linux cgroups. In addition, it actively monitors VNF backlogs and employs a back-pressure mechanism to early-drop packets for each congested service chain to spare resources. It also designs an asynchronous scheme to multiplex I/O with processing.

EdgeMiner [127] seeks to reuse the spared CPU resources of VNFs to execute data processing applications in the network edge. EdgeMiner chooses interrupt-based I/O for VNFs, instead of polling, to save CPU resources under low workload. It also employs a back-pressure scheme to dynamically detect service chain overloads and puts upstream VNFs into sleep to harvest the otherwise wasted CPU cycles. UNS [18] is a scheduling system tailored for poll-mode DPDK-based VNFs. For each worker core, it subsequently retrieves statistics from the inter-VNF buffer of the assigned SFCs and makes scheduling decisions based on buffer occupancy. The scheduling is non-intrusive as UNS just tunes parameters of the Linux Realtime Scheduling without rewriting the VNFs. SNF111The SNF here is not the same as the other SNF [54] [109] leverages serverless computing for stateful VNFs. It dynamically traces the workload demands for each VNF and allocates compute resources at fine granularity. A peer-to-peer in-memory store is deployed to proactively replicate the states and reduce packet processing latency. ResQ [112]

is a cluster-based resource management framework with guaranteed service layer objectives. It consists of a performance profiler and a scheduler. The profiler performs a set of experiments on the target VNFs to construct profiles. Based on the profiling results, the ResQ scheduler computes a resource-efficient allocation using a greedy approach. ResQ also periodically solves a Mix Integer Linear Programming (MILP) formulation to obtain the optimal allocation, which can be applied to substitute the current allocation if a pre-defined threshold is exceeded.

NetContainer [47] aims at exploiting cache locality to achieve maximum throughput and low latency for containerized VNFs. The authors firstly identify the random page allocation policy as root cause of cache pollution. Then they build an estimation model based on the footprint theory to infer the cache access overhead and model the cache mapping problem as a Minimum Cost/Maximum Flow (MCMF) problem to decide the optimal memory buffer mappings. NFV-throttle [21] controls VNF overload by spreading software modules in NFV infrastructure. These modules dynamically monitor system conditions and selectively drop excessive packets to prevent VNFs from being overwhelmed. To ensure strict processing isolation for co-located containerized VNFs, Iron [58] introduces an enforcement mechanism to account for the time each VNF spent in kernel stack. Then it throttles or even drops the packets for the aggressive VNFs through Linux scheduler or a hardware-based approach.

Profiling

Several platforms are dedicated to performance characterization. NFV-vital [15] is among the earlist efforts towards VNF performance characterization. NFV-vital consists of four components that can be seamlessly integrated into a ETSI-compatible NFV platform. With NFV-vital, users can specify their deployment and workload configurations, which are interpreted by the NFV-vital orchestrator to setup VNF and generate the workload accordingly. The orchestrator also receives statistics at runtime and performs posttest analysis. This design pattern is also adopted by ensuing platforms (e.g. [100], [29]). ConMon [80] is a distributed framework to monitor the performance of containerized VNFs. Symperf [93] predicts VNF performance through code analysis. KOMon [40] is a kernel-based online monitoring tool to measure packet processing times imposed by the target VNF. Instead of merely focusing on individual VNFs, there is an assemblage of platforms capable of characterizing performance for SFCs. For example, SFCPerf [102] is purposed for automated SFC performance evaluation. Similar to NFV-vital, it has a control module that receives an user-specified configuration file and deploys the corresponding service chain in the target infrastructure. The control module subsequently collects concerning statistics to perform data analysis and visualization. NFVPerf [82] is able to detect performance bottlenecks on a service chain by monitoring inter-component communication. Perfsight [117] also aggregates execution information from various components along the datapath to diagnose performance issues. VBaaS [101] profiles SFC performance for distributed NFV infrastructure.

However, none of the aforementioned work is suitable to measure the performance of high-speed VNFs running at multi-Gigabyte rate. OPNFV Barometer [9] is designed to monitor the performance of DPDK-accelerated VNFs. It can be attached to the target VNF as a secondary process to gather shared processing information. NFV-VIPP [27] can be integrated into DPDK-accelerated data plane to collect execution metrics and demonstrate internals of an NFVI node. BOLT [49] defines the concept of performance contract, which expresses the expected VNF or SFC performance as a function of critical parameters (e.g. execution instructions, CPU cycles, memory accesses). DeepDiag [42] monitors the runtime queuing statistics for each VNF and constructs an online impact graph to diagnose the cause of performance degradation. CASTAN [88] is able to parse VNF code and automatically generate the worst-cast workloads that lead to degraded performance. It adopts symbolic execution to identify the worst code path and a CPU cache model to determine the specific memory access pattern that causes L3 cache invalidation. In its current form, CASTAN has successfully analyzed a dozen of DPDK-based network functions.

Secure execution

There is another group of platforms specifically devoted to developing secure VNFs for execution in untrusted environments. vEPC-sec [94] incorporates a variety of traffic encryption, validation, and monitoring schemes to safeguard cloud-based LTE VNFs. SplitBox [5] distributes VNF functionalities to multiple cloud VMs to obscure its internals from public cloud. Embark [62] allows VNFs to operate on encrypted data leveraging a special HTTPS encryption scheme. BSec-NFVO [95] introduces a blockchain-based architecture to protect NFV orchestration by auditing all the operations over the SFCs. Other platforms exploit Intel® Software Guard Extensions (SGX) [73] instruction codes to secure VNFs from memory reading attacks. In specific, S-NFV [108] concentrates on the protection of VNF states by stashing them into the shielded SGX memory region (enclave) to prevent unauthorized access or snooping. TrustedClick [22] and ShieldBox [113] extend the Click modular router to secure packet processing within SGX enclave, and rely on SGX remote attestation to verify code correctness. ShieldBox additionally integrates DPDK for high-speed packet processing and ring buffers to support SFC deployment. However, neither of them protects VNF states. Based on these previous endeavors, SafeLib [71] plans for a generic platform that can offer comprehensive VNF protection, including user traffic, VNF code, policy, and execution states. The authors propose to integrate DPDK and libVNF to support TCP functionalities without compromising performance. Its implementation is currently underway. Safebricks [90] and LightBox [30] also strive for comprehensive protection while sustaining reasonable performance. LightBox implements a virtual interface to ensure secure packet VNF I/O in SGX enclaves, and adopts mOS stack to support stateful VNFs. It also implements a flow state management scheme by caching the states for active flows inside enclaves while encrypting and storing states of other flows in untrusted host. A space-efficient hash algorithm is also incorporated for efficient flow classification. SafeBricks relies on the NetBricks platform for packet processing and VNF code protection. It partitions VNF code to minimize the trusted compute base in enclaves and performs packet exchanges across the trust boundary through a shared memory mechanism. It also supports deploying an entire SFC inside an enclave and leverages Rust primitives to isolate the VNFs. Note that it is unclear if SafeBricks supports TCP functionalities.

3.3 NFVI acceleration

With the rapid growth of traffic volume, NFV platforms are expected to deliver service without compromising performance, even under immense data rates. As the data plane, NFVI is inarguably the principle point of optimization. There are many endeavors concentrating on NFVI acceleration.

Single NFVI-PoP optimization

NetVM [48] aims at achieving high performance, flexible deployment, easy management, and guaranteed security altogether. It achieves line-rate processing through a zero-copy packet delivery mechanism based on shared memory, and relies on a hypervisor-based switch to flexibly steer traffic between VNFs and the network. To ensure security, NetVM defines multi-level trust domains to limit the memory access of untrusted VNFs. A control plane is also implemented to facilitate system management by making decisions either locally or remotely. OpenNetVM [131] is based on NetVM architecture, but it adopts lightweight Docker containers to wrap VNFs. It also achieves more flexible traffic steering as both the VNFs and management entities can make routing decisions. ClickOS [72] targets a high-performance, flexible, and scalable NFV platform with guaranteed resource/performance isolation and multi-tenancy. It utilizes the Click Modular Router [59] to build a wide range of VNFs in Xen-based uni-kernel VMs. A set of optimizations is performed on the hypervisor datapath to boost the performance. The ClickOS VMs are instantiated from small images and take milliseconds to boot. Similarly, HyperNF [121] aims at achieving high performance and resource utilization. It advocates the consolidation of VNFs to share CPU cores and uses hypervisor-based virtual I/O to reduce synchronization overhead. Another platform akin to such design architecture is CliMBOS [37], which is based on Xen and ClickNF [38], a DPDK-accelerated Click router equipped with a modular TCP stack. It is devised to help developers construct lightweight, isolated, and modular IoT backends. MVMP [133] is also based on DPDK and containers. It uses a virtual device abstraction layer for VNFs to multiplex the physical NICs and steer traffic. NFF-Go [83] is designed to build and deploy network functions in the cloud. It leverages DPDK to accelerate packet I/O and uses the Go language to facilitate development. The Go language is also expected to improve concurrency and guarantee safety. Moreover, it contains a scheduler to dynamically scale packet processing based on current workload.

Some platforms mainly focus on SFC deployment and optimization. In Flurries [129], the authors introduce the concept of per-flow service provisioning, and implement a container-based NFV platform named Flurries that provides flexible service function chaining. In Flurries, each flow can be allocated to a corresponding SFC to allow for flow-level service customization and isolation. A combination of polling (for NICs) and interrupt (for VNFs) I/O scheme is also implemented to consolidate the large number of per-flow service chains on the server. Microboxes [67] concentrates on transport- and application-layer protocol consolidation for SFCs. It implements a modular, asynchronous TCP stack that can be customized on a per-flow basis to avoid redundant protocol processing on a service chain. It also provides a publish/subscribe-based communication mechanism to chain network functions and realize complex network services. SNF [54] is also implemented to eliminate redundant processing for SFC. It uses graph composition and set theory to decide the traffic classes for incoming packets before synthesizing per-class, functionally-equivalent, and optimized network functions. Likewise, NFCompass [46] also strives for shortening service chain by synthesizing VNFs and exploring parallelized execution. It additionally implements a graph-based model to minimize data transfer and balance traffic load. SpeedyBox [51] utilizes a table-based Match/Action technique to consolidate VNF actions at runtime and eliminate redundant processing along a service chain. ParaBox [132] explores to shorten SFCs by exploring parallelized VNF execution. It consists of a dependency analysis module to determine if some VNFs that can run in parallel, mirror/merge functions to distribute and aggregate packet copies across the parallelized VNFs. Similarly, NFP [111] also aims at exploring VNF parallelism for SFCs. It allows network operators to express SFC intent as policies, and implements an orchestrator to analyze VNF dependencies and compile the policies into optimized network service graphs. The NFP infrastructure sequentially takes care of the execution of the service graphs and deals with low-level details such as traffic steering, load balancing, as well as the paralleled VNF execution. Dysco [125] explores session protocol mechanism for runtime reconfiguration of TCP service function chains with no packet loss and minimal service interruption. MiddleClick [7] aims at building high-speed, parallelized service chains. It allows network operators to define SFC intents which are synthesized into a flow table and managed by the framework. A session abstraction is also implemented to facilitate per-flow inspection. ESFC [105] is designed for flexible SFC resource management. It implements a controller to monitor the VNF status and enforce resource allocation policies using an asynchronous notification mechanism. A hash algorithm is devised to balance packets across VNF replicas while ensuring flow-level affinity. SCC [55] collects system statistics such as hardware/software performance counters, and identifies root causes of excessive SFC delays. Based on the profiling results, the SCC runtime addresses the performance bottlenecks accordingly by performing a set of optimizations including tuning the I/O batch size, adjusting scheduling policies, priorities or time slices of the SFCs.

Hardware-assist design

P4SC [16] and P4NFV [44] explore P4 language to accelerate SFC processing. P4SC leverages the P4 to construct and consolidate SFCs. It parses the SFC policies specified by network operators and converts them into a P4 program, which is subsequently deployed onto the P4-compatible hardware. P4NFV is designed for both hardware and software targets, and supports runtime reconfiguration without violating state consistency. Albeit augmented with various software acceleration techniques, CPU cores might still fall short of performance. As a result, several platforms explore other hardware components for processing acceleration. OpenANFV [39] features automated provisioning and elastic management of network services. To overcome the performance issue incurred by computation-intensive VNFs, it delegates a subset of functionalities to programmable hardware. UNO [63] targets the SmartNICs (i.e. ASIC, FPGA, System on Chip) for computation offloading without violating the interoperability with existing orchestration plane. While still relying on centralized orchestrator to make global decisions, UNO selectively places new VNFs on the underlying SmartNICs to minimize host CPU usage, based on a placement algorithm taking local system status into consideration. It also actively reruns the algorithm and adjusts VNF placement between host and SmartNICs. To hide the complexity of SmartNICs from the remote orchestrator, UNO exposes a single-switch abstraction that correctly maps traffic steering rules to the host or SmartNIC switches. NICA [34] is a hardware-software co-designed platform for inline data path acceleration on SmartNICs with integrated FPGAs (F-NICs). The platform exposes a programming abstraction to give applications direct control over F-NIC accelerators and a I/O path virtualization to let multiple VMs share the F-NIC with guaranteed security and fairness. FlowShader [124], GPUNFV [123], Gen [134], Grus [135], and G-NET [126] seek to construct a concerted CPU-GPU pipeline to expedite SFC processing. FlowShader explores GPU and CPU for paralleled processing. It leverages the standard Linux TCP/IP stack to classify incoming traffic, and specialized data structures to buffer messages and maintain per-flow information. After a batch completes, it invokes a flow scheduling algorithm to balance the buffered data to GPU or CPU for processing. Notably, each GPU thread executes the entire processing logic of a VNF (or SFC) to ensure flow-level parallelism. FlowShader further provides a general API to develop compatible VNFs between CPU and GPU domains. GPUNFV also employs flow-level parallelism and wraps a whole SFC inside a single GPU thread. Compared with FlowShader, it only exploits GPU for processing, yet devotes CPU for kernel-bypassing packet I/O and flow classification based on the 5-tuple. Gen features the dynamic scheduling of GPU threads to elastically scale VNF instances. It also supports runtime SFC modification by exploring features of the CUDA library. In addition, it maintains a connection with a remote controller to orchestrate SFC execution. Grus reduces the processing latency through a set of datapath optimizations, including coordinated access of the PCI-E bus between CPU and GPU, fine-grained scheduling of consolidated VNFs, and dynamic batching. G-NET deploys network functions in VMs and offloads VNF processing to GPU. It manipulates the GPU context to allow for spatial GPU sharing across manifold VNF kernels and leverages safe pointers to guarantee GPU memory isolation. A scheduling algorithm is designed to calculate per-SFC cost and optimize the GPU resource sharing.

4 Design Space

In this section, we explore the design space, identify the critical design issues, and summarize different choices adopted by existing NFV platforms. Specific design choices of each platform have been illustrated in Table 1. Note that in general there is no absolutely superior choice over the others, and it is only a matter of use cases and application context.

Platform High-level Development TCP Packet Local VNF Virtualization Parallel Source
abstraction language I/O interconnect technique execution code
OpenBox C++, Java, Python Kernel VM
E2 F#, Python DPDK BESS
SDNFV C, Python DPDK OF table VM
Slick Python Kernel Custom switch Process
MicroNF C DPDK Open vSwitch VM+Container
OpenNF C++, Java Kernel
Split/Merge C, Python Kernel Open vSwitch VM
TFM C++, Java Kernel Open vSwitch
xOMB C++, C Kernel Process
CoMb C++, C Kernel Click Router Process
ClickNF C++ DPDK Process
NetStar C++ DPDK Process
Netbricks Rust DPDK Process
Flick C++ DPDK Process
S6 C++ Kernel Container
ClickOS C++ netmap VALE VM
HyperNF C netmap VALE VM
StatelessNF C++, C, Java DPDK Container
libVNF C++ Kernel VALE, BESS VM, Process
DPDK
netmap
CHC C++ VMA Splitter Container
MVMP C DPDK Container
NFP C DPDK Pipeline Container
Parabox C DPDK BESS Container
NetVM C DPDK OF table VM
OpenNetVM C DPDK OF table Process/container
Flurries C DPDK OF table Container
Microboxes C DPDK OF table Container
NFVNice C DPDK OF table Container
Metron C++ DPDK FastClick Process
UNO C, Java DPDK OVS-DPDK Process
GPUNFV C++, C DPDK - Process
Gen C++, C DPDK - Process
Grus C++, C DPDK - Process
FlowShader C++, C Kernel - Process
G-NET C++, C DPDK Custom switch VM/Process
NF C, C++, Python DPDK Ring buffer Process/container
NFMorph C DPDK Ring buffer Process
Eden C Kernel OF tables Process
SCC C++ DPDK FastClick All
MiddleClick C++ DPDK FastClick Process
netmap
UNiS C++ - Ring buffer Process
NetContainer C Kernel Container
NICA C, C++ VMA VM
Polycube C eBPF VM
FlowOS C Kernel VM
ShieldBox C++, C DPDK Ring buffer Container
LightBox C Kernel Process
SafeBricks Rust DPDK Process
Table 1: Design choices of the most representative NFV platforms

4.1 MANO plane

High-level API

Most of the existing NFV frameworks provide high-level APIs to specify service policies or smooth the process of VNF development. These APIs can be generally come as either Domain-Specific Language (DSL) or General-Purpose Language (GPL). GPLs such as C, C++, Java, and Python are mature programming languages capable of solving problems in multiple domains. They are shipped with multitudinous control primitives, miscellaneous data structures, and flexible operating patterns. Most of existing NFV platforms adopt GPL. For example, OpenNF relies on a collection of northbound C++ functions to develop control applications. NFVNice exposes a C library named “libnf” to perform I/O operations asynchronously and to monitor the workload (e.g. arrival rate, processing time) for each VNF. OpenBox exposes its northbound Java API for network operators to specify processing logic and subscribe to particular events. Slick provides a programming abstraction that allows developers to write custom VNFs and specify traffic steering policy in Python. ClickNF provides a standard socket API and a zero-copy interface as C++ functions to interact with its transport layer. S6 exposes a programming API to manipulate states across the shared object space. NICA introduces the “ikernel” programming abstraction over TCP/UDP sockets to facilitate user-space VNF development. GPUNFV exposes CUDA-based API to assist per-flow state abstraction and construct GPU kernel code. FlowShader and G-NET provide a CUDA-based API to develop VNFs compatible with both GPU and CPU semantics. Compared to GPLs, DSLs provide higher-level optimized abstractions for specific problems, and they usually operate at an environment with limited operation patterns and restricted resource usage. Several platforms incorporate DSL for network service development. For example, authors of [1] propose flick language that supports parallel execution and safe resource sharing. In addition to basic primitives such as event handling and common datatypes, flick can deserialize input packets into application-specific datatypes and vice versa, bringing application semantics into VNF development. E2 allows network operators to specify SFCs using a policy language, SONATA’s development toolchain allows developers to specify network services using DSL. On Eden platform, network operators specify service policies in the F# language, as it makes the VNF safety checking process straightforward.

Placement

During the VNF placement phase, existing NFV platforms usually conduct a set of pre-processing operations to merge and shorten SFCs before actually placing them to the anticipated NFVI PoPs with specific objectives. For instance, based on the specified network service description and infrastructural specification, CoMb seeks to consolidate each SFCs on the a single NFVI-PoP by solving an optimization model. OpenBox relies on a graph merging algorithm to optimally deploy the VNFs to the user-specified NFVI-PoPs. Through the OpenBox northbound interface, a developer can specify her VNF in the form of a processing graph along with the intended deployment domain or NFVI-PoP. The OpenBox controller parses the graphs intended for the same location and merges them into a single graph without violating their processing logics. Slick provides for a holistic solution for VNF placement and traffic steering. The Slick controller employs an inflation heuristic to consolidate VNFs with minimum cost, and uses a placement algorithm to deploy the consolidated VNFs. Traffic steering rules are also configured on each switch to realize the intended routes and processing sequence for each flow. E2 employs similar approaches to merge and synthesize multiple service graphs to reduce processing redundancy. Then it models VNF instance placement as a graph partition problem over the COTS servers and employs a modified Kernighan-Lin algorithm to minimize the inter-server traffic. SDNFV formulates the service placement problem as a mixed integer linear program (MILP) with the objective of maximizing resource utilization. Then the authors develop a heuristic algorithm to solve the placement of VNFs while configure the related traffic routes based on the service graph specified by network operator. Metron uses SNF to optimize its input processing graph and construct a synthesized graph, which is subsequently split into a stateful subgraph and a stateless subgraph. The stateful graph is deployed on COTS servers selected by Metron’s server selection scheme. The stateless graph is then offloaded to network elements based on the deployment locations of the stateful graph. MicroNF performs dependency analysis for the elements instead of VNFs, and reconstruct the service graph to reduce redundant processing and improve resource efficiency. It subsequently places the modularized SFCs to the COTS servers by resolving a 0-1 Integer Programming with the objective of minimizing the inter-VM overhead. The NF orchestrator also constructs an optimal forwarding graph by consolidating same types of VNFs, but the objectives based on which the VNFs are placed across COTS servers are not indicated by the authors.

State coordination

With the proliferation of stateful VNFs, it is hence critical to keep the processing states consistent upon instance scaling. However, it is extremely challenging to simultaneously satisfy all the state management requirements, e.g. flow affinity preservation, timely state synchronization, correct processing, minimal service interruption, etc, especially upon VNF scaling or instance failover. Strategies adopted by existing NFV platforms can be generally classified to state migration and migration avoidance. Among the platforms adopting state migration, OpenNF provides the most robust scheme by allowing control applications to move, copy, and share states with different granularities between two VNF replicas. In specific, it implements state move and copy operations to facilitate state migration. The move operation applies cautious coordination between source/destination instances and the last shared on-path OpenFlow switch to achieve lossless, order-preserving state migration. The copy operation allows for state clone in an eventually consistent manner, while the share operation guarantees strong consistency by instructing the controller to capture all events from the last shared switch, sending them to the corresponding VNF instances for processing, and applying state update sequentially in global scope. MicroNF, UNO, and OpenBox also advocate this solution for state coordination. Split/Merge also relies on SDN mechanism to migrate partitioned states and flows across multiple VNF replicas. Upon a scaling decision, it firstly instructs SDN to suspend traffic flow for all replicas, then transfers relevant states across the replicas and configures routes for the affected flows, and finally resumes the traffic flow. This approach preserves flow affinity, but the in-transit packets for the affected flow are lost. Metron migrates states by traffic classes. When a SFC is overloaded, Metron splits its traffic classes into two groups, and simply duplicates only states of the migrated group to the new SFC instance. Compared to OpenNF, this scheme might result in runtime inconsistency and waste of memory. TFM migrates states between two VNFs by instantiating a TFM box at each of them. Then TFM controller transmits flow states to the destination VNF and updates routing tables of the in-path switches to redirect incoming packets of the migrating flows to the destination VNF. The TFM boxes at both VNFs buffer all the in-transit packets, which are fed to the destination VNF according to receiving order after state migration. In contrast, some other platforms adopt the migration avoidance strategy to avoid the state migration overhead. For instance, upon scaling out, E2 splits incoming traffic based on their flow identifiers (e.g. 5-tuple) on the original VNF and identifies new flows that need to be directed to the newly instantiated VNFs. Then the E2 manager configures routing tables of the in-path software and hardware switches to steer new flows from the original VNF to new instances. Existing flows will keep being served by the original VNF until termination. Although traffic direction phase might incur extra delay, the authors believe that this overhead is merely transient and this approach outperforms state migration approaches. MicroNF employs a ”Push-Aside” scaling strategy in which the overloaded element kicks its upstream/downstream element to the neighboring VMs to spare resources. Therefore, instead of migrating states, MicroNF opts for moving VNFs’ elements, avoiding state migration. But it is unclear whether the element migration phase can cause any data loss or inconsistency. In fact, the most common migration avoidance strategy is externalizing the processing state. StatelessNF, CHC, and libVNF keep VNF processing states in high-speed external data stores to avoid the migration cost. In particular, libVNF maintains key-value data stores at multiple levels: a local data store shared by all threads of a single VNF, a global store shared across multiple VNF replicas, and a local cache for recently accessed data from the global store.

Scheduling

Another critical challenge is how to dynamically schedule the VNFs or SFCs to achieve fairness, isolation, and efficiency. We specifically consider the case of VNFs being collocated on the same CPU core, which is the most likely resource bottleneck [76, 18, 61]. MicroNF employs an incremental scheduling algorithm that tries to match the packet arrival and processing rates for each consolidated element. In particular, the MicroNF scheduler calculates the CPU quota based on current buffer occupancy and arrival rate, and assigns the quota to individual element in the next scheduling period. In Flick, tasks of an VNF employ cooperative scheduling by spontaneously yielding the core to other tasks under a fixed time quota.

4.2 VNF plane

To implement efficient VNFs, NFV platforms are required to consider several critical design choices.

Execution model

Figure 3: Execution models of a simple VNF instance sitting between two physical NIC. It consists a sequence of three elements, namely RX, processing, and TX.

In NFV domain, the VNF (or SFC) execution model can be classified into run-to-completion (RTC) and pipeline. In RTC model, all the VNFs of a given SFC run on a single thread, whereas in pipeline model, each VNF instance is pinned to a separate thread, as illustrated in Fig. 3. Performance of either model is highly dependent on processing complexity and input workload, which lead to different level of cache and memory access patterns. In general, RTC model demonstrates better throughput and latency when executing simple VNFs and/or short SFCs, since it obviates the inter-core transfer overhead [53]. It may also require fewer worker cores than pipeline model. However, the edge of RTC fades with complex VNFs and/or lengthy SFCs, in which cases the large amount of data/instructions and context switches might precipitate persistent L1/L2 cache misses. VNFs running in RTC model are also more difficult to be individually scaled based on input load. As a result, there is no absolutely superior solution and distinctive NFV platforms usually make choices based on their specific architectures. Some platforms adopts RTC model mainly because they only execute lightweight VNFs or trimmed SFCs in software. For instance, NetBricks, ClickNF, ClickOS, NetStar, and SafeBricks execute VNFs as processes in RTC model to avoid inter-core transfer overhead. Metron offloads part of its SFCs to the in-path hardware equipments, executing the trimmed SFC tasks in RTC model on COTS servers. CoMb executes an entire VNF or SFC in a single core to avoid inter-core synchronization overhead. GPU-based solutions also adopt RTC model due to the intrinsic inefficiency of pipeline model on GPU architecture [134]. Consequently, FlowShader, GPUNFV, and Gen embed one integral SFC in a single GPU thread. Other platforms employ the pipeline model. For example, NF and MicroNF decompose VNFs or SFCs into modular, fine-grained, loosely-coupled packet processing tasks (or elements), so that resources can be precisely allocated to scale individual elements. Flick allows for pipelined execution of individual tasks of its VNFs. The poll-mode VNFs scheduled by UNS also run in pipelines constructed with DPDK ring buffer. ShieldBox also adopts DPDK ring buffers to VNFs across multiple SGX enclaves. StatelessNF executes VNFs in pipelines. Each pipeline consists of a polling thread, a lockless queue, and a processing thread.

TCP functionality

As stateful VNFs have become an important building block in NFV ecosystem, it is worth pointing out existing platforms that implement or integrate TCP/IP stack to support stateful VNFs at layer 4 or beyond. ClickNF is equipped with a full-fledge modular TCP stack to facilitate the end-host application development. Microboxes comes with a modular, customizable TCP stack that can be shared among a group of VNFs to eliminate redundant processing. NICA even implements a simplified TCP stack in SmartNICs to enrich its in-path processing features. xOMB stack also implements simple functions such as TCP connection termination. Instead of developing TCP functionalities from scratch, some platforms choose to directly incorporate third-party solutions. For example, Flick integrates the high-speed kernel-bypassing mTCP stack [50] to realize transport layer VNFs, NetStar directly employs a third-party user-space TCP stack with future/promise abstraction. libVNF is designed to be generic by integrating both the standard networking stack and mTCP. Besides, all the NFV platforms using the kernel TCP/IP stack are granted TCP functionalities by default.

Vnf I/o

Just like the execution model, there are also two alternative means for VNFs to perform packet I/O, namely Polling and Interrupt. VNFs running in polling mode keep querying the NICs or upstream VNFs for data, which normally renders better performance at the cost of wasted CPU cycles and increased energy consumption. Interrupt-based I/O usually does not entail wasted resources, but suffers from performance losses due to interrupt propagation delay and cache line warm-up. In existing NFV platforms, UNO, CHC, and StatelessNF execute VNFs in poll-mode to enhance performance. UNiS is tailored to schedule poll-mode VNFs by manipulating the Linux realtime scheduler. Flick, ClickNF, NFVNice, xOMB, and libVNF execute VNFs in interrupt mode.

Secure execution

At present, it is increasingly common for VNFs to be delegated for execution in untrusted environments such as public or third-party clouds. Consequently, both traffic data and VNF information are exposed to different cyber attacks. Existing NFV platforms generally secure the execution of VNFs with either Encryption approach or Shield Execution. Platforms adopting encryption approach usually leverage various cryptographic schemes to allow VNFs to operate directly on encrypted network traffic. On the other hand, platforms adopting shield execution commonly run VNFs in private memory regions called enclaves. Contents inside an enclave is strictly protected and cannot be accessed by any external process. Compared to shield execution, encryption approaches generally incur higher overhead, imposed by the complex cryptographic computation, and support a limited set of functionalities. The only advantage is that they are not reliant on trusted hardware such as Intel SGX.

4.3 Nfvi

Virtualization technique

As the central point of any NFV platform, existing implementations typically deploy network functions inside Virtual Machines (VMs), Containers, or simply as bare-metal Processes.

Figure 4: State-of-the-art virtualization techniques

As illustrated in Fig. 4, they realize virtualization at different layers in the commodity hardware infrastructure, and therefore present different degrees of isolation and resource requirements. VM is a hardware-level virtualization technique that relies on the Virtual Machine Monitor (VMM) or hypervisor, a firmware that offers a virtual platform to accommodate a variety of guest operating systems, on which different applications or VNFs execute222Note that there is another kind of hypervisor that operates at OS-level (namely hosted hypervisor), we exclude them from our discussion as they are rarely used by existing NFV platforms.. Hypervisors also supervise and coordinate the VM instances for efficient hardware resource sharing across them. As the most traditional approach for virtualization, VM-based NFV platforms are commonplace. For example, NetVM, NICA, and SDNFV exploit KVM-based VMs, Split/Merge, HyperNF and FlowOS build VNFs inside Xen-based VMs. However, traditional VMs commonly incur heavy resource demands, huge memory footprints, and high migration cost. To cope with these issues, ClickOS and CliMBOS adopt unikernel VMs, which are essentially minimalistic VMs that are small, agile, and fast-to-boot. The advent of containerization techniques, such as LXC and Docker, bestow another option. Compared to VM, container is an OS-level virtualization technique with much smaller memory footprint and shorter instantiation time, much higher deployment density (up to thousands per server), and lower redistribution cost. Nonetheless, containers are not completely insulated from the host OS, they thus cannot grant the same level of isolation and security as VMs. In terms of performance, both of them can sustain line rate processing leveraging particular I/O paths and networking techniques [128]. At present, many existing NFV platforms opt for containerized VNFs. For instance, OpenNetVM, Flurries, MicroBoxes, NFVNice, MVMP, NFP, ParaBox, MVMP, statelessNF, and GNFC execute their VNFs inside Docker containers. CHC and Iron run VNFs in LXC containers. Aside from VMs and containers, a set of NFV platforms deploy VNFs as processes or threads to trade isolation for performance. These platform usually assume trusted VNFs and infrastructure. For example, NetBricks, ClickNF, libVNF, and SafeLib executes VNFs as processes. OPNFV barometer is designed to profile performance of the VNFs running as DPDK processes. SplitBox executes VNFs as processes inside FastClick. GPUNFV, Gen, FlowShader, Grus, and G-NET execute VNFs or SFCs as GPU threads. Notably, some platforms opt for multiple choices or introduce hybrid solutions. For example, NFV-VIPP can be integrated to any DPDK-accelerated VNF running ins VMs, containers, or as processes. VNFs in OpenNetVM and

NF can be deployed either as processes or inside containers. MicroNF even runs containers inside VMs probably to improve security.

VNF interconnects

NFV platforms achieve cost-efficiency by consolidating multiple instances of VNFs inside individual COTS servers. These VNFs are then purposely concatenated to form specific SFCs. Existing NFV platforms interconnect local VNFs using either state-of-the-art Software Switches or inventing custom mechanisms. Software switches are widely used as NFV data plane for efficient traffic steering. For instance, E2 chooses an augmented version of BESS [43] as data plane, while ClickOS and HyperNF extends VALE switch [98]. CoMb customizes the Click Modular Router [59] to classify and forward packets between VNFs of the same service chain. Metron, MiddleClick, SCC, and SplitBox leverage FastClick [8] to ferry packets between VNFs and the network. Split/Merge, TFM, and MicroNF employ Open vSwitch (OVS) [89] for VM-networking. UNO extends OVS-DPDK [85] to steer packets at both host and SmartNIC level. More details about performance of the aforementioned software switches can be found in [128]. Rather than adopting third-party solutions, G-NET uses a bespoke software switch to route packets between VNFs and physical NICs.

Packet I/O frameworks

Figure 5: Packet I/O: Kernel-based vs. Kernel-bypassing

Packet I/O frameworks can be classified into Kernel-based and Kernel-bypassing approaches, as illustrated in Fig. 5. Traditionally, network applications rely on the general-purpose OS kernel stack for packet I/O. For instance, Eden, FlowOS, and NetContainer employ kernel-based I/O to seamlessly utilize the rich features of the kernel stack. However, the OS kernel imposes non-negligible overhead on packet data path, making software applications fail to sustain high-speed processing [8]. To overcome this bottleneck, kernel-bypassing frameworks such as DPDK [25] and netmap [99] are proposed. They commonly embody data path optimization routines such as zero-copy delivery to avoid staging packets in kernel space, preallocated packet buffers to eschew runtime memory allocation, and batch processing to amortize the overhead of accessing hardware. netmap is compatible with standard kernel stack as it still incorporate system calls for data validation and interrupt-based packet reception. DPDK employs complete kernel bypassing and dispatches poll-mode drivers to enhance performance. DPDK further exposes a rich collection of APIs and primitives to simplify application development. As shown in Tab. 1, most of the existing NFV platforms adopting DPDK for packet I/O from physical NICs. For example, Flick, ClickNF, NetStar, NetVM, OpenNetVM, CoNFV, OPNFV barometer, NFV-VIPP, SafeLib, SplitBox, UNO, GPUNFV, Gen, Grus, G-NET all leverages DPDK for packet I/O. netmap is used by ClickOS and HyperNF. NICA and CHC rely on Mellanox Message Accelerator (VMA) [66], another kernel-bypass packet I/O framework with standard POSIX socket APIs and user-space networking library. Note that even if the traditional kernel-based approach fails to render comparable performance as kernel-bypassing stacks, it can still be useful when the VNFs are not I/O intensive or the cost to setup a kernel-bypassing stack becomes too high. Furthermore, a kernel-based high-speed packet I/O framework named eBPF is adopted by the Polycube platform for packet I/O to achieve multi-gigabyte processing. libVNF is a general platform employs kernel, DPDK, and netmap altogether, so that users can make choices according to their respective needs.

4.4 Other design choices

As discussed in our previous work [65], there is a large assortment of acceleration techniques for high-speed packet processing. In this part, we pick the most commonly utilized techniques and enumerate their adoption by existing NFV platforms. The optimizations we consider including zero-copy, batching, memory pre-allocation, instruction/data prefetching, parallel execution, CPU cache optimization, and computation offloading. Although these optimizations are commonly applied by the preceding packet I/O techniques, we discuss their applications in other parts of the NFV architecture.

Zero-copy

In high-speed packet processing domain, runtime memory copy is an expensive operation that usually leads to unbearable overhead. For the sake of performance, many existing NFV platforms deliver packets across VNFs or memory boundaries in a zero-copy manner, by copying only their associated packet descriptors. For example, NF implements a zero-copy port abstraction that only exchanges packet addresses instead of copying full packets between VNFs. The TCP stack of ClickNF exposes zero-copy interfaces to interact with user-space VNF. NICA leverages ring buffers for zero-copy message exchange between the F-NIC units and the user-space VNFs. GPUNFV achieves zero-copy packet delivery across CPU and GPU boundary through CUDA’s page-lock memory. G-NET’s switch also employs zero-copy design.

Batching

In high-speed packet processing frameworks, I/O batching is widely used to amortize the overhead of accessing the physical NIC over multiple packets. This technique is also employed by some NFV platforms to enhance performance. For example, NFVNice and EdgeMiner batch the I/O interrupts to amortize VNF wakeup overhead. SCC handles the system calls of VNF I/O in dynamic batches to reduce the overhead of context switches. The VNFs on NF platform perform packet I/O in batches through the intermediate ring buffers. The TCP stack of ClickNF exchanges packets with the user-space VNFs in batches. StatelessNF aggregates multiple read/write requests to the data store into a single request to amortize the overhead of remote procedure call (RPC). SafeBricks implements an in-enclave module for batched packet I/O from/to the host. LightBox adopts packet batching to amortize the system call overhead. GPUNFV, Grus, FlowShader, G-NET, and Gen deliver packets between CPU and GPU in dynamic batches.

Pre-allocation

As memory allocation at runtime remains an expensive operation. In addition to the pre-allocated packet buffers and descriptors by some packet I/O techniques, existing NFV platforms usually pre-allocate a particular memory region to stage and reuse other relevant packet processing data structures. For example, libVNF pre-allocates memory pools for its per-core, persistent request objects and lock-free packet buffers. Flick pre-allocates its task graphs and queues. S6 pre-allocates a pool of the cooperative, user-space, per-flow micro-threads to avoid the dynamic thread creation/deletion overhead. ShieldBox pre-allocates packet descriptor memory. LightBox pre-allocates state management data structures.

Parallel execution

To take advantage of the multicore CPUs, many platforms explore possibilities for parallelization. NF performs a dependency analysis on its forwarding graphs to identify parallelize VNFs. Consecutive VNFs are deemed parallelizable if they perform read-only operations or update disjoint packet regions. Then these VNFs are assigned independent CPU cores to process packets. A reference counter is attached as meta-data to avoid out-of-order operations from downstream VNFs. Likewise, SDNFV allows multiple VNFs to access a packet in parallel using a reference counter embedded in the packet descriptor. Eden exposes a concurrency model that create consistent state copies for multiple VNFs to execute in parallel in Eden enclave. CoMb allocates each SFC an independent shim layer to allow for parallel execution of multiple SFCs. Flick instantiates a new task graph for each new connection and schedules the tasks of these graphs onto multiple worker cores in parallel. ClickNF utilizes the Receive-Side Scaling (RSS) of physical NICs to distribute incoming packets to multiple cores with flow-level affinity guaranteed. NetStar builds VNFs with a share-nothing thread model and distributes incoming packets to different threads for paralleled multicore processing.

Cache optimization

Modern CPUs are equipped with hierarchical caches between its cores and the main memory. Cache misses result in extra access to other cache levels or the main memory, which significantly slows down the processing speed. Many existing NFV platforms are aware of this issue and explore opportunities for cache optimization. NetContainer aims at exploiting cache locality at inter-flow and intra-flow levels for NFV workload, and leverages page coloring technique to aggregate buffer pages to separate cache regions to avoid cache contention. ResQ exploits Intel Cache Allocation Technology with corresponding buffer sizing to eliminate last level cache invalidation while ensuring performance isolation. LightBox adapts cache line protection techniques to reduce cache miss rate. NF performs cache-line pre-fetching in batches to increase cache hit rate. Some platforms also make their critical internal data structures cache-optimized. For example, the request objects of libVNF are cache-optimized, all the per-core data structures of ClickNF are cache-aligned.

Computation offloading

Computation offloading is widely adopted by existing NFV platforms to alleviate the pressure of COTS servers. Potential resources to offload computing tasks including GPU, smartNICs, in-path programmable network equipments, or other specialized accelerators. E2 manager maintains a connection with OpenFlow switch and cognitively offloads simple VNF to unburden the servers. Likewise, Metron offloads stateless operations to the in-path programmable NICs and switches. OpenBox and Eden also support hardware implementation of their forwarding plane to accelerate processing. OpenNetVM and OpenANFV incorporate programmable NICs or FPGAs for computation offloading. ClickNF explores common NIC features to perform TCP/IP checksum offloading, TCP segmentation offloading (TSO), and large receive offloading (LRO). GPUNFV, Gen, FlowShader, Grus, and G-NET all achieve performance boost by completely or partially offloading their computation tasks to GPU. SmartNICs are commonly equipped with programmable, multi-core processors and an integrated operating system, make them ideal to execute computation tasks. UNO explores smartNICs to offload VNFs, forwarding rules, flow tables, and crypto/compression operations. NICA leverages the inline processing of FPGA on smartNICs to accelerate data plane processing. The “ikernel” programming abstraction of NICA grants user-space VNFs direct control over the computations in SmartNICs. CHC allows VNFs to offload operations to the external state store to speed up shared state updates. Another flavor of computation offloading comes with reusing a computation result. For instance, OpenNetVM and NFMorph reuse the Receive-Side Scaling (RSS) hash value computed by the NIC for traffic classification at a later stage, SDNFV caches flow table lookup results in packet descriptors to be reused by the VNFs.

Programming practices

Aside from the prior optimizations, developers are also recommended to follow several coding practices to make the underlying hardware optimization more convenient. For instance,

5 Conclusion

As a novel paradigm to shift network management and service provisioning, NFV is expected to revolutionize the next-generation telecommunication networks. To accelerate the innovation and commercial adoption of NFV, a large spectrum of platforms have been implemented in the last six years. While sharing the ultimate objective of promoting NFV, they usually tackle divergent problems in NFV eco-system and embrace disparate design choices to achieve different performance metrics or service layer agreements, and few work has been devoted to interpreting this huge collection of platform implementations. In this paper, we concentrate on existing NFV platforms and strive for comprehending their design. After introducing the NFV reference architecture, we present our taxonomy on existing NFV platforms based on primary design purposes. Then we explore the design space and investigate the various choices individual NFV platforms opt for to tackle different implementation challenges. Last but not least, we envision future challenges with respect to AI, 5G, IoT, and energy efficiency.

References

  • [1] A. Alim, R. G. Clegg, L. Mai, L. Rupprecht, E. Seckler, P. Costa, P. Pietzuch, A. L. Wolf, N. Sultana, J. Crowcroft, et al. (2016) flick: Developing and running application-specific network services. In USENIX ATC, Cited by: §3.1, §4.1.
  • [2] O. Alipourfard and M. Yu (2018) Decoupling Algorithms and Optimizations in Network Functions. In ACM HotNets, Cited by: §3.1.
  • [3] J. W. Anderson, R. Braud, R. Kapoor, G. Porter, and A. Vahdat (2012) XOMB: extensible open middleboxes with commodity servers. In ACM/IEEE ANCS, Cited by: §3.1.
  • [4] B. Anwer, T. Benson, N. Feamster, and D. Levin (2015) Programming slick network functions. In ACM SOSR ’15, Cited by: §3.1.
  • [5] H. J. Asghar, L. Melis, C. Soldani, E. De Cristofaro, M. A. Kaafar, and L. Mathy (2016) Splitbox: toward efficient private network function virtualization. In ACM HotMiddlebox ’16, Cited by: §3.2.
  • [6] H. Ballani, P. Costa, C. Gkantsidis, M. P. Grosvenor, T. Karagiannis, L. Koromilas, and G. O’Shea (2015) Enabling end-host network functions. In ACM SIGCOMM CCR, Cited by: §3.1.
  • [7] T. Barbette, C. Soldani, R. Gaillard, and L. Mathy (2018) Building a chain of high-speed vnfs in no time,”. In IEEE HPSR, Cited by: §3.3.
  • [8] T. Barbette, C. Soldani, and L. Mathy (2015) Fast userspace packet processing. In ACM/IEEE ANCS, Cited by: §4.3, §4.3.
  • [9] Barometer Home, OPNFV wiki. Note: https://wiki.opnfv.org/display/fastpath/Barometer+Home Cited by: §3.2.
  • [10] M. Bezahaf, A. Alim, and L. Mathy (2013) Flowos: a flow-based platform for middleboxes. In ACM HotMiddlebox ’13, Cited by: §3.1.
  • [11] D. Bhamare, R. Jain, M. Samaka, and A. Erbad (2016) A survey on service function chaining. Elsevier JNCA. Cited by: §1.
  • [12] R. Bonafiglia, S. Miano, S. Nuccio, F. Risso, and A. Sapio (2016) Enabling nfv services on resource-constrained cpes. In IEEE Cloudnet, Cited by: §3.1.
  • [13] P. Bosshart, D. Daly, G. Gibb, M. Izzard, N. McKeown, J. Rexford, C. Schlesinger, D. Talayco, A. Vahdat, G. Varghese, et al. (2014) P4: programming protocol-independent packet processors. ACM SIGCOMM CCR. Cited by: §2.2.
  • [14] A. Bremler-Barr, Y. Harchol, and D. Hay (2016) OpenBox: a software-defined framework for developing, deploying, and managing network functions. In ACM SIGCOMM, Cited by: §3.1.
  • [15] L. Cao, P. Sharma, S. Fahmy, and V. Saxena (2015) Nfv-vital: a framework for characterizing the performance of virtual network functions. In IEEE NFV-SDN, Cited by: §3.2.
  • [16] X. Chen, D. Zhang, X. Wang, K. Zhu, and H. Zhou (2019) P4SC: Towards High-Performance Service Function Chain Implementation on the P4-Capable Device. In IFIP/IEEE IM, Cited by: §3.3.
  • [17] S. R. Chowdhury, Anthony, H. Bian, T. Bai, and R. Boutaba (2019) : A disaggregated packet processing architecture. In IEEE NetSoft, External Links: Document Cited by: §3.1.
  • [18] S. R. Chowdhury, T. Bai, R. Boutaba, J. François, et al. (2018) UNiS: A user-space non-intrusive workflow-aware virtual network function scheduler. In IEEE CNSM, Cited by: §3.2, §4.1.
  • [19] V. Cisco (2018) Cisco visual networking index: forecast and trends, 2017–2022. White Paper. Cited by: §1.
  • [20] (2019) CloudBand: Adopt lean operations and increase business agility. Note: https://www.nokia.com/networks/solutions/cloudband/ Cited by: §3.1.
  • [21] D. Cotroneo, R. Natella, and S. Rosiello (2017) NFV-throttle: an overload control framework for network function virtualization. IEEE TNSM. Cited by: §3.2.
  • [22] M. Coughlin, E. Keller, and E. Wustrow (2017) Trusted click: overcoming security issues of NFV in the cloud. In ACM SDN-NFV Sec’17, Cited by: §3.2.
  • [23] A. Császár, W. John, M. Kind, C. Meirosu, G. Pongrácz, D. Staessens, A. Takács, and F. Westphal (2013) Unifying cloud and carrier network: eu fp7 project unify. In IEEE/ACM UCC, Cited by: §3.1.
  • [24] R. Cziva and D. P. Pezaros (2017) Container network functions: bringing NFV to the network edge. IEEE Communications Magazine. Cited by: §3.1.
  • [25] (2020) Data Plane Development Kit. Note: https://www.dpdk.org/ Cited by: §4.3.
  • [26] N. F. S. de Sousa, D. A. L. Perez, R. V. Rosa, M. A. Santos, and C. E. Rothenberg (2019) Network service orchestration: a survey. Computer Communications. Cited by: §1.
  • [27] M. Dodare, Y. Taguchi, R. Kawashima, H. Nakayama, T. Hayashi, and H. Matsuo (2019) NFV-VIPP: Catching Internal Figures of Packet Processing for Accelerating Development and Operations of NFV-nodes. In IFIP CNSM, Cited by: §3.2.
  • [28] S. Dräxler, H. Karl, M. Peuster, H. R. Kouchaksaraei, M. Bredel, J. Lessmann, T. Soenen, W. Tavernier, S. Mendel-Brin, and G. Xilouris (2017) SONATA: service programming and orchestration for virtualized software networks. In IEEE ICC Workshops, Cited by: §3.1.
  • [29] Q. Du, Z. Ni, R. Zhu, M. Xu, K. Guo, W. You, R. Huang, K. Yin, and Q. Zheng (2018) A service-based testing framework for nfv platform performance evaluation. In IEEE ICRMS, Cited by: §3.2.
  • [30] H. Duan, C. Wang, X. Yuan, Y. Zhou, Q. Wang, and K. Ren (2019) LightBox: full-stack protected stateful middlebox at lightning speed. In ACM SIGSAC CCS, Cited by: §3.2.
  • [31] J. Duan, X. Yi, J. Wang, C. Wu, and F. Le (2019) NetStar: a future/promise framework for asynchronous network functions. IEEE JSAC. Cited by: §3.1.
  • [32] J. Duan, X. Yi, S. Zhao, C. Wu, H. Cui, and F. Le (2019) NFVactor: a resilient nfv system using the distributed actor model. IEEE JSAC. Cited by: §3.2.
  • [33] (2020) eBPF - extended Berkeley Packet Filter. Note: https://prototype-kernel.readthedocs.io/en/latest/bpf/ Cited by: §3.1.
  • [34] H. Eran, L. Zeno, M. Tork, G. Malka, and M. Silberstein (2019) NICA: An Infrastructure for Inline Acceleration of Network Applications. In USENIX ATC, Cited by: §3.3.
  • [35] (2019) ETSI - Welcome to the World of Standards!. Note: https://www.etsi.org/ Cited by: §1.
  • [36] G. ETSI (2013) Network Functions Virtualisation (NFV): Architectural framework. ETsI Gs NFV. Cited by: §2.
  • [37] M. Gallo, S. Ghamri-Doudane, and F. Pianese (2018) Climbos: A modular nfv cloud backend for the internet of things. In IFIP NTMS, Cited by: §3.3.
  • [38] M. Gallo and R. Laufer (2018) ClickNF: a modular stack for custom network functions. In USENIX ATC 18, Cited by: §3.1, §3.3.
  • [39] X. Ge, Y. Liu, D. H. Du, L. Zhang, H. Guan, J. Chen, Y. Zhao, and X. Hu (2014) OpenANFV: accelerating network function virtualization with a consolidated framework in openstack. In ACM SIGCOMM CCR, Cited by: §3.3.
  • [40] S. Geissler, S. Lange, F. Wamser, T. Zinner, and T. Hoßfeld (2019) KOMon - Kernel-based Online Monitoring of VNF Packet Processing Times. In IEEE NetSys, Cited by: §3.2.
  • [41] A. Gember-Jacobson, R. Viswanathan, C. Prakash, R. Grandl, J. Khalid, S. Das, and A. Akella (2014) OpenNF: Enabling innovation in network function control. In ACM SIGCOMM CCR, Cited by: §3.2.
  • [42] J. Gong, Y. Li, B. Anwer, A. Shaikh, and M. Yu (2019) DeepDiag: detailed nfv performance diagnosis. In ACM SIGCOMM 2019 Conference Posters and Demos, Cited by: §3.2.
  • [43] S. Han, K. Jang, A. Panda, S. Palkar, D. Han, and S. Ratnasamy (2015) SoftNIC: a software nic to augment hardware. Tech. Rep. UCB/EECS-2015-155. Cited by: §4.3.
  • [44] M. He, A. Basta, A. Blenk, N. Deric, and W. Kellerer (2018) P4nfv: An nfv architecture with flexible data plane reconfiguration. In IEEE CNSM, Cited by: §3.3.
  • [45] J. G. Herrera and J. F. Botero (2016) Resource allocation in nfv: a comprehensive survey. IEEE TNSM. Cited by: §1.
  • [46] Y. Hu and T. Li (2018) Enabling efficient network service function chain deployment on heterogeneous server platform. In IEEE HPCA, Cited by: §3.3.
  • [47] Y. Hu, M. Song, and T. Li (2017) Towards full containerization in containerized network function virtualization. ACM SIGOPS Operating Systems Review. Cited by: §3.2.
  • [48] J. Hwang, K. K. Ramakrishnan, and T. Wood (2015) NetVM: high performance and flexible networking using virtualization on commodity platforms. IEEE TNSM. Cited by: §3.3.
  • [49] R. Iyer, L. Pedrosa, A. Zaostrovnykh, S. Pirelli, K. Argyraki, and G. Candea (2019) Performance contracts for software network functions. In USENIX NSDI, Cited by: §3.2.
  • [50] E. Jeong, S. Wood, M. Jamshed, H. Jeong, S. Ihm, D. Han, and K. Park (2014) mTCP: a Highly Scalable User-level TCP Stack for Multicore Systems. In USENIX NSDI, Cited by: §4.2.
  • [51] Y. Jiang, Y. Cui, W. Wu, Z. Xu, J. Gu, K. Ramakrishnan, Y. He, and X. Qian (2019) SpeedyBox: low-latency nfv service chains with cross-nf runtime consolidation. In IEEE ICDCS, Cited by: §3.3.
  • [52] M. Kablan, A. Alsudais, E. Keller, and F. Le (2017) Stateless network functions: Breaking the tight coupling of state and processing. In USENIX NSDI, Cited by: §3.1.
  • [53] G. P. Katsikas, T. Barbette, D. Kostic, R. Steinert, and G. Q. Maguire Jr (2018) Metron: NFV Service Chains at the True Speed of the Underlying Hardware. In USENIX NSDI, Cited by: §3.1, §4.2.
  • [54] G. P. Katsikas, M. Enguehard, M. Kuźniar, G. Q. Maguire Jr, and D. Kostić (2016) SNF: synthesizing high performance NFV service chains. PeerJ Computer Science. Cited by: §3.3, footnote 1.
  • [55] G. P. Katsikas, G. Q. Maguire Jr, and D. Kostić (2017) Profiling and accelerating commodity nfv service chains with scc. Journal of Systems and Software. Cited by: §3.3.
  • [56] J. Khalid and A. Akella (2019) Correctness and performance for stateful chained network functions.. In USENIX NSDI, Cited by: §3.2.
  • [57] J. Khalid, A. Gember-Jacobson, R. Michael, A. Abhashkumar, and A. Akella (2016) Paving the Way for NFV: Simplifying Middlebox Modifications Using StateAlyzr. In USENIX NSDI, Cited by: §3.2.
  • [58] J. Khalid, E. Rozner, W. Felter, C. Xu, K. Rajamani, A. Ferreira, and A. Akella (2018) Iron: Isolating Network-based CPU in Container Environments. In USENIX NSDI, Cited by: §3.2.
  • [59] E. Kohler, R. Morris, B. Chen, J. Jannotti, and M. F. Kaashoek (2000) The click modular router. ACM TOCS. Cited by: §3.1, §3.3, §4.3.
  • [60] B. Kothandaraman, M. Du, and P. Sköldström (2015) Centrally controlled distributed vnf state management. In ACM HotMiddiebox ’15, Cited by: §3.2.
  • [61] S. G. Kulkarni, W. Zhang, J. Hwang, S. Rajagopalan, K. Ramakrishnan, T. Wood, M. Arumaithurai, and X. Fu (2017) NFVNice: Dynamic backpressure and scheduling for nfv service chains. In ACM SIGCOMM ’17, Cited by: §3.2, §4.1.
  • [62] C. Lan, J. Sherry, R. A. Popa, S. Ratnasamy, and Z. Liu (2016) Embark: securely outsourcing middleboxes to the cloud. In USENIX NSDI, Cited by: §3.2.
  • [63] Y. Le, H. Chang, S. Mukherjee, L. Wang, A. Akella, M. M. Swift, and T. Lakshman (2017) UNO: uniflying host and smart nic offload for flexible packet processing. In ACM SoCC ’17, Cited by: §3.3.
  • [64] X. Li and C. Qian (2016) A survey of network function placement. In IEEE CCNC, Cited by: §1.
  • [65] L. Linguaglossa, S. Lange, S. Pontarelli, G. Rétvári, D. Rossi, T. Zinner, R. Bifulco, M. Jarschel, and G. Bianchi (2019) Survey of performance acceleration techniques for network function virtualization. Proceedings of the IEEE. Cited by: §4.4.
  • [66] (2019) Linux user space library for network socket acceleration based on RDMA compatible network adaptors. Note: https://github.com/Mellanox/libvma Cited by: §4.3.
  • [67] G. Liu, Y. Ren, M. Yurchenko, K. Ramakrishnan, and T. Wood (2018) Microboxes: high performance nfv with customizable, asynchronous tcp stacks and dynamic subscriptions. In ACM SIGCOMM ’18, Cited by: §3.3.
  • [68] L. Liu, H. Xu, Z. Niu, P. Wang, and D. Han (2016) U-haul: efficient state migration in nfv. In ACM APSys, Cited by: §3.2.
  • [69] A. Lombardo, A. Manzalini, G. Schembra, G. Faraci, C. Rametta, and V. Riccobene (2015) An open framework to enable netfate (network functions at the edge). In IEEE NetSoft, Cited by: §3.1.
  • [70] D. Lopez (2015) OpenMANO: the dataplane ready open source nfv mano stack. In IETF Meeting Proceedings, Dallas, Texas, USA, Cited by: §3.2.
  • [71] E. Marku, G. Biczók, and C. Boyd (2019) Towards protected vnfs for multi-operator service delivery. In IEEE NetSoft, Cited by: §3.2.
  • [72] J. Martins, M. Ahmed, C. Raiciu, V. Olteanu, M. Honda, R. Bifulco, and F. Huici (2014) ClickOS and the art of network function virtualization. In USENIX NSDI, Cited by: §1, §3.3.
  • [73] F. McKeen, I. Alexandrovich, A. Berenzon, C. V. Rozas, H. Shafi, V. Shanbhogue, and U. R. Savagaonkar (2013) Innovative instructions and software model for isolated execution.. Hasp@ isca. Cited by: §3.2.
  • [74] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker, and J. Turner (2008) OpenFlow: enabling innovation in campus networks. ACM SIGCOMM CCR. Cited by: §1.
  • [75] M. Mechtri, C. Ghribi, O. Soualah, and D. Zeghlache (2017) NFV orchestration framework addressing sfc challenges. IEEE Communications Magazine. Cited by: §3.2.
  • [76] Z. Meng, J. Bi, H. Wang, C. Sun, and H. Hu (2019) MicroNF: an efficient framework for enabling modularized service chains in nfv. IEEE JSAC. Cited by: §3.1, §4.1.
  • [77] S. Miano, M. Bertrone, F. Risso, M. V. Bernal, Y. Lu, J. Pi, and A. Shaikh (2019) A Service-Agnostic Software Framework for Fast and Efficient In-Kernel Network Services. In ACM/IEEE ANCS, Cited by: §3.1.
  • [78] R. Mijumbi, J. Serrat, J. Gorricho, N. Bouten, F. De Turck, and R. Boutaba (2015) Network function virtualization: State-of-the-art and research challenges. IEEE Communications Surveys & Tutorials. Cited by: §1.
  • [79] R. Mijumbi, J. Serrat, J. Gorricho, S. Latré, M. Charalambides, and D. Lopez (2016) Management and orchestration challenges in network functions virtualization. IEEE Communications Magazine. Cited by: §1.
  • [80] F. Moradi, C. Flinta, A. Johnsson, and C. Meirosu (2017) Conmon: an automated container based network performance monitoring system. In IFIP/IEEE IM, Cited by: §3.2.
  • [81] P. Naik, A. Kanase, T. Patel, and M. Vutukuru (2018) LibVNF: building virtual network functions made easy. In ACM SOCC, Cited by: §3.1.
  • [82] P. Naik, D. K. Shaw, and M. Vutukuru (2016) NFVPerf: online performance monitoring and bottleneck detection for nfv. In IEEE NFV-SDN, Cited by: §3.2.
  • [83] (2019) NFF-Go -Network Function Framework for GO (former YANFF). Note: https://github.com/intel-go/nff-go Cited by: §3.3.
  • [84] (2019) OPEN BATON: An extensible and customizable NFV MANO-compliant framework. Note: https://openbaton.github.io/ Cited by: §3.2.
  • [85] (2020) Open vSwitch with DPDK. Note: http://docs.openvswitch.org/en/latest/intro/install/dpdk/ Cited by: §4.3.
  • [86] S. Palkar, C. Lan, S. Han, K. Jang, A. Panda, S. Ratnasamy, L. Rizzo, and S. Shenker (2015) E2: a framework for nfv applications. In ACM SOSP ’15, Cited by: §3.1.
  • [87] A. Panda, S. Han, K. Jang, M. Walls, S. Ratnasamy, and S. Shenker (2016) NetBricks: Taking the V out of NFV. In USENIX OSDI, Cited by: §3.1.
  • [88] L. Pedrosa, R. Iyer, A. Zaostrovnykh, J. Fietz, and K. Argyraki (2018) Automated synthesis of adversarial workloads for network functions. In ACM SIGCOMM ’18, Cited by: §3.2.
  • [89] B. Pfaff, J. Pettit, T. Koponen, E. Jackson, A. Zhou, J. Rajahalme, J. Gross, A. Wang, J. Stringer, P. Shelar, et al. (2015) The design and implementation of open vswitch. In USENIX NSDI, Cited by: §4.3.
  • [90] R. Poddar, C. Lan, R. A. Popa, and S. Ratnasamy (2018) Safebricks: Shielding network functions in the cloud. In USENIX NSDI, Cited by: §3.2.
  • [91] C. Price, S. Rivera, et al. (2012) Opnfv: an open platform to accelerate nfv. White Paper. A Linux Foundation Collaborative Project. Cited by: §3.1.
  • [92] S. Rajagopalan, D. Williams, H. Jamjoom, and A. Warfield (2013) Split/merge: System support for elastic execution in virtual middleboxes. In USENIX NSDI, Cited by: §3.2.
  • [93] F. Rath, J. Krude, J. Rüth, D. Schemmel, O. Hohlfeld, J. Á. Bitsch, and K. Wehrle (2017) Symperf: predicting network function performance. In ACM SIGCOMM Posters and Demos, Cited by: §3.2.
  • [94] M. T. Raza, S. Lu, and M. Gerla (2019) VEPC-sec: securing lte network functions virtualization on public cloud. IEEE Transactions on Information Forensics and Security. Cited by: §3.2.
  • [95] G. A. F. Rebello, I. D. Alvarenga, I. J. Sanz, and O. C. M. Duarte (2019) BSec-nfvo: a blockchain-based security for network function virtualization orchestration. In IEEE ICC, Cited by: §3.2.
  • [96] J. F. Riera, J. Batallé, J. Bonnet, M. Días, M. McGrath, G. Petralia, F. Liberati, A. Giuseppi, A. Pietrabissa, A. Ceselli, et al. (2016) TeNOR: steps towards an orchestration platform for multi-pop nfv deployment. In IEEE NetSoft, Cited by: §3.2.
  • [97] R. Riggio, I. G. B. Yahia, S. Latré, and T. Rasheed (2016) Scylla: a language for virtual network functions orchestration in enterprise wlans. In IEEE/IFIP NOMS, Cited by: §3.1.
  • [98] L. Rizzo and G. Lettieri (2012) Vale, a switched ethernet for virtual machines. In ACM CoNEXT, Cited by: §4.3.
  • [99] L. Rizzo (2012) Netmap: a novel framework for fast packet i/o. In USENIX Security 12, Cited by: §4.3.
  • [100] R. V. Rosa, C. Bertoldo, and C. E. Rothenberg (2017) Take your vnf to the gym: a testing framework for automated nfv performance benchmarking. IEEE Communications Magazine. Cited by: §3.2.
  • [101] R. V. Rosa, C. E. Rothenberg, and R. Szabo (2015) VBaaS: vnf benchmark-as-a-service. In IEEE EWSDN ’15, Cited by: §3.2.
  • [102] I. J. Sanz, D. M. F. Mattos, and O. C. M. B. Duarte (2018) SFCPerf: an automatic performance evaluation framework for service function chaining. In IEEE/IFIP NOMS, Cited by: §3.2.
  • [103] V. Sekar, N. Egi, S. Ratnasamy, M. K. Reiter, and G. Shi (2012) Design and implementation of a consolidated middlebox architecture. In USENIX NSDI, Cited by: §3.1.
  • [104] W. Shen, M. Yoshida, K. Minato, and W. Imajuku (2015) vConductor: An enabler for achieving virtual network integration as a service. IEEE Communications Magazine. Cited by: §3.2.
  • [105] Z. Shen and Y. Zhang (2018) An NFV Framework for Supporting Elastic Scaling of Service Function Chain. In IEEE ICCC, Cited by: §3.3.
  • [106] J. Sherry, P. X. Gao, S. Basu, A. Panda, A. Krishnamurthy, C. Maciocco, M. Manesh, J. Martins, S. Ratnasamy, L. Rizzo, et al. (2015) Rollback-recovery for middleboxes. In ACM SIGCOMM CCR, Cited by: §3.2.
  • [107] J. Sherry, S. Hasan, C. Scott, A. Krishnamurthy, S. Ratnasamy, and V. Sekar (2012) Making middleboxes someone else’s problem: network processing as a cloud service. ACM SIGCOMM CCR. Cited by: §1.
  • [108] M. Shih, M. Kumar, T. Kim, and A. Gavrilovska (2016) S-nfv: securing nfv states by using sgx. In ACM SDN-NFV Security ’16, Cited by: §3.2.
  • [109] A. Singhvi, J. Khalid, A. Akella, and S. Banerjee (2019) SNF: serverless network functions. arXiv preprint arXiv:1910.07700. Cited by: §3.2.
  • [110] J. Soares, C. Gonçalves, B. Parreira, P. Tavares, J. Carapinha, J. P. Barraca, R. L. Aguiar, and S. Sargento (2015) Toward a telco cloud environment for service functions. IEEE Communications Magazine. Cited by: §3.1.
  • [111] C. Sun, J. Bi, Z. Zheng, H. Yu, and H. Hu (2017) NFP: Enabling network function parallelism in NFV. In ACM SIGCOMM ’17, Cited by: §3.3.
  • [112] A. Tootoonchian, A. Panda, C. Lan, M. Walls, K. Argyraki, S. Ratnasamy, and S. Shenker (2018) Resq: Enabling slos in network function virtualization. In USENIX NSDI, Cited by: §3.2.
  • [113] B. Trach, A. Krohmer, F. Gregor, S. Arnautov, P. Bhatotia, and C. Fetzer (2018) ShieldBox: Secure middleboxes using shielded execution. In ACM SOSR, Cited by: §3.2.
  • [114] B. Tschaen, Y. Zhang, T. Benson, S. Banerjee, J. Lee, and J. Kang (2016) SFC-checker: checking the correct forwarding behavior of service function chaining. In IEEE NFV-SDN, Cited by: §3.2.
  • [115] Y. Wang, G. Xie, Z. Li, P. He, and K. Salamatian (2016) Transparent flow migration for nfv. In IEEE ICNP, Cited by: §3.2.
  • [116] S. Woo, J. Sherry, S. Han, S. Moon, S. Ratnasamy, and S. Shenker (2018) Elastic scaling of stateful network functions. In USENIX NSDI, Cited by: §3.1.
  • [117] W. Wu, K. He, and A. Akella (2015) Perfsight: performance diagnosis for software dataplanes. In ACM IMC, Cited by: §3.2.
  • [118] G. Xilouris, M. Kourtis, M. J. McGrath, V. Riccobene, G. Petralia, E. Markakis, E. Palis, A. Georgios, G. Gardikis, J. F. Riera, et al. (2015) T-nova: Network functions as-a-service over virtualised infrastructures. In IEEE NFV-SDN, Cited by: §3.2.
  • [119] Z. Xu, Y. Cui, and Y. Jiang (2018) CoNFV: an endhost-cloud collaborated network function virtualization framework. In IEEE IMCEC, Cited by: §3.1.
  • [120] W. Yang and C. Fung (2016) A survey on security in network functions virtualization. In IEEE NetSoft, Cited by: §1.
  • [121] K. Yasukata, F. Huici, V. Maffione, G. Lettieri, and M. Honda (2017) HyperNF: Building a high performance, high utilization and fair NFV platform. In ACM SOCC, Cited by: §3.3.
  • [122] B. Yi, X. Wang, K. Li, M. Huang, et al. (2018) A comprehensive survey of network function virtualization. Computer Networks. Cited by: §1.
  • [123] X. Yi, J. Duan, and C. Wu (2017) Gpunfv: a gpu-accelerated nfv system. In ACM APNet’17, Cited by: §3.3.
  • [124] X. Yi, J. Wang, J. Duan, W. Bai, C. Wu, Y. Xiong, and D. Han (2019) FlowShader: a generalized framework for gpu-accelerated vnf flow processing. In IEEE ICNP, Cited by: §3.3.
  • [125] P. Zave, R. A. Ferreira, X. K. Zou, M. Morimoto, and J. Rexford (2017) Dynamic service chaining with dysco. In ACM SIGCOMM, Cited by: §3.3.
  • [126] K. Zhang, B. He, J. Hu, Z. Wang, B. Hua, J. Meng, and L. Yang (2018) G-NET: Effective GPU Sharing in NFV Systems. In USENIX NSDI, Cited by: §3.3.
  • [127] L. Zhang, C. Li, P. Wang, Y. Liu, Y. Hu, Q. Chen, and M. Guo (2019) Characterizing and orchestrating nfv-ready servers for efficient edge data processing. In ACM IWQoS ’19, Cited by: §3.2.
  • [128] T. Zhang, L. Linguaglossa, M. Gallo, P. Giaccone, L. Iannone, and J. Roberts (2019) Comparing the performance of state-of-the-art software switches for nfv. In ACM CoNEXT, Cited by: §4.3, §4.3.
  • [129] W. Zhang, J. Hwang, S. Rajagopalan, K. Ramakrishnan, and T. Wood (2016) Flurries: Countless fine-grained nfs for flexible per-flow customization. In ACM CoNEXT, Cited by: §3.3.
  • [130] W. Zhang, G. Liu, A. Mohammadkhan, J. Hwang, K. Ramakrishnan, and T. Wood (2016) SDNFV: Flexible and dynamic software defined control of an application-and flow-aware data plane. In ACM Middleware Industry ’16, Cited by: §3.1.
  • [131] W. Zhang, G. Liu, W. Zhang, N. Shah, P. Lopreiato, G. Todeschi, K. Ramakrishnan, and T. Wood (2016) OpenNetVM: a platform for high performance network service chains. In ACM HotMiddlebox’16, Cited by: §3.3.
  • [132] Y. Zhang, B. Anwer, V. Gopalakrishnan, B. Han, J. Reich, A. Shaikh, and Z. Zhang (2017) Parabox: Exploiting parallelism for virtual network functions in service chaining. In ACM SOSR, Cited by: §3.3.
  • [133] C. Zheng, Q. Lu, J. Li, Q. Liu, and B. Fang (2018) A flexible and efficient container-based nfv platform for middlebox networking. In ACM SAC, Cited by: §3.3.
  • [134] Z. Zheng, J. Bi, C. Sun, H. Yu, H. Hu, Z. Meng, S. Wang, K. Gao, and J. Wu (2018) Gen: a gpu-accelerated elastic framework for nfv. In ACM APNet ’18, Cited by: §3.3, §4.2.
  • [135] Z. Zheng, J. Bi, H. Wang, C. Sun, H. Yu, H. Hu, K. Gao, and J. Wu (2018) Grus: enabling latency slos for gpu-accelerated nfv systems. In IEEE ICNP, Cited by: §3.3.