--- title: NFV Service Density Benchmarking # abbrev: nf-svc-density docname: draft-mkonstan-nf-service-density-01 date: 2019-07-08 ipr: trust200902 area: ops wg: Benchmarking Working Group kw: Internet-Draft cat: info coding: us-ascii pi: # can use array (if all yes) or hash here # - toc # - sortrefs # - symrefs toc: yes sortrefs: # defaults to yes symrefs: yes author: - ins: M. Konstantynowicz name: Maciek Konstantynowicz org: Cisco Systems role: editor email: mkonstan@cisco.com - ins: P. Mikus name: Peter Mikus org: Cisco Systems role: editor email: pmikus@cisco.com normative: RFC2544: RFC8174: informative: RFC8204: TST009: target: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.01.01_60/gs_NFV-TST009v030101p.pdf title: "ETSI GS NFV-TST 009 V3.1.1 (2018-10), Network Functions Virtualisation (NFV) Release 3; Testing; Specification of Networking Benchmarks and Measurement Methods for NFVI" date: 2018-10 BSDP: target: https://fd.io/wp-content/uploads/sites/34/2019/03/benchmarking_sw_data_planes_skx_bdx_mar07_2019.pdf title: "Benchmarking Software Data Planes Intel® Xeon® Skylake vs. Broadwell" date: 2019-03 draft-vpolak-mkonstan-bmwg-mlrsearch: target: https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch title: "Multiple Loss Ratio Search for Packet Throughput (MLRsearch)" date: 2019-07 draft-vpolak-bmwg-plrsearch: target: https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch title: "Probabilistic Loss Ratio Search for Packet Throughput (PLRsearch)" date: 2019-07 LFN-FDio-CSIT: target: https://wiki.fd.io/view/CSIT title: "Fast Data io, Continuous System Integration and Testing Project" date: 2019-07 CNCF-CNF-Testbed: target: https://github.com/cncf/cnf-testbed/ title: "Cloud native Network Function (CNF) Testbed" date: 2019-07 TRex: target: https://github.com/cisco-system-traffic-generator/trex-core title: "TRex Low-Cost, High-Speed Stateful Traffic Generator" date: 2019-07 CSIT-1904-testbed-2n-skx: target: https://docs.fd.io/csit/rls1904/report/introduction/physical_testbeds.html#node-xeon-skylake-2n-skx title: "FD.io CSIT Test Bed" date: 2019-06 CSIT-1904-test-enviroment: target: https://docs.fd.io/csit/rls1904/report/vpp_performance_tests/test_environment.html title: "FD.io CSIT Test Environment" date: 2019-06 CSIT-1904-nfv-density-methodology: target: https://docs.fd.io/csit/rls1904/report/introduction/methodology_nfv_service_density.html title: "FD.io CSIT Test Methodology: NFV Service Density" date: 2019-06 CSIT-1904-nfv-density-results: target: https://docs.fd.io/csit/rls1904/report/vpp_performance_tests/nf_service_density/index.html title: "FD.io CSIT Test Results: NFV Service Density" date: 2019-06 CNCF-CNF-Testbed-Results: target: https://github.com/cncf/cnf-testbed/blob/master/comparison/doc/cncf-cnfs-results-summary.md title: "CNCF CNF Testbed: NFV Service Density Benchmarking" date: 2018-12 NFVbench: target: https://opnfv-nfvbench.readthedocs.io/en/latest/testing/user/userguide/readme.html title: NFVbench Data Plane Performance Measurement Features date: 2019-07 --- abstract Network Function Virtualization (NFV) system designers and operators continuously grapple with the problem of qualifying performance of network services realised with software Network Functions (NF) running on Commercial-Off-The-Shelf (COTS) servers. One of the main challenges is getting repeatable and portable benchmarking results and using them to derive deterministic operating range that is production deployment worthy. This document specifies benchmarking methodology for NFV services that aims to address this problem space. It defines a way for measuring performance of multiple NFV service instances, each composed of multiple software NFs, and running them at a varied service “packing” density on a single server. The aim is to discover deterministic usage range of NFV system. In addition specified methodology can be used to compare and contrast different NFV virtualization technologies. --- middle # Terminology * NFV: Network Function Virtualization, a general industry term describing network functionality implemented in software. * NFV service: a software based network service realized by a topology of interconnected constituent software network function applications. * NFV service instance: a single instantiation of NFV service. * Data-plane optimized software: any software with dedicated threads handling data-plane packet processing e.g. FD.io VPP (Vector Packet Processor), OVS-DPDK. * Packet Loss Ratio (PLR): ratio of packets received relative to packets transmitted over the test trial duration, calculated using formula: PLR = ( pkts_transmitted - pkts_received ) / pkts_transmitted. For bi-directional throughput tests aggregate PLR is calculated based on the aggregate number of packets transmitted and received. * Packet Throughput Rate: maximum packet offered load DUT/SUT forwards within the specified Packet Loss Ratio (PLR). In many cases the rate depends on the frame size processed by DUT/SUT. Hence packet throughput rate MUST be quoted with specific frame size as received by DUT/SUT during the measurement. For bi-directional tests, packet throughput rate should be reported as aggregate for both directions. Measured in packets-per-second (pps) or frames-per-second (fps), equivalent metrics. * Non Drop Rate (NDR): maximum packet/bandwith throughput rate sustained by DUT/SUT at PLR equal zero (zero packet loss) specific to tested frame size(s). MUST be quoted with specific packet size as received by DUT/SUT during the measurement. Packet NDR measured in packets-per-second (or fps), bandwidth NDR expressed in bits-per-second (bps). * Partial Drop Rate (PDR): maximum packet/bandwith throughput rate sustained by DUT/SUT at PLR greater than zero (non-zero packet loss) specific to tested frame size(s). MUST be quoted with specific packet size as received by DUT/SUT during the measurement. Packet PDR measured in packets-per-second (or fps), bandwidth PDR expressed in bits-per-second (bps). * Maximum Receive Rate (MRR): packet/bandwidth rate regardless of PLR sustained by DUT/SUT under specified Maximum Transmit Rate (MTR) packet load offered by traffic generator. MUST be quoted with both specific packet size and MTR as received by DUT/SUT during the measurement. Packet MRR measured in packets-per-second (or fps), bandwidth MRR expressed in bits-per-second (bps). # Motivation ## Problem Description Network Function Virtualization (NFV) system designers and operators continuously grapple with the problem of qualifying performance of network services realised with software Network Functions (NF) running on Commercial-Off-The-Shelf (COTS) servers. One of the main challenges is getting repeatable and portable benchmarking results and using them to derive deterministic operating range that is production deployment worthy. Lack of well defined and standardised NFV centric performance methodology and metrics makes it hard to address fundamental questions that underpin NFV production deployments: 1. What NFV service and how many instances can run on a single compute node? 2. How to choose the best compute resource allocation scheme to maximise service yield per node? 3. How do different NF applications compare from the service density perspective? 4. How do the virtualisation technologies compare e.g. Virtual Machines, Containers? Getting answers to these points should allow designers to make data based decisions about the NFV technology and service design best suited to meet requirements of their use cases. Thereby obtained benchmarking data would aid in selection of the most appropriate NFV infrastructure design and platform and enable more accurate capacity planning, an important element for commercial viability of the NFV service. ## Proposed Solution The primary goal of the proposed benchmarking methodology is to focus on NFV technologies used to construct NFV services. More specifically to i) measure packet data-plane performance of multiple NFV service instances while running them at varied service “packing” densities on a single server and ii) quantify the impact of using multiple NFs to construct each NFV service instance and introducing multiple packet processing hops and links on each packet path. The overarching aim is to discover a set of deterministic usage ranges that are of interest to NFV system designers and operators. In addition, specified methodology can be used to compare and contrast different NFV virtualisation technologies. In order to ensure wide applicability of the benchmarking methodology, the approach is to separate NFV service packet processing from the shared virtualisation infrastructure by decomposing the software technology stack into three building blocks: +-------------------------------+ | NFV Service | +-------------------------------+ | Virtualization Technology | +-------------------------------+ | Host Networking | +-------------------------------+ Figure 1. NFV software technology stack. Proposed methodology is complementary to existing NFV benchmarking industry efforts focusing on vSwitch benchmarking [RFC8204], [TST009] and extends the benchmarking scope to NFV services. This document does not describe a complete benchmarking methodology, instead it is focusing on the system under test configuration. Each of the compute node configurations identified in this document is to be evaluated for NFV service data-plane performance using existing and/or emerging network benchmarking standards. This may include methodologies specified in [RFC2544], [TST009], [draft-vpolak-mkonstan-bmwg-mlrsearch] and/or [draft-vpolak-bmwg-plrsearch]. # NFV Service It is assumed that each NFV service instance is built of one or more constituent NFs and is described by: topology, configuration and resulting packet path(s). Each set of NFs forms an independent NFV service instance, with multiple sets present in the host. ## Topology NFV topology describes the number of network functions per service instance, and their inter-connections over packet interfaces. It includes all point-to-point virtual packet links within the compute node, Layer-2 Ethernet or Layer-3 IP, including the ones to host networking data-plane. Theoretically, a large set of possible NFV topologies can be realised using software virtualisation topologies, e.g. ring, partial -/full- mesh, star, line, tree, ladder. In practice however, only a few topologies are in the actual use as NFV services mostly perform either bumps-in-a-wire packet operations (e.g. security filtering/inspection, monitoring/telemetry) and/or inter-site forwarding decisions (e.g. routing, switching). Two main NFV topologies have been identified so far for NFV service density benchmarking: 1. Chain topology: a set of NFs connect to host data-plane with minimum of two virtual interfaces each, enabling host data-plane to facilitate NF to NF service chain forwarding and provide connectivity with external network. 2. Pipeline topology: a set of NFs connect to each other in a line fashion with edge NFs homed to host data-plane. Host data-plane provides connectivity with external network. In both cases multiple NFV service topologies are running in parallel. Both topologies are shown in figures 2. and 3. below. NF chain topology: +-----------------------------------------------------------+ | Host Compute Node | | | | SmNF1 SmNF2 SmNFn Service-m | | ... ... ... ... | | S2NF1 S2NF2 S2NFn Service-2 | | +--------+ +--------+ +--------+ | | | S1NF1 | | S1NF2 | | S1NFn | | | | | | | .... | | Service-1 | | | | | | | | | | +-+----+-+ +-+----+-+ + + +-+----+-+ | | | | | | | | | | Virtual | | | |<-CS->| |<-CS->| |<-CS->| | Interfaces | | +-+----+------+----+------+----+------+----+-+ | | | | CS: Chain | | | | Segment | | | Host Data-Plane | | | +-+--+----------------------------------+--+-+ | | | | | | | +-----------------------------------------------------------+ | | | | Physical | | | | Interfaces +---+--+----------------------------------+--+--------------+ | | | Traffic Generator | | | +-----------------------------------------------------------+ Figure 2. NF chain topology forming a service instance. NF pipeline topology: +-----------------------------------------------------------+ | Host Compute Node | | | | SmNF1 SmNF2 SmNFn Service-m | | ... ... ... ... | | S2NF1 S2NF2 S2NFn Service-2 | | +--------+ +--------+ +--------+ | | | S1NF1 | | S1NF2 | | S1NFn | | | | +--+ +--+ .... +--+ | Service1 | | | | | | | | | | +-+------+ +--------+ +------+-+ | | | | Virtual | | |<-Pipeline Edge Pipeline Edge->| Interfaces | | +-+----------------------------------------+-+ | | | | | | | | | | | Host Data-Plane | | | +-+--+----------------------------------+--+-+ | | | | | | | +-----------------------------------------------------------+ | | | | Physical | | | | Interfaces +---+--+----------------------------------+--+--------------+ | | | Traffic Generator | | | +-----------------------------------------------------------+ Figure 3. NF pipeline topology forming a service instance. ## Configuration NFV configuration includes all packet processing functions in NFs including Layer-2, Layer-3 and/or Layer-4-to-7 processing as appropriate to specific NF and NFV service design. L2 sub- interface encapsulations (e.g. 802.1q, 802.1ad) and IP overlay encapsulation (e.g. VXLAN, IPSec, GRE) may be represented here too as appropriate, although in most cases they are used as external encapsulation and handled by host networking data-plane. NFV configuration determines logical network connectivity that is Layer-2 and/or IPv4/IPv6 switching/routing modes, as well as NFV service specific aspects. In the context of NFV density benchmarking methodology the initial focus is on logical network connectivity between the NFs, and no NFV service specific configurations. NF specific functionality is emulated using IPv4/IPv6 routing. Building on the two identified NFV topologies, two common NFV configurations are considered: 1. Chain configuration: * Relies on chain topology to form NFV service chains. * NF packet forwarding designs: * IPv4/IPv6 routing. * Requirements for host data-plane: * L2 switching with L2 forwarding context per each NF chain segment, or * IPv4/IPv6 routing with IP forwarding context per each NF chain segment or per NF chain. 2. Pipeline configuration: * Relies on pipeline topology to form NFV service pipelines. * Packet forwarding designs: * IPv4/IPv6 routing. * Requirements for host data-plane: * L2 switching with L2 forwarding context per each NF pipeline edge link, or * IPv4/IPv6 routing with IP forwarding context per each NF pipeline edge link or per NF pipeline. ## Packet Path(s) NFV packet path(s) describe the actual packet forwarding path(s) used for benchmarking, resulting from NFV topology and configuration. They are aimed to resemble true packet forwarding actions during the NFV service lifecycle. Based on the specified NFV topologies and configurations two NFV packet paths are taken for benchmarking: 1. Snake packet path * Requires chain topology and configuration. * Packets enter the NFV chain through one edge NF and progress to the other edge NF of the chain. * Within the chain, packets follow a zigzagging "snake" path entering and leaving host data-plane as they progress through the NF chain. * Host data-plane is involved in packet forwarding operations between NIC interfaces and edge NFs, as well as between NFs in the chain. 2. Pipeline packet path * Requires pipeline topology and configuration. * Packets enter the NFV chain through one edge NF and progress to the other edge NF of the pipeline. * Within the chain, packets follow a straight path entering and leaving subsequent NFs as they progress through the NF pipeline. * Host data-plane is involved in packet forwarding operations between NIC interfaces and edge NFs only. Both packet paths are shown in figures below. Snake packet path: +-----------------------------------------------------------+ | Host Compute Node | | | | SmNF1 SmNF2 SmNFn Service-m | | ... ... ... ... | | S2NF1 S2NF2 S2NFn Service-2 | | +--------+ +--------+ +--------+ | | | S1NF1 | | S1NF2 | | S1NFn | | | | | | | .... | | Service1 | | | XXXX | | XXXX | | XXXX | | | +-+X--X+-+ +-+X--X+-+ +X X+ +-+X--X+-+ | | |X X| |X X| |X X| |X X| Virtual | | |X X| |X X| |X X| |X X| Interfaces | | +-+X--X+------+X--X+------+X--X+------+X--X+-+ | | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | | | | X X | | | | X Host Data-Plane X | | | +-+X-+----------------------------------+-X+-+ | | |X | | X| | +----X--------------------------------------X---------------+ |X | | X| Physical |X | | X| Interfaces +---+X-+----------------------------------+-X+--------------+ | | | Traffic Generator | | | +-----------------------------------------------------------+ Figure 4. Snake packet path thru NF chain topology. Pipeline packet path: +-----------------------------------------------------------+ | Host Compute Node | | | | SmNF1 SmNF2 SmNFn Service-m | | ... ... ... ... | | S2NF1 S2NF2 S2NFn Service-2 | | +--------+ +--------+ +--------+ | | | S1NF1 | | S1NF2 | | S1NFn | | | | +--+ +--+ .... +--+ | Service1 | | | XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXX | | | +--X-----+ +--------+ +-----X--+ | | |X X| Virtual | | |X X| Interfaces | | +-+X--------------------------------------X+-+ | | | X X | | | | X X | | | | X Host Data-Plane X | | | +-+X-+----------------------------------+-X+-+ | | |X | | X| | +----X--------------------------------------X---------------+ |X | | X| Physical |X | | X| Interfaces +---+X-+----------------------------------+-X+--------------+ | | | Traffic Generator | | | +-----------------------------------------------------------+ Figure 5. Pipeline packet path thru NF pipeline topology. In all cases packets enter NFV system via shared physical NIC interfaces controlled by shared host data-plane, are then associated with specific NFV service (based on service discriminator) and subsequently are cross- connected/switched/routed by host data-plane to and through NF topologies per one of the above listed schemes. # Virtualization Technology NFV services are built of composite isolated NFs, with virtualisation technology providing the workload isolation. Following virtualisation technology types are considered for NFV service density benchmarking: 1. Virtual Machines (VMs) * Relying on host hypervisor technology e.g. KVM, ESXi, Xen. * NFs running in VMs are referred to as VNFs. 2. Containers * Relying on Linux container technology e.g. LXC, Docker. * NFs running in Containers are referred to as CNFs. Different virtual interface types are available to VNFs and CNFs: 1. VNF * virtio-vhostuser: fully user-mode based virtual interface. * virtio-vhostnet: involves kernel-mode based backend. 2. CNF * memif: fully user-mode based virtual interface. * af_packet: involves kernel-mode based backend. * (add more common ones) # Host Networking Host networking data-plane is the central shared resource that underpins creation of NFV services. It handles all of the connectivity to external physical network devices through physical network connections using NICs, through which the benchmarking is done. Assuming that NIC interface resources are shared, here is the list of widely available host data-plane options for providing packet connectivity to/from NICs and constructing NFV chain and pipeline topologies and configurations: * Linux Kernel-Mode Networking. * Linux User-Mode vSwitch. * Virtual Machine vSwitch. * Linux Container vSwitch. * SRIOV NIC Virtual Function - note: restricted support for chain and pipeline topologies, as it requires hair-pinning through the NIC and oftentimes also through external physical switch. Analysing properties of each of these options and their Pros/Cons for specified NFV topologies and configurations is outside the scope of this document. From all listed options, performance optimised Linux user-mode vswitch deserves special attention. Linux user-mode switch decouples NFV service from the underlying NIC hardware, offers rich multi-tenant functionality and most flexibility for supporting NFV services. But in the same time it is consuming compute resources and is harder to benchmark in NFV service density scenarios. Following sections focus on using Linux user-mode vSwitch, focusing on its performance benchmarking at increasing levels of NFV service density. # NFV Service Density Matrix In order to evaluate performance of multiple NFV services running on a compute node, NFV service instances are benchmarked at increasing density, allowing to construct an NFV Service Density Matrix. Table below shows an example of such a matrix, capturing number of NFV service instances (row indices), number of NFs per service instance (column indices) and resulting total number of NFs (values). NFV Service Density - NF Count View SVC 001 002 004 006 008 00N 001 1 2 4 6 8 1*N 002 2 4 8 12 16 2*N 004 4 8 16 24 32 4*N 006 6 12 24 36 48 6*N 008 8 16 32 48 64 8*N 00M M*1 M*2 M*4 M*6 M*8 M*N RowIndex: Number of NFV Service Instances, 1..M. ColumnIndex: Number of NFs per NFV Service Instance, 1..N. Value: Total number of NFs running in the system. In order to deliver good and repeatable network data-plane performance, NFs and host data-plane software require direct access to critical compute resources. Due to a shared nature of all resources on a compute node, a clearly defined resource allocation scheme is defined in the next section to address this. In each tested configuration host data-plane is a gateway between the external network and the internal NFV network topologies. Offered packet load is generated and received by an external traffic generator per usual benchmarking practice. It is proposed that benchmarks are done with the offered packet load distributed equally across all configured NFV service instances. This approach should provide representative benchmarking data for each tested topology and configuraiton, and a good guesstimate of maximum performance required for capacity planning. Following sections specify compute resource allocation, followed by examples of applying NFV service density methodology to VNF and CNF benchmarking use cases. # Compute Resource Allocation Performance optimized NF and host data-plane software threads require timely execution of packet processing instructions and are very sensitive to any interruptions (or stalls) to this execution e.g. cpu core context switching, or cpu jitter. To that end, NFV service density methodology treats controlled mapping ratios of data plane software threads to physical processor cores with directly allocated cache hierarchies as the first order requirement. Other compute resources including memory bandwidth and PCIe bandwidth have lesser impact and as such are subject for further study. For more detail and deep-dive analysis of software data plane performance and impact on different shared compute resources is available in [BSDP]. It is assumed that NFs as well as host data-plane (e.g. vswitch) are performance optimized, with their tasks executed in two types of software threads: * data-plane - handling data-plane packet processing and forwarding, time critical, requires dedicated cores. To scale data-plane performance, most NF apps use multiple data-plane threads and rely on NIC RSS (Receive Side Scaling), virtual interface multi-queue and/or integrated software hashing to distribute packets across the data threads. * main-control - handling application management, statistics and control-planes, less time critical, allows for core sharing. For most NF apps this is a single main thread, but often statistics (counters) and various control protocol software are run in separate threads. Core mapping scheme described below allocates cores for all threads of specified type belonging to each NF app instance, and separately lists number of threads to a number of logical/physical core mappings for processor configurations with enabled/disabled Symmetric Multi- Threading (SMT) (e.g. AMD SMT, Intel Hyper-Threading). If NFV service density benchmarking is run on server nodes with Symmetric Multi-Threading (SMT) (e.g. AMD SMT, Intel Hyper-Threading) for higher performance and efficiency, logical cores allocated to data- plane threads should be allocated as pairs of sibling logical cores corresponding to the hyper-threads running on the same physical core. Separate core ratios are defined for mapping threads of vSwitch and NFs. In order to get consistent benchmarking results, the mapping ratios are enforced using Linux core pinning. | application | thread type | app:core ratio | threads/pcores (SMT disabled) | threads/lcores map (SMT enabled) | |:-----------:|:-----------:|:--------------:|:-------------------------------:|:----------------------------------:| | vSwitch-1c | data | 1:1 | 1DT/1PC | 2DT/2LC | | | main | 1:S2 | 1MT/S2PC | 1MT/1LC | | | | | | | | vSwitch-2c | data | 1:2 | 2DT/2PC | 4DT/4LC | | | main | 1:S2 | 1MT/S2PC | 1MT/1LC | | | | | | | | vSwitch-4c | data | 1:4 | 4DT/4PC | 8DT/8LC | | | main | 1:S2 | 1MT/S2PC | 1MT/1LC | | | | | | | | NF-0.5c | data | 1:S2 | 1DT/S2PC | 1DT/1LC | | | main | 1:S2 | 1MT/S2PC | 1MT/1LC | | | | | | | | NF-1c | data | 1:1 | 1DT/1PC | 2DT/2LC | | | main | 1:S2 | 1MT/S2PC | 1MT/1LC | | | | | | | | NF-2c | data | 1:2 | 2DT/2PC | 4DT/4LC | | | main | 1:S2 | 1MT/S2PC | 1MT/1LC | * Legend to table * Header row * application - network application with optimized data-plane, a vSwitch or Network Function (NF) application. * thread type - either "data", short for data-plane; or "main", short for all main-control threads. * app:core ratio - ratio of per application instance threads of specific thread type to physical cores. * threads/pcores (SMT disabled) - number of threads of specific type (DT for data-plane thread, MT for main thread) running on a number of physical cores, with SMT disabled. * threads/lcores map (SMT enabled) - number of threads of specific type (DT, MT) running on a number of logical cores, with SMT enabled. Two logical cores per one physical core. * Content rows * vSwitch-(1c|2c|4c) - vSwitch with 1 physical core (or 2, or 4) allocated to its data-plane software worker threads. * NF-(0.5c|1c|2c) - NF application with half of a physical core (or 1, or 2) allocated to its data-plane software worker threads. * Sn - shared core, sharing ratio of (n). * DT - data-plane thread. * MT - main-control thread. * PC - physical core, with SMT/HT enabled has many (mostly 2 today) logical cores associated with it. * LC - logical core, if more than one lc get allocated in sets of two sibling logical cores running on the same physical core. * SnPC - shared physical core, sharing ratio of (n). * SnLC - shared logical core, sharing ratio of (n). Maximum benchmarked NFV service densities are limited by a number of physical cores on a compute node. A sample physical core usage view is shown in the matrix below. NFV Service Density - Core Usage View vSwitch-1c, NF-1c SVC 001 002 004 006 008 010 001 2 3 6 9 12 15 002 3 6 12 18 24 30 004 6 12 24 36 48 60 006 9 18 36 54 72 90 008 12 24 48 72 96 120 010 15 30 60 90 120 150 RowIndex: Number of NFV Service Instances, 1..10. ColumnIndex: Number of NFs per NFV Service Instance, 1..10. Value: Total number of physical processor cores used for NFs. # NFV Service Data-Plane Benchmarking NF service density scenarios should have their data-plane performance benchmarked using existing and/or emerging network benchmarking standards as noted earlier. Following metrics should be measured (or calculated) and reported: * Packet throughput rate (packets-per-second) * Specific to tested packet size or packet sequence (e.g. some type of packet size mix sent in recurrent sequence). * Applicable types of throughput rate: NDR, PDR, MRR. * (Calculated) Bandwidth throughput rate (bits-per-second) corresponding to the measured packet throughput rate. * Packet one-way latency (seconds) * Measured at different packet throughput rates load e.g. light, medium, heavy. Listed metrics should be itemized per service instance and per direction (e.g. forward/reverse) for latency. # Sample NFV Service Density Benchmarks To illustrate defined NFV service density applicability, following sections describe three sets of NFV service topologies and configurations that have been benchmarked in open-source: i) in [LFN-FDio-CSIT], a continuous testing and data-plane benchmarking project, ii) as part of CNCF CNF Testbed initiative [CNCF-CNF-Testbed] and iii) in OPNFV NFVbench project. In the first two cases each NFV service instance definition is based on the same set of NF applications, and varies only by network addressing configuration to emulate multi-tenant operating environment. OPNFV NFVbench project is focusing on benchmarking the actual production deployments that are aligned with OPNFV specifications. ## Intrepreting the Sample Results TODO How to interpret and avoid misreading included results? And how to avoid falling into the trap of using these results to draw generilized conclusions about performance of different virtualization technologies, e.g. VM and Containers, irrespective of deployment scenarios and what VNFs and CNFs are in the actual use. ## Benchmarking MRR Throughput Initial NFV density throughput benchmarks have been performed using Maximum Receive Rate (MRR) test methodology defined and used in FD.io CSIT. MRR tests measure the packet forwarding rate under specified Maximum Transmit Rate (MTR) packet load offered by traffic generator over a set trial duration, regardless of packet loss ratio (PLR). MTR for specified Ethernet frame size was set to the bi-directional link rate, 2x 10GbE in referred results. Tests were conducted with two traffic profiles: i) continuous stream of 64B frames, ii) continuous stream of IMIX sequence of (7x 64B, 4x 570B, 1x 1518B), all sizes are L2 untagged Ethernet. NFV service topologies tested include: VNF service chains, CNF service chains and CNF service pipelines. ## VNF Service Chain VNF Service Chain (VSC) topology is tested with KVM hypervisor (Ubuntu 18.04-LTS), with NFV service instances consisting of NFs running in VMs (VNFs). Host data-plane is provided by FD.io VPP vswitch. Virtual interfaces are virtio-vhostuser. Snake forwarding packet path is tested using [TRex] traffic generator, see figure. +-----------------------------------------------------------+ | Host Compute Node | | | | +--------+ +--------+ +--------+ | | | S1VNF1 | | S1VNF2 | | S1VNFn | | | | | | | .... | | Service1 | | | XXXX | | XXXX | | XXXX | | | +-+X--X+-+ +-+X--X+-+ +-+X--X+-+ | | |X X| |X X| |X X| Virtual | | |X X| |X X| |X X| |X X| Interfaces | | +-+X--X+------+X--X+------+X--X+------+X--X+-+ | | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | | | | X X | | | | X FD.io VPP vSwitch X | | | +-+X-+----------------------------------+-X+-+ | | |X | | X| | +----X--------------------------------------X---------------+ |X | | X| Physical |X | | X| Interfaces +---+X-+----------------------------------+-X+--------------+ | | | Traffic Generator (TRex) | | | +-----------------------------------------------------------+ Figure 6. VNF service chain test setup. ## CNF Service Chain CNF Service Chain (CSC) topology is tested with Docker containers (Ubuntu 18.04-LTS), with NFV service instances consisting of NFs running in Containers (CNFs). Host data-plane is provided by FD.io VPP vswitch. Virtual interfaces are memif. Snake forwarding packet path is tested using [TRex] traffic generator, see figure. +-----------------------------------------------------------+ | Host Compute Node | | | | +--------+ +--------+ +--------+ | | | S1CNF1 | | S1CNF2 | | S1CNFn | | | | | | | .... | | Service1 | | | XXXX | | XXXX | | XXXX | | | +-+X--X+-+ +-+X--X+-+ +-+X--X+-+ | | |X X| |X X| |X X| Virtual | | |X X| |X X| |X X| |X X| Interfaces | | +-+X--X+------+X--X+------+X--X+------+X--X+-+ | | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | | | | X X | | | | X FD.io VPP vSwitch X | | | +-+X-+----------------------------------+-X+-+ | | |X | | X| | +----X--------------------------------------X---------------+ |X | | X| Physical |X | | X| Interfaces +---+X-+----------------------------------+-X+--------------+ | | | Traffic Generator (TRex) | | | +-----------------------------------------------------------+ Figure 7. CNF service chain test setup. ## CNF Service Pipeline CNF Service Pipeline (CSP) topology is tested with Docker containers (Ubuntu 18.04-LTS), with NFV service instances consisting of NFs running in Containers (CNFs). Host data-plane is provided by FD.io VPP vswitch. Virtual interfaces are memif. Pipeline forwarding packet path is tested using [TRex] traffic generator, see figure. +-----------------------------------------------------------+ | Host Compute Node | | | | +--------+ +--------+ +--------+ | | | S1NF1 | | S1NF2 | | S1NFn | | | | +--+ +--+ .... +--+ | Service1 | | | XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXX | | | +--X-----+ +--------+ +-----X--+ | | |X X| Virtual | | |X X| Interfaces | | +-+X--------------------------------------X+-+ | | | X X | | | | X X | | | | X FD.io VPP vSwitch X | | | +-+X-+----------------------------------+-X+-+ | | |X | | X| | +----X--------------------------------------X---------------+ |X | | X| Physical |X | | X| Interfaces +---+X-+----------------------------------+-X+--------------+ | | | Traffic Generator (TRex) | | | +-----------------------------------------------------------+ Figure 8. CNF service chain test setup. ## Sample Results: FD.io CSIT FD.io CSIT project introduced NFV density benchmarking in release CSIT-1904 and published results for the following NFV service topologies and configurations: 1. VNF Service Chains * VNF: DPDK-L3FWD v19.02 * IPv4 forwarding * NF-1c * vSwitch: VPP v19.04-release * L2 MAC switching * vSwitch-1c, vSwitch-2c * frame sizes: 64B, IMIX 2. CNF Service Chains * CNF: VPP v19.04-release * IPv4 routing * NF-1c * vSwitch: VPP v19.04-release * L2 MAC switching * vSwitch-1c, vSwitch-2c * frame sizes: 64B, IMIX 3. CNF Service Pipelines * CNF: VPP v19.04-release * IPv4 routing * NF-1c * vSwitch: VPP v19.04-release * L2 MAC switching * vSwitch-1c, vSwitch-2c * frame sizes: 64B, IMIX More information is available in FD.io CSIT-1904 report, with specific references listed below: * Testbed: [CSIT-1904-testbed-2n-skx] * Test environment: [CSIT-1904-test-enviroment] * Methodology: [CSIT-1904-nfv-density-methodology] * Results: [CSIT-1904-nfv-density-results] ## Sample Results: CNCF/CNFs CNCF CI team introduced a CNF testbed initiative focusing on benchmaring NFV density with open-source network applications running as VNFs and CNFs. Following NFV service topologies and configurations have been tested to date: 1. VNF Service Chains * VNF: VPP v18.10-release * IPv4 routing * NF-1c * vSwitch: VPP v18.10-release * L2 MAC switching * vSwitch-1c, vSwitch-2c * frame sizes: 64B, IMIX 2. CNF Service Chains * CNF: VPP v18.10-release * IPv4 routing * NF-1c * vSwitch: VPP v18.10-release * L2 MAC switching * vSwitch-1c, vSwitch-2c * frame sizes: 64B, IMIX 3. CNF Service Pipelines * CNF: VPP v18.10-release * IPv4 routing * NF-1c * vSwitch: VPP v18.10-release * L2 MAC switching * vSwitch-1c, vSwitch-2c * frame sizes: 64B, IMIX More information is available in CNCF CNF Testbed github, with summary test results presented in summary markdown file, references listed below: * Results: [CNCF-CNF-Testbed-Results] ## Sample Results: OPNFV NFVbench TODO Add short NFVbench based test description, and NFVbench sweep chart with single VM per service instance: Y-axis packet throughput rate or bandwidth throughput rate, X-axis number of concurrent service instances. # IANA Considerations No requests of IANA. # Security Considerations Benchmarking activities as described in this memo are limited to technology characterization of a DUT/SUT using controlled stimuli in a laboratory environment, with dedicated address space and the constraints specified in the sections above. The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network or misroute traffic to the test management network. Further, benchmarking is performed on a "black-box" basis, relying solely on measurements observable external to the DUT/SUT. Special capabilities SHOULD NOT exist in the DUT/SUT specifically for benchmarking purposes. Any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks. # Acknowledgements Thanks to Vratko Polak of FD.io CSIT project and Michael Pedersen of the CNCF Testbed initiative for their contributions and useful suggestions. Extended thanks to Alec Hothan of OPNFV NFVbench project for numerous comments, suggestions and references to his/team work in the OPNFV/NVFbench project. --- back