--- /dev/null
+---
+title: NFV Service Density Benchmarking
+# abbrev: nf-svc-density
+docname: draft-mkonstan-nf-service-density-01
+date: 2019-07-08
+
+ipr: trust200902
+area: ops
+wg: Benchmarking Working Group
+kw: Internet-Draft
+cat: info
+
+coding: us-ascii
+pi: # can use array (if all yes) or hash here
+# - toc
+# - sortrefs
+# - symrefs
+ toc: yes
+ sortrefs: # defaults to yes
+ symrefs: yes
+
+author:
+ -
+ ins: M. Konstantynowicz
+ name: Maciek Konstantynowicz
+ org: Cisco Systems
+ role: editor
+ email: mkonstan@cisco.com
+ -
+ ins: P. Mikus
+ name: Peter Mikus
+ org: Cisco Systems
+ role: editor
+ email: pmikus@cisco.com
+
+normative:
+ RFC2544:
+ RFC8174:
+
+informative:
+ RFC8204:
+ TST009:
+ target: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.01.01_60/gs_NFV-TST009v030101p.pdf
+ title: "ETSI GS NFV-TST 009 V3.1.1 (2018-10), Network Functions Virtualisation (NFV) Release 3; Testing; Specification of Networking Benchmarks and Measurement Methods for NFVI"
+ date: 2018-10
+ BSDP:
+ target: https://fd.io/wp-content/uploads/sites/34/2019/03/benchmarking_sw_data_planes_skx_bdx_mar07_2019.pdf
+ title: "Benchmarking Software Data Planes Intel® Xeon® Skylake vs. Broadwell"
+ date: 2019-03
+ draft-vpolak-mkonstan-bmwg-mlrsearch:
+ target: https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch
+ title: "Multiple Loss Ratio Search for Packet Throughput (MLRsearch)"
+ date: 2019-07
+ draft-vpolak-bmwg-plrsearch:
+ target: https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch
+ title: "Probabilistic Loss Ratio Search for Packet Throughput (PLRsearch)"
+ date: 2019-07
+ LFN-FDio-CSIT:
+ target: https://wiki.fd.io/view/CSIT
+ title: "Fast Data io, Continuous System Integration and Testing Project"
+ date: 2019-07
+ CNCF-CNF-Testbed:
+ target: https://github.com/cncf/cnf-testbed/
+ title: "Cloud native Network Function (CNF) Testbed"
+ date: 2019-07
+ TRex:
+ target: https://github.com/cisco-system-traffic-generator/trex-core
+ title: "TRex Low-Cost, High-Speed Stateful Traffic Generator"
+ date: 2019-07
+ CSIT-1904-testbed-2n-skx:
+ target: https://docs.fd.io/csit/rls1904/report/introduction/physical_testbeds.html#node-xeon-skylake-2n-skx
+ title: "FD.io CSIT Test Bed"
+ date: 2019-06
+ CSIT-1904-test-enviroment:
+ target: https://docs.fd.io/csit/rls1904/report/vpp_performance_tests/test_environment.html
+ title: "FD.io CSIT Test Environment"
+ date: 2019-06
+ CSIT-1904-nfv-density-methodology:
+ target: https://docs.fd.io/csit/rls1904/report/introduction/methodology_nfv_service_density.html
+ title: "FD.io CSIT Test Methodology: NFV Service Density"
+ date: 2019-06
+ CSIT-1904-nfv-density-results:
+ target: https://docs.fd.io/csit/rls1904/report/vpp_performance_tests/nf_service_density/index.html
+ title: "FD.io CSIT Test Results: NFV Service Density"
+ date: 2019-06
+ CNCF-CNF-Testbed-Results:
+ target: https://github.com/cncf/cnf-testbed/blob/master/comparison/doc/cncf-cnfs-results-summary.md
+ title: "CNCF CNF Testbed: NFV Service Density Benchmarking"
+ date: 2018-12
+ NFVbench:
+ target: https://opnfv-nfvbench.readthedocs.io/en/latest/testing/user/userguide/readme.html
+ title: NFVbench Data Plane Performance Measurement Features
+ date: 2019-07
+
+--- abstract
+
+Network Function Virtualization (NFV) system designers and operators
+continuously grapple with the problem of qualifying performance of
+network services realised with software Network Functions (NF) running
+on Commercial-Off-The-Shelf (COTS) servers. One of the main challenges
+is getting repeatable and portable benchmarking results and using them
+to derive deterministic operating range that is production deployment
+worthy.
+
+This document specifies benchmarking methodology for NFV services that
+aims to address this problem space. It defines a way for measuring
+performance of multiple NFV service instances, each composed of multiple
+software NFs, and running them at a varied service “packing” density on
+a single server.
+
+The aim is to discover deterministic usage range of NFV system. In
+addition specified methodology can be used to compare and contrast
+different NFV virtualization technologies.
+
+--- middle
+
+# Terminology
+
+* NFV: Network Function Virtualization, a general industry term
+ describing network functionality implemented in software.
+* NFV service: a software based network service realized by a topology
+ of interconnected constituent software network function applications.
+* NFV service instance: a single instantiation of NFV service.
+* Data-plane optimized software: any software with dedicated threads
+ handling data-plane packet processing e.g. FD.io VPP (Vector Packet
+ Processor), OVS-DPDK.
+* Packet Loss Ratio (PLR): ratio of packets received relative to packets
+ transmitted over the test trial duration, calculated using formula:
+ PLR = ( pkts_transmitted - pkts_received ) / pkts_transmitted.
+ For bi-directional throughput tests aggregate PLR is calculated based
+ on the aggregate number of packets transmitted and received.
+* Packet Throughput Rate: maximum packet offered load DUT/SUT forwards
+ within the specified Packet Loss Ratio (PLR). In many cases the rate
+ depends on the frame size processed by DUT/SUT. Hence packet
+ throughput rate MUST be quoted with specific frame size as received by
+ DUT/SUT during the measurement. For bi-directional tests, packet
+ throughput rate should be reported as aggregate for both directions.
+ Measured in packets-per-second (pps) or frames-per-second (fps),
+ equivalent metrics.
+* Non Drop Rate (NDR): maximum packet/bandwith throughput rate sustained
+ by DUT/SUT at PLR equal zero (zero packet loss) specific to tested
+ frame size(s). MUST be quoted with specific packet size as received by
+ DUT/SUT during the measurement. Packet NDR measured in
+ packets-per-second (or fps), bandwidth NDR expressed in
+ bits-per-second (bps).
+* Partial Drop Rate (PDR): maximum packet/bandwith throughput rate
+ sustained by DUT/SUT at PLR greater than zero (non-zero packet loss)
+ specific to tested frame size(s). MUST be quoted with specific packet
+ size as received by DUT/SUT during the measurement. Packet PDR
+ measured in packets-per-second (or fps), bandwidth PDR expressed in
+ bits-per-second (bps).
+* Maximum Receive Rate (MRR): packet/bandwidth rate regardless of PLR
+ sustained by DUT/SUT under specified Maximum Transmit Rate (MTR)
+ packet load offered by traffic generator. MUST be quoted with both
+ specific packet size and MTR as received by DUT/SUT during the
+ measurement. Packet MRR measured in packets-per-second (or fps),
+ bandwidth MRR expressed in bits-per-second (bps).
+
+# Motivation
+
+## Problem Description
+
+Network Function Virtualization (NFV) system designers and operators
+continuously grapple with the problem of qualifying performance of
+network services realised with software Network Functions (NF) running
+on Commercial-Off-The-Shelf (COTS) servers. One of the main challenges
+is getting repeatable and portable benchmarking results and using them
+to derive deterministic operating range that is production deployment
+worthy.
+
+Lack of well defined and standardised NFV centric performance
+methodology and metrics makes it hard to address fundamental questions
+that underpin NFV production deployments:
+
+1. What NFV service and how many instances can run on a single compute
+ node?
+2. How to choose the best compute resource allocation scheme to maximise
+ service yield per node?
+3. How do different NF applications compare from the service density
+ perspective?
+4. How do the virtualisation technologies compare e.g. Virtual Machines,
+ Containers?
+
+Getting answers to these points should allow designers to make data
+based decisions about the NFV technology and service design best suited
+to meet requirements of their use cases. Thereby obtained benchmarking
+data would aid in selection of the most appropriate NFV infrastructure
+design and platform and enable more accurate capacity planning, an
+important element for commercial viability of the NFV service.
+
+## Proposed Solution
+
+The primary goal of the proposed benchmarking methodology is to focus on
+NFV technologies used to construct NFV services. More specifically to i)
+measure packet data-plane performance of multiple NFV service instances
+while running them at varied service “packing” densities on a single
+server and ii) quantify the impact of using multiple NFs to construct
+each NFV service instance and introducing multiple packet processing
+hops and links on each packet path.
+
+The overarching aim is to discover a set of deterministic usage ranges
+that are of interest to NFV system designers and operators. In addition,
+specified methodology can be used to compare and contrast different NFV
+virtualisation technologies.
+
+In order to ensure wide applicability of the benchmarking methodology,
+the approach is to separate NFV service packet processing from the
+shared virtualisation infrastructure by decomposing the software
+technology stack into three building blocks:
+
+ +-------------------------------+
+ | NFV Service |
+ +-------------------------------+
+ | Virtualization Technology |
+ +-------------------------------+
+ | Host Networking |
+ +-------------------------------+
+
+ Figure 1. NFV software technology stack.
+
+Proposed methodology is complementary to existing NFV benchmarking
+industry efforts focusing on vSwitch benchmarking [RFC8204], [TST009]
+and extends the benchmarking scope to NFV services.
+
+This document does not describe a complete benchmarking methodology,
+instead it is focusing on the system under test configuration. Each of
+the compute node configurations identified in this document is
+to be evaluated for NFV service data-plane performance using existing
+and/or emerging network benchmarking standards. This may include
+methodologies specified in [RFC2544], [TST009],
+[draft-vpolak-mkonstan-bmwg-mlrsearch] and/or
+[draft-vpolak-bmwg-plrsearch].
+
+# NFV Service
+
+It is assumed that each NFV service instance is built of one or more
+constituent NFs and is described by: topology, configuration and
+resulting packet path(s).
+
+Each set of NFs forms an independent NFV service instance, with multiple
+sets present in the host.
+
+## Topology
+
+NFV topology describes the number of network functions per service
+instance, and their inter-connections over packet interfaces. It
+includes all point-to-point virtual packet links within the compute
+node, Layer-2 Ethernet or Layer-3 IP, including the ones to host
+networking data-plane.
+
+Theoretically, a large set of possible NFV topologies can be realised
+using software virtualisation topologies, e.g. ring, partial -/full-
+mesh, star, line, tree, ladder. In practice however, only a few
+topologies are in the actual use as NFV services mostly perform either
+bumps-in-a-wire packet operations (e.g. security filtering/inspection,
+monitoring/telemetry) and/or inter-site forwarding decisions (e.g.
+routing, switching).
+
+Two main NFV topologies have been identified so far for NFV service
+density benchmarking:
+
+1. Chain topology: a set of NFs connect to host data-plane with minimum
+ of two virtual interfaces each, enabling host data-plane to
+ facilitate NF to NF service chain forwarding and provide connectivity
+ with external network.
+
+2. Pipeline topology: a set of NFs connect to each other in a line
+ fashion with edge NFs homed to host data-plane. Host data-plane
+ provides connectivity with external network.
+
+In both cases multiple NFV service topologies are running in parallel.
+Both topologies are shown in figures 2. and 3. below.
+
+NF chain topology:
+
+ +-----------------------------------------------------------+
+ | Host Compute Node |
+ | |
+ | SmNF1 SmNF2 SmNFn Service-m |
+ | ... ... ... ... |
+ | S2NF1 S2NF2 S2NFn Service-2 |
+ | +--------+ +--------+ +--------+ |
+ | | S1NF1 | | S1NF2 | | S1NFn | |
+ | | | | | .... | | Service-1 |
+ | | | | | | | |
+ | +-+----+-+ +-+----+-+ + + +-+----+-+ |
+ | | | | | | | | | Virtual |
+ | | |<-CS->| |<-CS->| |<-CS->| | Interfaces |
+ | +-+----+------+----+------+----+------+----+-+ |
+ | | | CS: Chain |
+ | | | Segment |
+ | | Host Data-Plane | |
+ | +-+--+----------------------------------+--+-+ |
+ | | | | | |
+ +-----------------------------------------------------------+
+ | | | | Physical
+ | | | | Interfaces
+ +---+--+----------------------------------+--+--------------+
+ | |
+ | Traffic Generator |
+ | |
+ +-----------------------------------------------------------+
+
+ Figure 2. NF chain topology forming a service instance.
+
+NF pipeline topology:
+
+ +-----------------------------------------------------------+
+ | Host Compute Node |
+ | |
+ | SmNF1 SmNF2 SmNFn Service-m |
+ | ... ... ... ... |
+ | S2NF1 S2NF2 S2NFn Service-2 |
+ | +--------+ +--------+ +--------+ |
+ | | S1NF1 | | S1NF2 | | S1NFn | |
+ | | +--+ +--+ .... +--+ | Service1 |
+ | | | | | | | |
+ | +-+------+ +--------+ +------+-+ |
+ | | | Virtual |
+ | |<-Pipeline Edge Pipeline Edge->| Interfaces |
+ | +-+----------------------------------------+-+ |
+ | | | |
+ | | | |
+ | | Host Data-Plane | |
+ | +-+--+----------------------------------+--+-+ |
+ | | | | | |
+ +-----------------------------------------------------------+
+ | | | | Physical
+ | | | | Interfaces
+ +---+--+----------------------------------+--+--------------+
+ | |
+ | Traffic Generator |
+ | |
+ +-----------------------------------------------------------+
+
+ Figure 3. NF pipeline topology forming a service instance.
+
+
+## Configuration
+
+NFV configuration includes all packet processing functions in NFs
+including Layer-2, Layer-3 and/or Layer-4-to-7 processing as appropriate
+to specific NF and NFV service design. L2 sub- interface encapsulations
+(e.g. 802.1q, 802.1ad) and IP overlay encapsulation (e.g. VXLAN, IPSec,
+GRE) may be represented here too as appropriate, although in most cases
+they are used as external encapsulation and handled by host networking
+data-plane.
+
+NFV configuration determines logical network connectivity that is
+Layer-2 and/or IPv4/IPv6 switching/routing modes, as well as NFV service
+specific aspects. In the context of NFV density benchmarking methodology
+the initial focus is on logical network connectivity between the NFs,
+and no NFV service specific configurations. NF specific functionality is
+emulated using IPv4/IPv6 routing.
+
+Building on the two identified NFV topologies, two common NFV
+configurations are considered:
+
+1. Chain configuration:
+ * Relies on chain topology to form NFV service chains.
+ * NF packet forwarding designs:
+ * IPv4/IPv6 routing.
+ * Requirements for host data-plane:
+ * L2 switching with L2 forwarding context per each NF chain
+ segment, or
+ * IPv4/IPv6 routing with IP forwarding context per each NF chain
+ segment or per NF chain.
+
+2. Pipeline configuration:
+ * Relies on pipeline topology to form NFV service pipelines.
+ * Packet forwarding designs:
+ * IPv4/IPv6 routing.
+ * Requirements for host data-plane:
+ * L2 switching with L2 forwarding context per each NF pipeline
+ edge link, or
+ * IPv4/IPv6 routing with IP forwarding context per each NF pipeline
+ edge link or per NF pipeline.
+
+## Packet Path(s)
+
+NFV packet path(s) describe the actual packet forwarding path(s) used
+for benchmarking, resulting from NFV topology and configuration. They
+are aimed to resemble true packet forwarding actions during the NFV
+service lifecycle.
+
+Based on the specified NFV topologies and configurations two NFV packet
+paths are taken for benchmarking:
+
+1. Snake packet path
+ * Requires chain topology and configuration.
+ * Packets enter the NFV chain through one edge NF and progress to the
+ other edge NF of the chain.
+ * Within the chain, packets follow a zigzagging "snake" path entering
+ and leaving host data-plane as they progress through the NF chain.
+ * Host data-plane is involved in packet forwarding operations between
+ NIC interfaces and edge NFs, as well as between NFs in the chain.
+
+2. Pipeline packet path
+ * Requires pipeline topology and configuration.
+ * Packets enter the NFV chain through one edge NF and progress to the
+ other edge NF of the pipeline.
+ * Within the chain, packets follow a straight path entering and
+ leaving subsequent NFs as they progress through the NF pipeline.
+ * Host data-plane is involved in packet forwarding operations between
+ NIC interfaces and edge NFs only.
+
+Both packet paths are shown in figures below.
+
+Snake packet path:
+
+ +-----------------------------------------------------------+
+ | Host Compute Node |
+ | |
+ | SmNF1 SmNF2 SmNFn Service-m |
+ | ... ... ... ... |
+ | S2NF1 S2NF2 S2NFn Service-2 |
+ | +--------+ +--------+ +--------+ |
+ | | S1NF1 | | S1NF2 | | S1NFn | |
+ | | | | | .... | | Service1 |
+ | | XXXX | | XXXX | | XXXX | |
+ | +-+X--X+-+ +-+X--X+-+ +X X+ +-+X--X+-+ |
+ | |X X| |X X| |X X| |X X| Virtual |
+ | |X X| |X X| |X X| |X X| Interfaces |
+ | +-+X--X+------+X--X+------+X--X+------+X--X+-+ |
+ | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | |
+ | | X X | |
+ | | X Host Data-Plane X | |
+ | +-+X-+----------------------------------+-X+-+ |
+ | |X | | X| |
+ +----X--------------------------------------X---------------+
+ |X | | X| Physical
+ |X | | X| Interfaces
+ +---+X-+----------------------------------+-X+--------------+
+ | |
+ | Traffic Generator |
+ | |
+ +-----------------------------------------------------------+
+
+ Figure 4. Snake packet path thru NF chain topology.
+
+
+Pipeline packet path:
+
+ +-----------------------------------------------------------+
+ | Host Compute Node |
+ | |
+ | SmNF1 SmNF2 SmNFn Service-m |
+ | ... ... ... ... |
+ | S2NF1 S2NF2 S2NFn Service-2 |
+ | +--------+ +--------+ +--------+ |
+ | | S1NF1 | | S1NF2 | | S1NFn | |
+ | | +--+ +--+ .... +--+ | Service1 |
+ | | XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXX | |
+ | +--X-----+ +--------+ +-----X--+ |
+ | |X X| Virtual |
+ | |X X| Interfaces |
+ | +-+X--------------------------------------X+-+ |
+ | | X X | |
+ | | X X | |
+ | | X Host Data-Plane X | |
+ | +-+X-+----------------------------------+-X+-+ |
+ | |X | | X| |
+ +----X--------------------------------------X---------------+
+ |X | | X| Physical
+ |X | | X| Interfaces
+ +---+X-+----------------------------------+-X+--------------+
+ | |
+ | Traffic Generator |
+ | |
+ +-----------------------------------------------------------+
+
+ Figure 5. Pipeline packet path thru NF pipeline topology.
+
+In all cases packets enter NFV system via shared physical NIC interfaces
+controlled by shared host data-plane, are then associated with specific
+NFV service (based on service discriminator) and subsequently are cross-
+connected/switched/routed by host data-plane to and through NF
+topologies per one of the above listed schemes.
+
+# Virtualization Technology
+
+NFV services are built of composite isolated NFs, with virtualisation
+technology providing the workload isolation. Following virtualisation
+technology types are considered for NFV service density benchmarking:
+
+1. Virtual Machines (VMs)
+ * Relying on host hypervisor technology e.g. KVM, ESXi, Xen.
+ * NFs running in VMs are referred to as VNFs.
+2. Containers
+ * Relying on Linux container technology e.g. LXC, Docker.
+ * NFs running in Containers are referred to as CNFs.
+
+Different virtual interface types are available to VNFs and CNFs:
+
+1. VNF
+ * virtio-vhostuser: fully user-mode based virtual interface.
+ * virtio-vhostnet: involves kernel-mode based backend.
+2. CNF
+ * memif: fully user-mode based virtual interface.
+ * af_packet: involves kernel-mode based backend.
+ * (add more common ones)
+
+# Host Networking
+
+Host networking data-plane is the central shared resource that underpins
+creation of NFV services. It handles all of the connectivity to external
+physical network devices through physical network connections using
+NICs, through which the benchmarking is done.
+
+Assuming that NIC interface resources are shared, here is the list of
+widely available host data-plane options for providing packet
+connectivity to/from NICs and constructing NFV chain and pipeline
+topologies and configurations:
+
+* Linux Kernel-Mode Networking.
+* Linux User-Mode vSwitch.
+* Virtual Machine vSwitch.
+* Linux Container vSwitch.
+* SRIOV NIC Virtual Function - note: restricted support for chain and
+ pipeline topologies, as it requires hair-pinning through the NIC and
+ oftentimes also through external physical switch.
+
+Analysing properties of each of these options and their Pros/Cons for
+specified NFV topologies and configurations is outside the scope of this
+document.
+
+From all listed options, performance optimised Linux user-mode vswitch
+deserves special attention. Linux user-mode switch decouples NFV service
+from the underlying NIC hardware, offers rich multi-tenant functionality
+and most flexibility for supporting NFV services. But in the same time
+it is consuming compute resources and is harder to benchmark in NFV
+service density scenarios.
+
+Following sections focus on using Linux user-mode vSwitch, focusing on
+its performance benchmarking at increasing levels of NFV service
+density.
+
+# NFV Service Density Matrix
+
+In order to evaluate performance of multiple NFV services running on a
+compute node, NFV service instances are benchmarked at increasing
+density, allowing to construct an NFV Service Density Matrix. Table
+below shows an example of such a matrix, capturing number of NFV service
+instances (row indices), number of NFs per service instance (column
+indices) and resulting total number of NFs (values).
+
+ NFV Service Density - NF Count View
+
+ SVC 001 002 004 006 008 00N
+ 001 1 2 4 6 8 1*N
+ 002 2 4 8 12 16 2*N
+ 004 4 8 16 24 32 4*N
+ 006 6 12 24 36 48 6*N
+ 008 8 16 32 48 64 8*N
+ 00M M*1 M*2 M*4 M*6 M*8 M*N
+
+ RowIndex: Number of NFV Service Instances, 1..M.
+ ColumnIndex: Number of NFs per NFV Service Instance, 1..N.
+ Value: Total number of NFs running in the system.
+
+In order to deliver good and repeatable network data-plane performance,
+NFs and host data-plane software require direct access to critical
+compute resources. Due to a shared nature of all resources on a compute
+node, a clearly defined resource allocation scheme is defined in the
+next section to address this.
+
+In each tested configuration host data-plane is a gateway between the
+external network and the internal NFV network topologies. Offered packet
+load is generated and received by an external traffic generator per
+usual benchmarking practice.
+
+It is proposed that benchmarks are done with the offered packet load
+distributed equally across all configured NFV service instances.
+This approach should provide representative benchmarking data for each
+tested topology and configuraiton, and a good guesstimate of maximum
+performance required for capacity planning.
+
+Following sections specify compute resource allocation, followed by
+examples of applying NFV service density methodology to VNF and CNF
+benchmarking use cases.
+
+# Compute Resource Allocation
+
+Performance optimized NF and host data-plane software threads require
+timely execution of packet processing instructions and are very
+sensitive to any interruptions (or stalls) to this execution e.g. cpu
+core context switching, or cpu jitter. To that end, NFV service density
+methodology treats controlled mapping ratios of data plane software
+threads to physical processor cores with directly allocated cache
+hierarchies as the first order requirement.
+
+Other compute resources including memory bandwidth and PCIe bandwidth
+have lesser impact and as such are subject for further study. For more
+detail and deep-dive analysis of software data plane performance and
+impact on different shared compute resources is available in [BSDP].
+
+It is assumed that NFs as well as host data-plane (e.g. vswitch) are
+performance optimized, with their tasks executed in two types of
+software threads:
+
+* data-plane - handling data-plane packet processing and forwarding,
+ time critical, requires dedicated cores. To scale data-plane
+ performance, most NF apps use multiple data-plane threads and rely on
+ NIC RSS (Receive Side Scaling), virtual interface multi-queue and/or
+ integrated software hashing to distribute packets across the data
+ threads.
+
+* main-control - handling application management, statistics and
+ control-planes, less time critical, allows for core sharing. For most
+ NF apps this is a single main thread, but often statistics (counters)
+ and various control protocol software are run in separate threads.
+
+Core mapping scheme described below allocates cores for all threads of
+specified type belonging to each NF app instance, and separately lists
+number of threads to a number of logical/physical core mappings for
+processor configurations with enabled/disabled Symmetric Multi-
+Threading (SMT) (e.g. AMD SMT, Intel Hyper-Threading).
+
+If NFV service density benchmarking is run on server nodes with
+Symmetric Multi-Threading (SMT) (e.g. AMD SMT, Intel Hyper-Threading)
+for higher performance and efficiency, logical cores allocated to data-
+plane threads should be allocated as pairs of sibling logical cores
+corresponding to the hyper-threads running on the same physical core.
+
+Separate core ratios are defined for mapping threads of vSwitch and NFs.
+In order to get consistent benchmarking results, the mapping ratios are
+enforced using Linux core pinning.
+
+| application | thread type | app:core ratio | threads/pcores (SMT disabled) | threads/lcores map (SMT enabled) |
+|:-----------:|:-----------:|:--------------:|:-------------------------------:|:----------------------------------:|
+| vSwitch-1c | data | 1:1 | 1DT/1PC | 2DT/2LC |
+| | main | 1:S2 | 1MT/S2PC | 1MT/1LC |
+| | | | | |
+| vSwitch-2c | data | 1:2 | 2DT/2PC | 4DT/4LC |
+| | main | 1:S2 | 1MT/S2PC | 1MT/1LC |
+| | | | | |
+| vSwitch-4c | data | 1:4 | 4DT/4PC | 8DT/8LC |
+| | main | 1:S2 | 1MT/S2PC | 1MT/1LC |
+| | | | | |
+| NF-0.5c | data | 1:S2 | 1DT/S2PC | 1DT/1LC |
+| | main | 1:S2 | 1MT/S2PC | 1MT/1LC |
+| | | | | |
+| NF-1c | data | 1:1 | 1DT/1PC | 2DT/2LC |
+| | main | 1:S2 | 1MT/S2PC | 1MT/1LC |
+| | | | | |
+| NF-2c | data | 1:2 | 2DT/2PC | 4DT/4LC |
+| | main | 1:S2 | 1MT/S2PC | 1MT/1LC |
+
+* Legend to table
+ * Header row
+ * application - network application with optimized data-plane, a
+ vSwitch or Network Function (NF) application.
+ * thread type - either "data", short for data-plane; or "main",
+ short for all main-control threads.
+ * app:core ratio - ratio of per application instance threads of
+ specific thread type to physical cores.
+ * threads/pcores (SMT disabled) - number of threads of specific
+ type (DT for data-plane thread, MT for main thread) running on a
+ number of physical cores, with SMT disabled.
+ * threads/lcores map (SMT enabled) - number of threads of specific
+ type (DT, MT) running on a number of logical cores, with SMT
+ enabled. Two logical cores per one physical core.
+ * Content rows
+ * vSwitch-(1c|2c|4c) - vSwitch with 1 physical core (or 2, or 4)
+ allocated to its data-plane software worker threads.
+ * NF-(0.5c|1c|2c) - NF application with half of a physical core (or
+ 1, or 2) allocated to its data-plane software worker threads.
+ * Sn - shared core, sharing ratio of (n).
+ * DT - data-plane thread.
+ * MT - main-control thread.
+ * PC - physical core, with SMT/HT enabled has many (mostly 2 today)
+ logical cores associated with it.
+ * LC - logical core, if more than one lc get allocated in sets of
+ two sibling logical cores running on the same physical core.
+ * SnPC - shared physical core, sharing ratio of (n).
+ * SnLC - shared logical core, sharing ratio of (n).
+
+Maximum benchmarked NFV service densities are limited by a number of
+physical cores on a compute node.
+
+A sample physical core usage view is shown in the matrix below.
+
+ NFV Service Density - Core Usage View
+ vSwitch-1c, NF-1c
+
+ SVC 001 002 004 006 008 010
+ 001 2 3 6 9 12 15
+ 002 3 6 12 18 24 30
+ 004 6 12 24 36 48 60
+ 006 9 18 36 54 72 90
+ 008 12 24 48 72 96 120
+ 010 15 30 60 90 120 150
+
+ RowIndex: Number of NFV Service Instances, 1..10.
+ ColumnIndex: Number of NFs per NFV Service Instance, 1..10.
+ Value: Total number of physical processor cores used for NFs.
+
+# NFV Service Data-Plane Benchmarking
+
+NF service density scenarios should have their data-plane performance
+benchmarked using existing and/or emerging network benchmarking
+standards as noted earlier.
+
+Following metrics should be measured (or calculated) and reported:
+
+* Packet throughput rate (packets-per-second)
+ * Specific to tested packet size or packet sequence (e.g. some type of
+ packet size mix sent in recurrent sequence).
+ * Applicable types of throughput rate: NDR, PDR, MRR.
+* (Calculated) Bandwidth throughput rate (bits-per-second) corresponding
+ to the measured packet throughput rate.
+* Packet one-way latency (seconds)
+ * Measured at different packet throughput rates load e.g. light,
+ medium, heavy.
+
+Listed metrics should be itemized per service instance and per direction
+(e.g. forward/reverse) for latency.
+
+# Sample NFV Service Density Benchmarks
+
+To illustrate defined NFV service density applicability, following
+sections describe three sets of NFV service topologies and
+configurations that have been benchmarked in open-source: i) in
+[LFN-FDio-CSIT], a continuous testing and data-plane benchmarking
+project, ii) as part of CNCF CNF Testbed initiative [CNCF-CNF-Testbed]
+and iii) in OPNFV NFVbench project.
+
+In the first two cases each NFV service instance definition is based on
+the same set of NF applications, and varies only by network addressing
+configuration to emulate multi-tenant operating environment.
+
+OPNFV NFVbench project is focusing on benchmarking the actual production
+deployments that are aligned with OPNFV specifications.
+
+## Intrepreting the Sample Results
+
+TODO How to interpret and avoid misreading included results? And how to
+avoid falling into the trap of using these results to draw generilized
+conclusions about performance of different virtualization technologies,
+e.g. VM and Containers, irrespective of deployment scenarios and what
+VNFs and CNFs are in the actual use.
+
+## Benchmarking MRR Throughput
+
+Initial NFV density throughput benchmarks have been performed using
+Maximum Receive Rate (MRR) test methodology defined and used in FD.io
+CSIT.
+
+MRR tests measure the packet forwarding rate under specified Maximum
+Transmit Rate (MTR) packet load offered by traffic generator over a set
+trial duration, regardless of packet loss ratio (PLR). MTR for specified
+Ethernet frame size was set to the bi-directional link rate, 2x 10GbE in
+referred results.
+
+Tests were conducted with two traffic profiles: i) continuous stream of
+64B frames, ii) continuous stream of IMIX sequence of (7x 64B, 4x 570B,
+1x 1518B), all sizes are L2 untagged Ethernet.
+
+NFV service topologies tested include: VNF service chains, CNF service
+chains and CNF service pipelines.
+
+## VNF Service Chain
+
+VNF Service Chain (VSC) topology is tested with KVM hypervisor (Ubuntu
+18.04-LTS), with NFV service instances consisting of NFs running in VMs
+(VNFs). Host data-plane is provided by FD.io VPP vswitch. Virtual
+interfaces are virtio-vhostuser. Snake forwarding packet path is tested
+using [TRex] traffic generator, see figure.
+
+ +-----------------------------------------------------------+
+ | Host Compute Node |
+ | |
+ | +--------+ +--------+ +--------+ |
+ | | S1VNF1 | | S1VNF2 | | S1VNFn | |
+ | | | | | .... | | Service1 |
+ | | XXXX | | XXXX | | XXXX | |
+ | +-+X--X+-+ +-+X--X+-+ +-+X--X+-+ |
+ | |X X| |X X| |X X| Virtual |
+ | |X X| |X X| |X X| |X X| Interfaces |
+ | +-+X--X+------+X--X+------+X--X+------+X--X+-+ |
+ | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | |
+ | | X X | |
+ | | X FD.io VPP vSwitch X | |
+ | +-+X-+----------------------------------+-X+-+ |
+ | |X | | X| |
+ +----X--------------------------------------X---------------+
+ |X | | X| Physical
+ |X | | X| Interfaces
+ +---+X-+----------------------------------+-X+--------------+
+ | |
+ | Traffic Generator (TRex) |
+ | |
+ +-----------------------------------------------------------+
+
+ Figure 6. VNF service chain test setup.
+
+
+## CNF Service Chain
+
+CNF Service Chain (CSC) topology is tested with Docker containers
+(Ubuntu 18.04-LTS), with NFV service instances consisting of NFs running
+in Containers (CNFs). Host data-plane is provided by FD.io VPP vswitch.
+Virtual interfaces are memif. Snake forwarding packet path is tested
+using [TRex] traffic generator, see figure.
+
+ +-----------------------------------------------------------+
+ | Host Compute Node |
+ | |
+ | +--------+ +--------+ +--------+ |
+ | | S1CNF1 | | S1CNF2 | | S1CNFn | |
+ | | | | | .... | | Service1 |
+ | | XXXX | | XXXX | | XXXX | |
+ | +-+X--X+-+ +-+X--X+-+ +-+X--X+-+ |
+ | |X X| |X X| |X X| Virtual |
+ | |X X| |X X| |X X| |X X| Interfaces |
+ | +-+X--X+------+X--X+------+X--X+------+X--X+-+ |
+ | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | |
+ | | X X | |
+ | | X FD.io VPP vSwitch X | |
+ | +-+X-+----------------------------------+-X+-+ |
+ | |X | | X| |
+ +----X--------------------------------------X---------------+
+ |X | | X| Physical
+ |X | | X| Interfaces
+ +---+X-+----------------------------------+-X+--------------+
+ | |
+ | Traffic Generator (TRex) |
+ | |
+ +-----------------------------------------------------------+
+
+ Figure 7. CNF service chain test setup.
+
+## CNF Service Pipeline
+
+CNF Service Pipeline (CSP) topology is tested with Docker containers
+(Ubuntu 18.04-LTS), with NFV service instances consisting of NFs running
+in Containers (CNFs). Host data-plane is provided by FD.io VPP vswitch.
+Virtual interfaces are memif. Pipeline forwarding packet path is tested
+using [TRex] traffic generator, see figure.
+
+ +-----------------------------------------------------------+
+ | Host Compute Node |
+ | |
+ | +--------+ +--------+ +--------+ |
+ | | S1NF1 | | S1NF2 | | S1NFn | |
+ | | +--+ +--+ .... +--+ | Service1 |
+ | | XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXX | |
+ | +--X-----+ +--------+ +-----X--+ |
+ | |X X| Virtual |
+ | |X X| Interfaces |
+ | +-+X--------------------------------------X+-+ |
+ | | X X | |
+ | | X X | |
+ | | X FD.io VPP vSwitch X | |
+ | +-+X-+----------------------------------+-X+-+ |
+ | |X | | X| |
+ +----X--------------------------------------X---------------+
+ |X | | X| Physical
+ |X | | X| Interfaces
+ +---+X-+----------------------------------+-X+--------------+
+ | |
+ | Traffic Generator (TRex) |
+ | |
+ +-----------------------------------------------------------+
+
+ Figure 8. CNF service chain test setup.
+
+## Sample Results: FD.io CSIT
+
+FD.io CSIT project introduced NFV density benchmarking in release
+CSIT-1904 and published results for the following NFV service topologies
+and configurations:
+
+1. VNF Service Chains
+ * VNF: DPDK-L3FWD v19.02
+ * IPv4 forwarding
+ * NF-1c
+ * vSwitch: VPP v19.04-release
+ * L2 MAC switching
+ * vSwitch-1c, vSwitch-2c
+ * frame sizes: 64B, IMIX
+2. CNF Service Chains
+ * CNF: VPP v19.04-release
+ * IPv4 routing
+ * NF-1c
+ * vSwitch: VPP v19.04-release
+ * L2 MAC switching
+ * vSwitch-1c, vSwitch-2c
+ * frame sizes: 64B, IMIX
+3. CNF Service Pipelines
+ * CNF: VPP v19.04-release
+ * IPv4 routing
+ * NF-1c
+ * vSwitch: VPP v19.04-release
+ * L2 MAC switching
+ * vSwitch-1c, vSwitch-2c
+ * frame sizes: 64B, IMIX
+
+More information is available in FD.io CSIT-1904 report, with specific
+references listed below:
+
+* Testbed: [CSIT-1904-testbed-2n-skx]
+* Test environment: [CSIT-1904-test-enviroment]
+* Methodology: [CSIT-1904-nfv-density-methodology]
+* Results: [CSIT-1904-nfv-density-results]
+
+## Sample Results: CNCF/CNFs
+
+CNCF CI team introduced a CNF testbed initiative focusing on benchmaring
+NFV density with open-source network applications running as VNFs and
+CNFs. Following NFV service topologies and configurations have been
+tested to date:
+
+1. VNF Service Chains
+ * VNF: VPP v18.10-release
+ * IPv4 routing
+ * NF-1c
+ * vSwitch: VPP v18.10-release
+ * L2 MAC switching
+ * vSwitch-1c, vSwitch-2c
+ * frame sizes: 64B, IMIX
+2. CNF Service Chains
+ * CNF: VPP v18.10-release
+ * IPv4 routing
+ * NF-1c
+ * vSwitch: VPP v18.10-release
+ * L2 MAC switching
+ * vSwitch-1c, vSwitch-2c
+ * frame sizes: 64B, IMIX
+3. CNF Service Pipelines
+ * CNF: VPP v18.10-release
+ * IPv4 routing
+ * NF-1c
+ * vSwitch: VPP v18.10-release
+ * L2 MAC switching
+ * vSwitch-1c, vSwitch-2c
+ * frame sizes: 64B, IMIX
+
+More information is available in CNCF CNF Testbed github, with summary
+test results presented in summary markdown file, references listed
+below:
+
+* Results: [CNCF-CNF-Testbed-Results]
+
+## Sample Results: OPNFV NFVbench
+
+TODO Add short NFVbench based test description, and NFVbench sweep chart
+with single VM per service instance: Y-axis packet throughput rate or
+bandwidth throughput rate, X-axis number of concurrent service
+instances.
+
+# IANA Considerations
+
+No requests of IANA.
+
+# Security Considerations
+
+Benchmarking activities as described in this memo are limited to
+technology characterization of a DUT/SUT using controlled stimuli in a
+laboratory environment, with dedicated address space and the constraints
+specified in the sections above.
+
+The benchmarking network topology will be an independent test setup and
+MUST NOT be connected to devices that may forward the test traffic into
+a production network or misroute traffic to the test management network.
+
+Further, benchmarking is performed on a "black-box" basis, relying
+solely on measurements observable external to the DUT/SUT.
+
+Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
+benchmarking purposes. Any implications for network security arising
+from the DUT/SUT SHOULD be identical in the lab and in production
+networks.
+
+# Acknowledgements
+
+Thanks to Vratko Polak of FD.io CSIT project and Michael Pedersen of the
+CNCF Testbed initiative for their contributions and useful suggestions.
+Extended thanks to Alec Hothan of OPNFV NFVbench project for numerous
+comments, suggestions and references to his/team work in the
+OPNFV/NVFbench project.
+
+--- back
\ No newline at end of file
Code Review / csit.git / blobdiff