-Overview\r
-========\r
-\r
-Tested Physical Topologies\r
---------------------------\r
-\r
-CSIT VPP performance tests are executed on physical baremetal servers hosted by LF\r
-FD.io project. Testbed physical topology is shown in the figure below.\r
-\r
-::\r
-\r
- +------------------------+ +------------------------+\r
- | | | |\r
- | +------------------+ | | +------------------+ |\r
- | | | | | | | |\r
- | | <-----------------> | |\r
- | | DUT1 | | | | DUT2 | |\r
- | +--^---------------+ | | +---------------^--+ |\r
- | | | | | |\r
- | | SUT1 | | SUT2 | |\r
- +------------------------+ +------------------^-----+\r
- | |\r
- | |\r
- | +-----------+ |\r
- | | | |\r
- +------------------> TG <------------------+\r
- | |\r
- +-----------+\r
-\r
-SUT1 and SUT2 are two System Under Test servers (Cisco UCS C240, each with two\r
-Intel XEON CPUs), TG is a Traffic Generator (TG, another Cisco UCS C240, with\r
-two Intel XEON CPUs). SUTs run VPP SW application in Linux user-mode as a\r
-Device Under Test (DUT). TG runs TRex SW application as a packet Traffic\r
-Generator. Physical connectivity between SUTs and to TG is provided using\r
-different NIC models that need to be tested for performance. Currently\r
-installed and tested NIC models include:\r
-\r
-#. 2port10GE X520-DA2 Intel.\r
-#. 2port10GE X710 Intel.\r
-#. 2port10GE VIC1227 Cisco.\r
-#. 2port40GE VIC1385 Cisco.\r
-#. 2port40GE XL710 Intel.\r
-\r
-From SUT and DUT perspective, all performance tests involve forwarding packets\r
-between two physical Ethernet ports (10GE or 40GE). Due to the number of\r
-listed NIC models tested and available PCI slot capacity in SUT servers, in\r
-all of the above cases both physical ports are located on the same NIC. In\r
-some test cases this results in measured packet throughput being limited not\r
-by VPP DUT but by either the physical interface or the NIC capacity.\r
-\r
-Going forward CSIT project will be looking to add more hardware into FD.io\r
-performance labs to address larger scale multi-interface and multi-NIC\r
-performance testing scenarios.\r
-\r
-For test cases that require DUT (VPP) to communicate with VM(s) over vhost-user\r
-interfaces, N of VM instances are created on SUT1 and SUT2. For N=1 DUT (VPP) forwards packets between vhostuser and physical interfaces. For N>1 DUT (VPP) a logical service chain forwarding topology is created on DUT (VPP) by applying L2 or IPv4/IPv6 configuration depending on the test suite.\r
-DUT (VPP) test topology with N VM instances\r
-is shown in the figure below including applicable packet flow thru the DUTs and VMs\r
-(marked in the figure with ``***``).\r
-\r
-::\r
-\r
- +-------------------------+ +-------------------------+\r
- | +---------+ +---------+ | | +---------+ +---------+ |\r
- | | VM[1] | | VM[N] | | | | VM[1] | | VM[N] | |\r
- | | ***** | | ***** | | | | ***** | | ***** | |\r
- | +--^---^--+ +--^---^--+ | | +--^---^--+ +--^---^--+ |\r
- | *| |* *| |* | | *| |* *| |* |\r
- | +--v---v-------v---v--+ | | +--v---v-------v---v--+ |\r
- | | * * * * |*|***********|*| * * * * | |\r
- | | * ********* ***<-|-----------|->*** ********* * | |\r
- | | * DUT1 | | | | DUT2 * | |\r
- | +--^------------------+ | | +------------------^--+ |\r
- | *| | | |* |\r
- | *| SUT1 | | SUT2 |* |\r
- +-------------------------+ +-------------------------+\r
- *| |*\r
- *| |*\r
- *| +-----------+ |*\r
- *| | | |*\r
- *+--------------------> TG <--------------------+*\r
- **********************| |**********************\r
- +-----------+\r
-\r
-For VM tests, packets are switched by DUT (VPP) multiple times: twice for a single VM, three times for two VMs, N+1 times for N VMs.\r
-Hence the external\r
-throughput rates measured by TG and listed in this report must be multiplied\r
-by (N+1) to represent the actual DUT aggregate packet forwarding rate.\r
-\r
-CSIT |release|\r
-\r
-Note that reported VPP performance results are specific to the SUTs tested.\r
-Current LF FD.io SUTs are based on Intel XEON E5-2699v3 2.3GHz CPUs. SUTs with\r
-other CPUs are likely to yield different results. A good rule of thumb, that\r
-can be applied to estimate VPP packet thoughput for Phy-to-Phy (NIC-to-NIC,\r
-PCI-to-PCI) topology, is to expect the forwarding performance to be\r
-proportional to CPU core frequency, assuming CPU is the only limiting factor\r
-and all other SUT parameters equivalent to FD.io CSIT environment. The same rule of\r
-thumb can be also applied for Phy-to-VM-to-Phy (NIC-to-VM-to-NIC) topology,\r
-but due to much higher dependency on intensive memory operations and\r
-sensitivity to Linux kernel scheduler settings and behaviour, this estimation\r
-may not always yield good enough accuracy.\r
-\r
-For detailed LF FD.io test bed specification and physical topology please refer to `LF FDio CSIT testbed wiki page <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.\r
-\r
-Performance Tests Coverage\r
---------------------------\r
-\r
-Performance tests are split into the two main categories:\r
-\r
-- Throughput discovery - discovery of packet forwarding rate using binary search\r
- in accordance to RFC2544.\r
-\r
- - NDR - discovery of Non Drop Rate packet throughput, at zero packet loss;\r
- followed by one-way packet latency measurements at 10%, 50% and 100% of\r
- discovered NDR throughput.\r
- - PDR - discovery of Partial Drop Rate, with specified non-zero packet loss\r
- currently set to 0.5%; followed by one-way packet latency measurements at\r
- 100% of discovered PDR throughput.\r
-\r
-- Throughput verification - verification of packet forwarding rate against\r
- previously discovered throughput rate. These tests are currently done against\r
- 0.9 of reference NDR, with reference rates updated periodically.\r
-\r
-CSIT |release| includes following performance test suites, listed per NIC type:\r
-\r
-- 2port10GE X520-DA2 Intel\r
-\r
- - **L2XC** - L2 Cross-Connect switched-forwarding of untagged, dot1q, dot1ad\r
- VLAN tagged Ethernet frames.\r
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
- with MAC learning; disabled MAC learning i.e. static MAC tests to be added.\r
- - **IPv4** - IPv4 routed-forwarding.\r
- - **IPv6** - IPv6 routed-forwarding.\r
- - **IPv4 Scale** - IPv4 routed-forwarding with 20k, 200k and 2M FIB entries.\r
- - **IPv6 Scale** - IPv6 routed-forwarding with 20k, 200k and 2M FIB entries.\r
- - **VM with vhost-user** - switching between NIC ports and VM over vhost-user\r
- interfaces in different switching modes incl. L2 Cross-Connect, L2\r
- Bridge-Domain, VXLAN with L2BD, IPv4 routed-forwarding.\r
- - **COP** - IPv4 and IPv6 routed-forwarding with COP address security.\r
- - **iACL** - IPv4 and IPv6 routed-forwarding with iACL address security.\r
- - **LISP** - LISP overlay tunneling for IPv4-over-IPV4, IPv6-over-IPv4,\r
- IPv6-over-IPv6, IPv4-over-IPv6 in IPv4 and IPv6 routed-forwarding modes.\r
- - **VXLAN** - VXLAN overlay tunnelling integration with L2XC and L2BD.\r
- - **QoS Policer** - ingress packet rate measuring, marking and limiting\r
- (IPv4).\r
-\r
-- 2port40GE XL710 Intel\r
-\r
- - **L2XC** - L2 Cross-Connect switched-forwarding of untagged Ethernet frames.\r
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
- with MAC learning.\r
- - **IPv4** - IPv4 routed-forwarding.\r
- - **IPv6** - IPv6 routed-forwarding.\r
- - **VM with vhost-user** - switching between NIC ports and VM over vhost-user\r
- interfaces in different switching modes incl. L2 Bridge-Domain.\r
-\r
-- 2port10GE X710 Intel\r
-\r
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
- with MAC learning.\r
- - **VM with vhost-user** - switching between NIC ports and VM over vhost-user\r
- interfaces in different switching modes incl. L2 Bridge-Domain.\r
-\r
-- 2port10GE VIC1227 Cisco\r
-\r
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
- with MAC learning.\r
-\r
-- 2port40GE VIC1385 Cisco\r
-\r
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames\r
- with MAC learning.\r
-\r
-Execution of performance tests takes time, especially the throughput discovery\r
-tests. Due to limited HW testbed resources available within FD.io labs hosted\r
-by Linux Foundation, the number of tests for NICs other than X520 (a.k.a.\r
-Niantic) has been limited to few baseline tests. Over time we expect the HW\r
-testbed resources to grow, and will be adding complete set of performance\r
-tests for all models of hardware to be executed regularly and(or)\r
-continuously.\r
-\r
-Performance Tests Naming\r
-------------------------\r
-\r
-CSIT |release| follows a common structured naming convention for all\r
-performance and system functional tests, introduced in CSIT rls1701.\r
-\r
-The naming should be intuitive for majority of the tests. Complete\r
-description of CSIT test naming convention is provided on `CSIT test naming wiki\r
-<https://wiki.fd.io/view/CSIT/csit-test-naming>`_.\r
-\r
-Here few illustrative examples of the new naming usage for performance test\r
-suites:\r
-\r
-#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**\r
-\r
- - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-\r
- PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType*\r
- - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on\r
- Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching\r
- with MAC learning, NDR throughput discovery.\r
- - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE\r
- on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline\r
- switching with MAC learning, NDR throughput discovery.\r
- - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel\r
- x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.\r
- - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on\r
- Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput\r
- discovery.\r
-\r
-#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,\r
- P2V2P, NIC2VMchain2NIC, P2V2V2P**\r
-\r
- - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-\r
- PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-\r
- VirtPortConfig-VMconfig-TestType*\r
- - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports\r
- of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain\r
- switching to/from two vhost interfaces and one VM, NDR throughput\r
- discovery.\r
- - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2\r
- ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain\r
- switching to/from two vhost interfaces and one VM, NDR throughput\r
- discovery.\r
- - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2\r
- ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain\r
- switching to/from four vhost interfaces and two VMs, NDR throughput\r
- discovery.\r
-\r
-Methodology: Multi-Thread and Multi-Core\r
-----------------------------------------\r
-\r
-**HyperThreading** - CSIT |release| performance tests are executed with SUT\r
-servers' Intel XEON CPUs configured in HyperThreading Disabled mode (BIOS\r
-settings). This is the simplest configuration used to establish baseline\r
-single-thread single-core SW packet processing and forwarding performance.\r
-Subsequent releases of CSIT will add performance tests with Intel\r
-HyperThreading Enabled (requires BIOS settings change and hard reboot).\r
-\r
-**Multi-core Test** - CSIT |release| multi-core tests are executed in the\r
-following VPP thread and core configurations:\r
-\r
-#. 1t1c - 1 VPP worker thread on 1 CPU physical core.\r
-#. 2t2c - 2 VPP worker threads on 2 CPU physical cores.\r
-\r
-Note that in quite a few test cases running VPP on 2 physical cores hits\r
-the tested NIC I/O bandwidth or packets-per-second limit.\r
-\r
-Methodology: Packet Throughput\r
-------------------------------\r
-\r
-Following values are measured and reported for packet throughput tests:\r
-\r
-- NDR binary search per RFC2544:\r
-\r
- - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps\r
- (2x <per direction packets-per-second>)"\r
- - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per\r
- second> Gbps (untagged)"\r
-\r
-- PDR binary search per RFC2544:\r
-\r
- - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x\r
- <per direction packets-per-second>)"\r
- - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per\r
- second> Gbps (untagged)"\r
- - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets\r
- lost at PDR rate>""\r
-\r
-- NDR and PDR are measured for the following L2 frame sizes:\r
-\r
- - IPv4: 64B, IMIX_v4_1 (28x64B,16x570B,4x1518B), 1518B, 9000B.\r
- - IPv6: 78B, 1518B, 9000B.\r
-\r
-\r
-Methodology: Packet Latency\r
----------------------------\r
-\r
-TRex Traffic Generator (TG) is used for measuring latency of VPP DUTs. Reported\r
-latency values are measured using following methodology:\r
-\r
-- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate)\r
- for each NDR throughput test and packet size (except IMIX).\r
-- TG sends dedicated latency streams, one per direction, each at the rate of\r
- 10kpps at the prescribed packet size; these are sent in addition to the main\r
- load streams.\r
-- TG reports min/avg/max latency values per stream direction, hence two sets\r
- of latency values are reported per test case; future release of TRex is\r
- expected to report latency percentiles.\r
-- Reported latency values are aggregate across two SUTs due to three node\r
- topology used for all performance tests; for per SUT latency, reported value\r
- should be divided by two.\r
-- 1usec is the measurement accuracy advertised by TRex TG for the setup used in\r
- FD.io labs used by CSIT project.\r
-- TRex setup introduces an always-on error of about 2*2usec per latency flow -\r
- additonal Tx/Rx interface latency induced by TRex SW writing and reading\r
- packet timestamps on CPU cores without HW acceleration on NICs closer to the\r
- interface line.\r
-\r
-\r
-Methodology: KVM VM vhost\r
--------------------------\r
-\r
-CSIT |release| introduced environment configuration changes to KVM Qemu vhost-\r
-user tests in order to more representatively measure VPP-17.01 performance in\r
-configurations with vhost-user interfaces and VMs.\r
-\r
-Current setup of CSIT FD.io performance lab is using tuned settings for more\r
-optimal performance of KVM Qemu:\r
-\r
-- Default Qemu virtio queue size of 256 descriptors.\r
-- Adjusted Linux kernel CFS scheduler settings, as detailed on this CSIT wiki\r
- page: https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604.\r
-\r
-Adjusted Linux kernel CFS settings make the NDR and PDR throughput performance\r
-of VPP+VM system less sensitive to other Linux OS system tasks by reducing\r
-their interference on CPU cores that are designated for critical software\r
-tasks under test, namely VPP worker threads in host and Testpmd threads in\r
-guest dealing with data plan.\r
+Overview
+========
+
+Tested Physical Topologies
+--------------------------
+
+CSIT VPP performance tests are executed on physical baremetal servers hosted by
+LF FD.io project. Testbed physical topology is shown in the figure below.
+
+::
+
+ +------------------------+ +------------------------+
+ | | | |
+ | +------------------+ | | +------------------+ |
+ | | | | | | | |
+ | | <-----------------> | |
+ | | DUT1 | | | | DUT2 | |
+ | +--^---------------+ | | +---------------^--+ |
+ | | | | | |
+ | | SUT1 | | SUT2 | |
+ +------------------------+ +------------------^-----+
+ | |
+ | |
+ | +-----------+ |
+ | | | |
+ +------------------> TG <------------------+
+ | |
+ +-----------+
+
+SUT1 and SUT2 are two System Under Test servers (Cisco UCS C240, each with two
+Intel XEON CPUs), TG is a Traffic Generator (TG, another Cisco UCS C240, with
+two Intel XEON CPUs). SUTs run VPP SW application in Linux user-mode as a
+Device Under Test (DUT). TG runs TRex SW application as a packet Traffic
+Generator. Physical connectivity between SUTs and to TG is provided using
+different NIC models that need to be tested for performance. Currently
+installed and tested NIC models include:
+
+#. 2port10GE X520-DA2 Intel.
+#. 2port10GE X710 Intel.
+#. 2port10GE VIC1227 Cisco.
+#. 2port40GE VIC1385 Cisco.
+#. 2port40GE XL710 Intel.
+
+From SUT and DUT perspective, all performance tests involve forwarding packets
+between two physical Ethernet ports (10GE or 40GE). Due to the number of
+listed NIC models tested and available PCI slot capacity in SUT servers, in
+all of the above cases both physical ports are located on the same NIC. In
+some test cases this results in measured packet throughput being limited not
+by VPP DUT but by either the physical interface or the NIC capacity.
+
+Going forward CSIT project will be looking to add more hardware into FD.io
+performance labs to address larger scale multi-interface and multi-NIC
+performance testing scenarios.
+
+For test cases that require DUT (VPP) to communicate with
+VirtualMachines(VMs)/LinuxContainers(LXCs) over vhost-user/memif
+interfaces, N of VM/LXC instances are created on SUT1 and SUT2. For N=1
+DUT forwards packets between vhost/memif and physical interfaces. For
+N>1 DUT a logical service chain forwarding topology is created on DUT by
+applying L2 or IPv4/IPv6 configuration depending on the test suite. DUT
+test topology with N VM/LXC instances is shown in the figure below
+including applicable packet flow thru the DUTs and VMs/LXCs (marked in
+the figure with ``***``).
+
+::
+
+ +-------------------------+ +-------------------------+
+ | +---------+ +---------+ | | +---------+ +---------+ |
+ | |VM/LXC[1]| |VM/LXC[N]| | | |VM/LXC[1]| |VM/LXC[N]| |
+ | | ***** | | ***** | | | | ***** | | ***** | |
+ | +--^---^--+ +--^---^--+ | | +--^---^--+ +--^---^--+ |
+ | *| |* *| |* | | *| |* *| |* |
+ | +--v---v-------v---v--+ | | +--v---v-------v---v--+ |
+ | | * * * * |*|***********|*| * * * * | |
+ | | * ********* ***<-|-----------|->*** ********* * | |
+ | | * DUT1 | | | | DUT2 * | |
+ | +--^------------------+ | | +------------------^--+ |
+ | *| | | |* |
+ | *| SUT1 | | SUT2 |* |
+ +-------------------------+ +-------------------------+
+ *| |*
+ *| |*
+ *| +-----------+ |*
+ *| | | |*
+ *+--------------------> TG <--------------------+*
+ **********************| |**********************
+ +-----------+
+
+For VM/LXC tests, packets are switched by DUT multiple times: twice for
+a single VM/LXC, three times for two VMs/LXCs, N+1 times for N VMs/LXCs.
+Hence the external throughput rates measured by TG and listed in this
+report must be multiplied by (N+1) to represent the actual DUT aggregate
+packet forwarding rate.
+
+Note that reported DUT (VPP) performance results are specific to the
+SUTs tested. Current LF FD.io SUTs are based on Intel XEON E5-2699v3
+2.3GHz CPUs. SUTs with other CPUs are likely to yield different results.
+A good rule of thumb, that can be applied to estimate VPP packet
+thoughput for Phy-to-Phy (NIC-to-NIC, PCI-to-PCI) topology, is to expect
+the forwarding performance to be proportional to CPU core frequency,
+assuming CPU is the only limiting factor and all other SUT parameters
+equivalent to FD.io CSIT environment. The same rule of thumb can be also
+applied for Phy-to-VM/LXC-to-Phy (NIC-to-VM/LXC-to-NIC) topology, but
+due to much higher dependency on intensive memory operations and
+sensitivity to Linux kernel scheduler settings and behaviour, this
+estimation may not always yield good enough accuracy.
+
+For detailed LF FD.io test bed specification and physical topology
+please refer to
+`LF FD.io CSIT testbed wiki page <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
+
+Performance Tests Coverage
+--------------------------
+
+Performance tests are split into two main categories:
+
+- Throughput discovery - discovery of packet forwarding rate using binary search
+ in accordance to RFC2544.
+
+ - NDR - discovery of Non Drop Rate packet throughput, at zero packet loss;
+ followed by one-way packet latency measurements at 10%, 50% and 100% of
+ discovered NDR throughput.
+ - PDR - discovery of Partial Drop Rate, with specified non-zero packet loss
+ currently set to 0.5%; followed by one-way packet latency measurements at
+ 100% of discovered PDR throughput.
+
+- Throughput verification - verification of packet forwarding rate against
+ previously discovered throughput rate. These tests are currently done against
+ 0.9 of reference NDR, with reference rates updated periodically.
+
+CSIT |release| includes following performance test suites, listed per NIC type:
+
+- 2port10GE X520-DA2 Intel
+
+ - **L2XC** - L2 Cross-Connect switched-forwarding of untagged, dot1q, dot1ad
+ VLAN tagged Ethernet frames.
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
+ with MAC learning; disabled MAC learning i.e. static MAC tests to be added.
+ - **IPv4** - IPv4 routed-forwarding.
+ - **IPv6** - IPv6 routed-forwarding.
+ - **IPv4 Scale** - IPv4 routed-forwarding with 20k, 200k and 2M FIB entries.
+ - **IPv6 Scale** - IPv6 routed-forwarding with 20k, 200k and 2M FIB entries.
+ - **VMs with vhost-user** - virtual topologies with 1 VM and service chains
+ of 2 VMs using vhost-user interfaces, with VPP forwarding modes incl. L2
+ Cross-Connect, L2 Bridge-Domain, VXLAN with L2BD, IPv4 routed-forwarding.
+ - **COP** - IPv4 and IPv6 routed-forwarding with COP address security.
+ - **iACL** - IPv4 and IPv6 routed-forwarding with iACL address security.
+ - **LISP** - LISP overlay tunneling for IPv4-over-IPv4, IPv6-over-IPv4,
+ IPv6-over-IPv6, IPv4-over-IPv6 in IPv4 and IPv6 routed-forwarding modes.
+ - **VXLAN** - VXLAN overlay tunnelling integration with L2XC and L2BD.
+ - **QoS Policer** - ingress packet rate measuring, marking and limiting
+ (IPv4).
+ - **CGNAT** - Carrier Grade Network Address Translation tests with varying
+ number of users and ports per user.
+
+- 2port40GE XL710 Intel
+
+ - **L2XC** - L2 Cross-Connect switched-forwarding of untagged Ethernet frames.
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
+ with MAC learning.
+ - **IPv4** - IPv4 routed-forwarding.
+ - **IPv6** - IPv6 routed-forwarding.
+ - **VMs with vhost-user** - virtual topologies with 1 VM and service chains
+ of 2 VMs using vhost-user interfaces, with VPP forwarding modes incl. L2
+ Cross-Connect, L2 Bridge-Domain, VXLAN with L2BD, IPv4 routed-forwarding.
+ - **IPSec** - IPSec encryption with AES-GCM, CBC-SHA1 ciphers, in combination
+ with IPv4 routed-forwarding.
+ - **IPSec+LISP** - IPSec encryption with CBC-SHA1 ciphers, in combination
+ with LISP-GPE overlay tunneling for IPv4-over-IPv4.
+
+- 2port10GE X710 Intel
+
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
+ with MAC learning.
+ - **VMs with vhost-user** - virtual topologies with 1 VM using vhost-user
+ interfaces, with VPP forwarding modes incl. L2 Bridge-Domain.
+
+- 2port10GE VIC1227 Cisco
+
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
+ with MAC learning.
+
+- 2port40GE VIC1385 Cisco
+
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
+ with MAC learning.
+
+Execution of performance tests takes time, especially the throughput
+discovery tests. Due to limited HW testbed resources available within
+FD.io labs hosted by Linux Foundation, the number of tests for NICs
+other than X520 (a.k.a. Niantic) has been limited to few baseline tests.
+CSIT team expect the HW testbed resources to grow over time, so that
+complete set of performance tests can be regularly and(or) continuously
+executed against all models of hardware present in FD.io labs.
+
+Performance Tests Naming
+------------------------
+
+CSIT |release| follows a common structured naming convention for all
+performance and system functional tests, introduced in CSIT |release-1|.
+
+The naming should be intuitive for majority of the tests. Complete
+description of CSIT test naming convention is provided on `CSIT test naming wiki
+<https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
+
+Methodology: Multi-Core and Multi-Threading
+-------------------------------------------
+
+**Intel Hyper-Threading** - CSIT |release| performance tests are
+executed with SUT servers' Intel XEON processors configured in Intel
+Hyper-Threading Disabled mode (BIOS setting). This is the simplest
+configuration used to establish baseline single-thread single-core
+application packet processing and forwarding performance. Subsequent
+releases of CSIT will add performance tests with Intel Hyper-Threading
+Enabled (requires BIOS settings change and hard reboot of server).
+
+**Multi-core Tests** - CSIT |release| multi-core tests are executed in
+the following VPP thread and core configurations:
+
+#. 1t1c - 1 VPP worker thread on 1 CPU physical core.
+#. 2t2c - 2 VPP worker threads on 2 CPU physical cores.
+
+VPP worker threads are the data plane threads. VPP control thread is
+running on a separate non-isolated core together with other Linux
+processes. Note that in quite a few test cases running VPP workers on 2
+physical cores hits the tested NIC I/O bandwidth or packets-per-second
+limit.
+
+Methodology: Packet Throughput
+------------------------------
+
+Following values are measured and reported for packet throughput tests:
+
+- NDR binary search per RFC2544:
+
+ - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps
+ (2x <per direction packets-per-second>)"
+ - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
+ second> Gbps (untagged)"
+
+- PDR binary search per RFC2544:
+
+ - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x
+ <per direction packets-per-second>)"
+ - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
+ second> Gbps (untagged)"
+ - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets
+ lost at PDR rate>""
+
+- NDR and PDR are measured for the following L2 frame sizes:
+
+ - IPv4: 64B, IMIX_v4_1 (28x64B,16x570B,4x1518B), 1518B, 9000B.
+ - IPv6: 78B, 1518B, 9000B.
+
+All rates are reported from external Traffic Generator perspective.
+
+Methodology: Packet Latency
+---------------------------
+
+TRex Traffic Generator (TG) is used for measuring latency of VPP DUTs. Reported
+latency values are measured using following methodology:
+
+- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate)
+ for each NDR throughput test and packet size (except IMIX).
+- TG sends dedicated latency streams, one per direction, each at the rate of
+ 10kpps at the prescribed packet size; these are sent in addition to the main
+ load streams.
+- TG reports min/avg/max latency values per stream direction, hence two sets
+ of latency values are reported per test case; future release of TRex is
+ expected to report latency percentiles.
+- Reported latency values are aggregate across two SUTs due to three node
+ topology used for all performance tests; for per SUT latency, reported value
+ should be divided by two.
+- 1usec is the measurement accuracy advertised by TRex TG for the setup used in
+ FD.io labs used by CSIT project.
+- TRex setup introduces an always-on error of about 2*2usec per latency flow -
+ additonal Tx/Rx interface latency induced by TRex SW writing and reading
+ packet timestamps on CPU cores without HW acceleration on NICs closer to the
+ interface line.
+
+
+Methodology: KVM VM vhost
+-------------------------
+
+CSIT |release| introduced test environment configuration changes to KVM Qemu vhost-
+user tests in order to more representatively measure |vpp-release| performance
+in configurations with vhost-user interfaces and different Qemu settings.
+
+FD.io CSIT performance lab is testing VPP vhost with KVM VMs using following environment settings
+
+- Tests with varying Qemu virtio queue (a.k.a. vring) sizes:
+ [vr256] default 256 descriptors, [vr1024] 1024 descriptors to
+ optimize for packet throughput;
+
+- Tests with varying Linux CFS (Completely Fair Scheduler)
+ settings: [cfs] default settings, [cfsrr1] CFS RoundRobin(1)
+ policy applied to all data plane threads handling test packet
+ path including all VPP worker threads and all Qemu testpmd
+ poll-mode threads;
+
+- Resulting test cases are all combinations with [vr256,vr1024] and
+ [cfs,cfsrr1] settings;
+
+- Adjusted Linux kernel CFS scheduler policy for data plane threads used
+ in CSIT is documented in
+ `CSIT Performance Environment Tuning wiki <https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604>`_.
+ The purpose is to verify performance impact (NDR, PDR throughput) and
+ same test measurements repeatability, by making VPP and VM data plane
+ threads less susceptible to other Linux OS system tasks hijacking CPU
+ cores running those data plane threads.
+
+Methodology: LXC Container memif
+--------------------------------
+
+CSIT |release| introduced new tests - VPP Memif virtual interface
+(shared memory interface) tests interconnecting VPP instances over
+memif. VPP vswitch instance runs in bare-metal user-mode handling Intel
+x520 NIC 10GbE interfaces and connecting over memif (Master side)
+virtual interfaces to another instance of VPP running in bare-metal
+Linux Container (LXC) with memif virtual interfaces (Slave side). LXC
+runs in a priviliged mode with VPP data plane worker threads pinned to
+dedicated physical CPU cores per usual CSIT practice. Both VPP run the
+same version of software. This test topology is equivalent to existing
+tests with vhost-user and VMs.
+
+Methodology: IPSec with Intel QAT HW cards
+------------------------------------------
+
+VPP IPSec performance tests are using DPDK cryptodev device driver in
+combination with HW cryptodev devices - Intel QAT 8950 50G - present in
+LF FD.io physical testbeds. DPDK cryptodev can be used for all IPSec
+data plane functions supported by VPP.
+
+Currently CSIT |release| implements following IPSec test cases:
+
+- AES-GCM, CBC-SHA1 ciphers, in combination with IPv4 routed-forwarding
+ with Intel xl710 NIC.
+- CBC-SHA1 ciphers, in combination with LISP-GPE overlay tunneling for
+ IPv4-over-IPv4 with Intel xl710 NIC.
+
+Methodology: TRex Traffic Generator Usage
+-----------------------------------------
+
+The `TRex traffic generator <https://wiki.fd.io/view/TRex>`_ is used for all
+CSIT performance tests. TRex stateless mode is used to measure NDR and PDR
+throughputs using binary search (NDR and PDR discovery tests) and for quick
+checks of DUT performance against the reference NDRs (NDR check tests) for
+specific configuration.
+
+TRex is installed and run on the TG compute node. The typical procedure is:
+
+- If the TRex is not already installed on TG, it is installed in the
+ suite setup phase - see `TRex intallation`_.
+- TRex configuration is set in its configuration file
+ ::
+
+ /etc/trex_cfg.yaml
+
+- TRex is started in the background mode
+ ::
+
+ $ sh -c 'cd /opt/trex-core-2.25/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --iom 0 > /dev/null 2>&1 &' > /dev/null
+
+- There are traffic streams dynamically prepared for each test. The traffic
+ is sent and the statistics obtained using trex_stl_lib.api.STLClient.
+
+**Measuring packet loss**
+
+- Create an instance of STLClient
+- Connect to the client
+- Add all streams
+- Clear statistics
+- Send the traffic for defined time
+- Get the statistics
+
+If there is a warm-up phase required, the traffic is sent also before test and
+the statistics are ignored.
+
+**Measuring latency**
+
+If measurement of latency is requested, two more packet streams are created (one
+for each direction) with TRex flow_stats parameter set to STLFlowLatencyStats. In
+that case, returned statistics will also include min/avg/max latency values.
Code Review / csit.git / blobdiff