performance labs to address larger scale multi-interface and multi-NIC
performance testing scenarios.
-For test cases that require DUT (VPP) to communicate with VM(s) over vhost-user
-interfaces, N of VM instances are created on SUT1 and SUT2. For N=1 DUT (VPP)
-forwards packets between vhostuser and physical interfaces. For N>1 DUT (VPP) a
-logical service chain forwarding topology is created on DUT (VPP) by applying L2
-or IPv4/IPv6 configuration depending on the test suite.
-DUT (VPP) test topology with N VM instances
-is shown in the figure below including applicable packet flow thru the DUTs and
-VMs (marked in the figure with ``***``).
+For test cases that require DUT (VPP) to communicate with
+VirtualMachines(VMs)/LinuxContainers(LXCs) over vhost-user/memif
+interfaces, N of VM/LXC instances are created on SUT1 and SUT2. For N=1
+DUT forwards packets between vhost/memif and physical interfaces. For
+N>1 DUT a logical service chain forwarding topology is created on DUT by
+applying L2 or IPv4/IPv6 configuration depending on the test suite. DUT
+test topology with N VM/LXC instances is shown in the figure below
+including applicable packet flow thru the DUTs and VMs/LXCs (marked in
+the figure with ``***``).
::
+-------------------------+ +-------------------------+
| +---------+ +---------+ | | +---------+ +---------+ |
- | | VM[1] | | VM[N] | | | | VM[1] | | VM[N] | |
+ | |VM/LXC[1]| |VM/LXC[N]| | | |VM/LXC[1]| |VM/LXC[N]| |
| | ***** | | ***** | | | | ***** | | ***** | |
| +--^---^--+ +--^---^--+ | | +--^---^--+ +--^---^--+ |
| *| |* *| |* | | *| |* *| |* |
**********************| |**********************
+-----------+
-For VM tests, packets are switched by DUT (VPP) multiple times: twice for a
-single VM, three times for two VMs, N+1 times for N VMs.
-Hence the external
-throughput rates measured by TG and listed in this report must be multiplied
-by (N+1) to represent the actual DUT aggregate packet forwarding rate.
-
-Note that reported VPP performance results are specific to the SUTs tested.
-Current LF FD.io SUTs are based on Intel XEON E5-2699v3 2.3GHz CPUs. SUTs with
-other CPUs are likely to yield different results. A good rule of thumb, that
-can be applied to estimate VPP packet thoughput for Phy-to-Phy (NIC-to-NIC,
-PCI-to-PCI) topology, is to expect the forwarding performance to be
-proportional to CPU core frequency, assuming CPU is the only limiting factor
-and all other SUT parameters equivalent to FD.io CSIT environment. The same rule
-of thumb can be also applied for Phy-to-VM-to-Phy (NIC-to-VM-to-NIC) topology,
-but due to much higher dependency on intensive memory operations and
-sensitivity to Linux kernel scheduler settings and behaviour, this estimation
-may not always yield good enough accuracy.
-
-For detailed LF FD.io test bed specification and physical topology please refer
-to `LF FDio CSIT testbed wiki page <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
+For VM/LXC tests, packets are switched by DUT multiple times: twice for
+a single VM/LXC, three times for two VMs/LXCs, N+1 times for N VMs/LXCs.
+Hence the external throughput rates measured by TG and listed in this
+report must be multiplied by (N+1) to represent the actual DUT aggregate
+packet forwarding rate.
+
+Note that reported DUT (VPP) performance results are specific to the
+SUTs tested. Current LF FD.io SUTs are based on Intel XEON E5-2699v3
+2.3GHz CPUs. SUTs with other CPUs are likely to yield different results.
+A good rule of thumb, that can be applied to estimate VPP packet
+thoughput for Phy-to-Phy (NIC-to-NIC, PCI-to-PCI) topology, is to expect
+the forwarding performance to be proportional to CPU core frequency,
+assuming CPU is the only limiting factor and all other SUT parameters
+equivalent to FD.io CSIT environment. The same rule of thumb can be also
+applied for Phy-to-VM/LXC-to-Phy (NIC-to-VM/LXC-to-NIC) topology, but
+due to much higher dependency on intensive memory operations and
+sensitivity to Linux kernel scheduler settings and behaviour, this
+estimation may not always yield good enough accuracy.
+
+For detailed LF FD.io test bed specification and physical topology
+please refer to
+`LF FD.io CSIT testbed wiki page <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
Performance Tests Coverage
--------------------------
- **VXLAN** - VXLAN overlay tunnelling integration with L2XC and L2BD.
- **QoS Policer** - ingress packet rate measuring, marking and limiting
(IPv4).
+ - **CGNAT** - Carrier Grade Network Address Translation tests with varying
+ number of users and ports per user.
- 2port40GE XL710 Intel
------------------------
CSIT |release| follows a common structured naming convention for all
-performance and system functional tests, introduced in CSIT rls1701.
+performance and system functional tests, introduced in CSIT |release-1|.
The naming should be intuitive for majority of the tests. Complete
description of CSIT test naming convention is provided on `CSIT test naming wiki
#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**
- - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
- PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType*
- - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on
- Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching
- with MAC learning, NDR throughput discovery.
- - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE
- on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline
- switching with MAC learning, NDR throughput discovery.
- - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel
- x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
- - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on
- Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput
- discovery.
+ - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
+ PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType*
+ - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on
+ Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching
+ with MAC learning, NDR throughput discovery.
+ - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE
+ on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline
+ switching with MAC learning, NDR throughput discovery.
+ - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel
+ x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
+ - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on
+ Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput
+ discovery.
#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,
P2V2P, NIC2VMchain2NIC, P2V2V2P**
- - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
- PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-
- VirtPortConfig-VMconfig-TestType*
- - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports
- of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain
- switching to/from two vhost interfaces and one VM, NDR throughput
- discovery.
- - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2
- ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
- switching to/from two vhost interfaces and one VM, NDR throughput
- discovery.
- - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2
- ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
- switching to/from four vhost interfaces and two VMs, NDR throughput
- discovery.
+ - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
+ PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-
+ VirtPortConfig-VMconfig-TestType*
+ - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports
+ of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain
+ switching to/from two vhost interfaces and one VM, NDR throughput
+ discovery.
+ - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2
+ ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
+ switching to/from two vhost interfaces and one VM, NDR throughput
+ discovery.
+ - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2
+ ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
+ switching to/from four vhost interfaces and two VMs, NDR throughput
+ discovery.
Methodology: Multi-Thread and Multi-Core
----------------------------------------
-------------------------
CSIT |release| introduced environment configuration changes to KVM Qemu vhost-
-user tests in order to more representatively measure VPP-17.04 performance in
-configurations with vhost-user interfaces and VMs.
+user tests in order to more representatively measure |vpp-release| performance
+in configurations with vhost-user interfaces and VMs.
Current setup of CSIT FD.io performance lab is using tuned settings for more
optimal performance of KVM Qemu:
their interference on CPU cores that are designated for critical software
tasks under test, namely VPP worker threads in host and Testpmd threads in
guest dealing with data plan.
+
+Methodology: IPSec with Intel QAT HW cards
+------------------------------------------
+
+VPP IPSec performance tests are using DPDK cryptodev device driver in
+combination with HW cryptodev devices - Intel QAT 8950 50G - present in
+LF FD.io physical testbeds. DPDK cryptodev can be used for all IPSec
+data plane functions supported by VPP.
+
+Currently CSIT |release| implements following IPSec test cases:
+
+- AES-GCM, CBC-SHA1 ciphers, in combination with IPv4 routed-forwarding
+ with Intel xl710 NIC.
+- CBC-SHA1 ciphers, in combination with LISP-GPE overlay tunneling for
+ IPv4-over-IPv4 with Intel xl710 NIC.
+
+Methodology: TRex Traffic Generator Usage
+-----------------------------------------
+
+The `TRex traffic generator <https://wiki.fd.io/view/TRex>`_ is used for all
+CSIT performance tests. TRex stateless mode is used to measure NDR and PDR
+throughputs using binary search (NDR and PDR discovery tests) and for quick
+checks of DUT performance against the reference NDRs (NDR check tests) for
+specific configuration.
+
+TRex is installed and run on the TG compute node. The typical procedure is:
+
+- If the TRex is not already installed on TG, it is installed in the
+ suite setup phase - see `TRex intallation`_.
+- TRex configuration is set in its configuration file
+ ::
+
+ /etc/trex_cfg.yaml
+
+- TRex is started in the background mode
+ ::
+
+ $ sh -c 'cd /opt/trex-core-2.25/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --iom 0 > /dev/null 2>&1 &' > /dev/null
+
+- There are traffic streams dynamically prepared for each test. The traffic
+ is sent and the statistics obtained using trex_stl_lib.api.STLClient.
+
+**Measuring packet loss**
+
+- Create an instance of STLClient
+- Connect to the client
+- Add all streams
+- Clear statistics
+- Send the traffic for defined time
+- Get the statistics
+
+If there is a warm-up phase required, the traffic is sent also before test and
+the statistics are ignored.
+
+**Measuring latency**
+
+If measurement of latency is requested, two more packet streams are created (one
+for each direction) with TRex flow_stats parameter set to STLFlowLatencyStats. In
+that case, returned statistics will also include min/avg/max latency values.
Code Review / csit.git / blobdiff