-CSIT |release| follows a common structured naming convention for all
-performance and system functional tests, introduced in CSIT |release-1|.
-
-The naming should be intuitive for majority of the tests. Complete
-description of CSIT test naming convention is provided on `CSIT test naming wiki
-<https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
-
-Methodology: Multi-Core and Multi-Threading
--------------------------------------------
-
-**Intel Hyper-Threading** - CSIT |release| performance tests are
-executed with SUT servers' Intel XEON processors configured in Intel
-Hyper-Threading Disabled mode (BIOS setting). This is the simplest
-configuration used to establish baseline single-thread single-core
-application packet processing and forwarding performance. Subsequent
-releases of CSIT will add performance tests with Intel Hyper-Threading
-Enabled (requires BIOS settings change and hard reboot of server).
-
-**Multi-core Tests** - CSIT |release| multi-core tests are executed in
-the following VPP thread and core configurations:
-
-#. 1t1c - 1 VPP worker thread on 1 CPU physical core.
-#. 2t2c - 2 VPP worker threads on 2 CPU physical cores.
-
-VPP worker threads are the data plane threads. VPP control thread is
-running on a separate non-isolated core together with other Linux
-processes. Note that in quite a few test cases running VPP workers on 2
-physical cores hits the tested NIC I/O bandwidth or packets-per-second
-limit.
-
-Methodology: Packet Throughput
-------------------------------
-
-Following values are measured and reported for packet throughput tests:
-
-- NDR binary search per RFC2544:
-
- - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps
- (2x <per direction packets-per-second>)"
- - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
- second> Gbps (untagged)"
-
-- PDR binary search per RFC2544:
-
- - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x
- <per direction packets-per-second>)"
- - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
- second> Gbps (untagged)"
- - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets
- lost at PDR rate>""
-
-- NDR and PDR are measured for the following L2 frame sizes:
-
- - IPv4: 64B, IMIX_v4_1 (28x64B,16x570B,4x1518B), 1518B, 9000B.
- - IPv6: 78B, 1518B, 9000B.
-
-All rates are reported from external Traffic Generator perspective.
-
-Methodology: Packet Latency
----------------------------
-
-TRex Traffic Generator (TG) is used for measuring latency of VPP DUTs. Reported
-latency values are measured using following methodology:
-
-- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate)
- for each NDR throughput test and packet size (except IMIX).
-- TG sends dedicated latency streams, one per direction, each at the rate of
- 10kpps at the prescribed packet size; these are sent in addition to the main
- load streams.
-- TG reports min/avg/max latency values per stream direction, hence two sets
- of latency values are reported per test case; future release of TRex is
- expected to report latency percentiles.
-- Reported latency values are aggregate across two SUTs due to three node
- topology used for all performance tests; for per SUT latency, reported value
- should be divided by two.
-- 1usec is the measurement accuracy advertised by TRex TG for the setup used in
- FD.io labs used by CSIT project.
-- TRex setup introduces an always-on error of about 2*2usec per latency flow -
- additonal Tx/Rx interface latency induced by TRex SW writing and reading
- packet timestamps on CPU cores without HW acceleration on NICs closer to the
- interface line.
-
-
-Methodology: KVM VM vhost
--------------------------
-
-CSIT |release| introduced test environment configuration changes to KVM Qemu vhost-
-user tests in order to more representatively measure |vpp-release| performance
-in configurations with vhost-user interfaces and different Qemu settings.
-
-FD.io CSIT performance lab is testing VPP vhost with KVM VMs using following environment settings
-
-- Tests with varying Qemu virtio queue (a.k.a. vring) sizes:
- [vr256] default 256 descriptors, [vr1024] 1024 descriptors to
- optimize for packet throughput;
-
-- Tests with varying Linux CFS (Completely Fair Scheduler)
- settings: [cfs] default settings, [cfsrr1] CFS RoundRobin(1)
- policy applied to all data plane threads handling test packet
- path including all VPP worker threads and all Qemu testpmd
- poll-mode threads;
-
-- Resulting test cases are all combinations with [vr256,vr1024] and
- [cfs,cfsrr1] settings;
-
-- Adjusted Linux kernel CFS scheduler policy for data plane threads used
- in CSIT is documented in
- `CSIT Performance Environment Tuning wiki <https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604>`_.
- The purpose is to verify performance impact (NDR, PDR throughput) and
- same test measurements repeatability, by making VPP and VM data plane
- threads less susceptible to other Linux OS system tasks hijacking CPU
- cores running those data plane threads.
-
-Methodology: LXC Container memif
---------------------------------
-
-CSIT |release| introduced new tests - VPP Memif virtual interface
-(shared memory interface) tests interconnecting VPP instances over
-memif. VPP vswitch instance runs in bare-metal user-mode handling Intel
-x520 NIC 10GbE interfaces and connecting over memif (Master side)
-virtual interfaces to another instance of VPP running in bare-metal
-Linux Container (LXC) with memif virtual interfaces (Slave side). LXC
-runs in a priviliged mode with VPP data plane worker threads pinned to
-dedicated physical CPU cores per usual CSIT practice. Both VPP run the
-same version of software. This test topology is equivalent to existing
-tests with vhost-user and VMs.
-
-Methodology: IPSec with Intel QAT HW cards
-------------------------------------------
-
-VPP IPSec performance tests are using DPDK cryptodev device driver in
-combination with HW cryptodev devices - Intel QAT 8950 50G - present in
-LF FD.io physical testbeds. DPDK cryptodev can be used for all IPSec
-data plane functions supported by VPP.
-
-Currently CSIT |release| implements following IPSec test cases:
-
-- AES-GCM, CBC-SHA1 ciphers, in combination with IPv4 routed-forwarding
- with Intel xl710 NIC.
-- CBC-SHA1 ciphers, in combination with LISP-GPE overlay tunneling for
- IPv4-over-IPv4 with Intel xl710 NIC.
-
-Methodology: TRex Traffic Generator Usage
------------------------------------------
-
-The `TRex traffic generator <https://wiki.fd.io/view/TRex>`_ is used for all
-CSIT performance tests. TRex stateless mode is used to measure NDR and PDR
-throughputs using binary search (NDR and PDR discovery tests) and for quick
-checks of DUT performance against the reference NDRs (NDR check tests) for
-specific configuration.
-
-TRex is installed and run on the TG compute node. The typical procedure is:
-
-- If the TRex is not already installed on TG, it is installed in the
- suite setup phase - see `TRex intallation`_.
-- TRex configuration is set in its configuration file
- ::
-
- /etc/trex_cfg.yaml
-
-- TRex is started in the background mode
- ::
-
- $ sh -c 'cd /opt/trex-core-2.25/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --iom 0 > /dev/null 2>&1 &' > /dev/null
-
-- There are traffic streams dynamically prepared for each test. The traffic
- is sent and the statistics obtained using trex_stl_lib.api.STLClient.
-
-**Measuring packet loss**
-
-- Create an instance of STLClient
-- Connect to the client
-- Add all streams
-- Clear statistics
-- Send the traffic for defined time
-- Get the statistics
-
-If there is a warm-up phase required, the traffic is sent also before test and
-the statistics are ignored.
-
-**Measuring latency**