- - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now
- configurable trial duration and number of consecutive executions.
- Coverage of MRR tests has been extended across more test
- scenarios. MRR tests are used for continuous performance trending
- and for comparison between VPP releases.
-
- - **MLRsearch Throughput**: MLRsearch algorithm has been introduced
- for all NDR and PDR throughput tests. All tests that previously
- used binary search got converted to MLRsearch. Coverage of NDR/PDR
- tests has been extended across more test scenarios.
-
- - **L2patch Tests**: Tests measure performance of VPP L2patch, the
- fastest L2 forwarding path implemented in VPP, that cross-links
- RX and TX of two physical interfaces.
-
- - **2-Node Tests**: A new baseline set of 2-node tests covering base
- ip4, ip6, l2patch, l2bd, l2xc, running on new Xeon Skylake
- testbeds.
-
- - **Generated tests**: Simplified and unified test structure, semi-
- autogenerated by generator script. Test generator is currently
- able to create test combinations with various frame size and
- cores combinations. All existing test cases were converted to new
- format.
-
- - **Simultaneous Multi-Threading**: SMT-aware detection of server
- processor operation mode (HyperThreading enabled/disabled) with
- associated compute resource configuration including thread
- affinity, number of Rx queues and DPDK I/O mbufs. Tests are
- automatically tagged during execution to indicate executed thread
- configuration.
-
- - **Intel Xeon Skylake Support**: Support for 2-Node and 3-Node
- physical testbed topologies based on the new SuperMirco servers
- each with two Intel Xeon Skylake Platinum processors. Full
- Ansible playbooks refactor for quick server (re)installation and
- reference pointers of configuration.
+ - **Intel Xeon 2n-skx, 3n-skx testbeds**: VPP performance test data
+ is now included in this report version. Due to substantial impact
+ of test environment changes (applied during the CSIT-2001
+ development cycle) on the performance of VPP software, a new
+ approach to performance comparison and progression/regression
+ root cause analysis (RCA) has been applied.
+
+ - CSIT test environment is now versioned, with ver. 1 associated
+ with CSIT rls1908 git branch as of 2019-08-21, and ver. 2
+ associated with CSIT rls2001 git branch as of 2020-03-27.
+
+ - To identify performance changes due to CSIT test environment
+ changes from ver. 1 to ver. 2, VPP v19.08.1 has been re-tested in
+ ver. 2 and compared against the past results obtained with
+ testing in ver. 1. Separate RCA1 analysis has been applied to
+ this part. See :ref:`vpp_compare_current_vs_previous_release` and
+ :ref:`vpp_known_issues`.
+
+ - To identify performance changes due to VPP code changes from
+ v19.08.1 to v20.01.0, both have been tested in CSIT environment
+ ver. 2 and compared against each other. Separate RCA2 analysis
+ has been applied to this part. See
+ :ref:`vpp_compare_current_vs_previous_release` and
+ :ref:`vpp_known_issues`.
+
+ - **Intel Xeon 2n-clx testbeds**: VPP performance test data is now
+ included in this report. See :ref:`vpp_known_issues`.
+
+ - **Service density 2n-skx tests**: Added new NF density tests with
+ IPsec encryption between DUTs.
+
+ - **AVF tests**: Full test coveraged based on code changes in CSIT
+ core layer (driver/interface awareness) and generated by suite
+ generator (Intel Fortville NICs only).
+
+ - **Hoststack tests**: Major refactor of VPP Hoststack TCP/IP
+ performance tests using WRK generator talking to the VPP HTTP
+ static server plugin measuring connections per second and
+ requests per second. Added new iperf3 with LDPreload tests,
+ iperf3/LDPreload tests with packet loss induced via the VPP NSIM
+ (Network Simulator) plugin, and QUIC/UDP/IP transport tests.
+ All of the new tests measure goodput through the VPP Hoststack
+ from client to server.
+
+ - **Latency HDRHistogram**: Added High Dynamic Range Histogram
+ latency measurements based on the new capability in TRex traffic
+ generator. HDRH latency data presented in latency packet
+ percentile graphs and in detailed results tables.
+
+ - **Mellanox CX556A-EDAT tests**: Added tests with Mellanox
+ ConnectX5-2p100GE NICs in 2n-clx testbeds using VPP native rdma
+ driver.
+
+ - **IPsec reconfiguration tests**: Added tests measuring the impact
+ of IPsec tunnels creations and removals.
+
+ - **Load Balancer tests**: Added VPP performance tests for Maglev,
+ L3DSR (Direct Server Return), Layer 4 Load Balancing NAT Mode.
+
+#. TEST FRAMEWORK
+
+ - **CSIT Python3 support**: Full migration of CSIT from Python2.7 to
+ Python3.6. This change includes library migration, PIP dependency
+ upgrade, CSIT container images, infrastructure packages
+ ugrade/installation.
+
+ - **CSIT PAPI support**: Finished conversion of CSIT VAT L1 keywords
+ to PAPI L1 KWs in CSIT using VPP Python bindings (VPP PAPI).
+ Redesign of key components of PAPI Socket Executor and PAPI
+ history. Due to issues with PAPI performance, VAT is still used
+ in CSIT for all VPP scale tests. See known issues below.
+
+ - **Test Suite Generator**: Added capability to generate suites for
+ different drivers per NIC model including DPDK, AVF, RDMA.
+ Extended coverage for all tests.
+
+ - **General Code Housekeeping**: Ongoing RF keywords optimizations,
+ removal of redundant RF keywords and aligning of suite/test
+ setup/teardowns.
+
+#. TEST ENVIRONMENT
+
+ - **TRex Fortville NIC Performance**: Received FVL fix from Intel
+ resolving TRex low throughput issue. TRex per FVL NIC throughput
+ increased from ~27 Mpps to the nominal ~37 Mpps. For detail see
+ `CSIT-1503 <https://jira.fd.io/browse/CSIT-1503>`_ and `TRex-519
+ <https://trex-tgn.cisco.com/youtrack/issue/trex-519>`_].
+
+ - **New Intel Xeon Cascadelake Testbeds**: Added performance tests
+ for 2-Node-Cascadelake (2n-clx) testbeds with x710, xxv710 and
+ cx556a-edat NIC cards.