-CSIT Release Notes\r
-==================\r
-\r
-Changes in CSIT |release|\r
--------------------------\r
-\r
-#. Added VPP performance tests\r
-\r
- - **Container Service Chain Topologies Orchestrated by K8s with VPP Memif**\r
-\r
- - Added tests with VPP vswitch in container connecting a number of VPP-\r
- in-container service chain topologies with L2 Cross-Connect and L2\r
- Bridge-Domain configurations, orchestrated by Kubernetes. Added\r
- following forwarding topologies: i) "Parallel" with packets flowing from\r
- NIC via VPP to container and back to VPP and NIC; ii) "Chained" (a.k.a.\r
- "Snake") with packets flowing via VPP to container, back to VPP, to next\r
- container, back to VPP and so on until the last container in a chain,\r
- then back to VPP and NIC; iii) "Horizontal" with packets flowing via VPP\r
- to container, then via "horizontal" memif to next container, and so on\r
- until the last container, then back to VPP and NIC;\r
-\r
- - **VPP TCP/IP stack**\r
-\r
- - Added tests for VPP TCP/IP stack using VPP built-in HTTP server.\r
- WRK traffic generator is used as a client-side;\r
-\r
- - **SRv6**\r
-\r
- - Initial SRv6 (Segment Routing IPv6) tests verifying performance of\r
- IPv6 and SRH (Segment Routing Header) encapsulation, decapsulation,\r
- lookups and rewrites based on configured End and End.DX6 SRv6 egress\r
- functions;\r
-\r
- - **IPSecSW**\r
-\r
- - SW computed IPSec encryption with AES-GCM, CBC-SHA1 ciphers, in\r
- combination with IPv4 routed-forwarding;\r
-\r
-#. Presentation and Analytics Layer\r
-\r
- - Added throughput speedup analysis for multi-core and multi-thread\r
- VPP tests into Presentation and Analytics Layer (PAL) for automated\r
- CSIT test results analysis;\r
-\r
-#. Other changes\r
-\r
- - **Framework optimizations**\r
-\r
- - Ability to run CSIT framework on ARM architecture;\r
-\r
- - Overall stability improvements;\r
-\r
- - **NDR and PDR throughput binary search change**\r
-\r
- - Increased binary search resolution by reducing final step from\r
- 100kpps to 50kpps;\r
-\r
- - **VPP plugin loaded as needed by tests**\r
-\r
- - From this release only plugins required by tests are loaded at\r
- VPP initialization time. Previously all plugins were loaded for\r
- all tests;\r
-\r
-Performance Changes\r
--------------------\r
-\r
-Relative performance changes in measured packet throughput in CSIT\r
-|release| are calculated against the results from CSIT |release-1|\r
-report. Listed mean and standard deviation values are computed based on\r
-a series of the same tests executed against respective VPP releases to\r
-verify test results repeatibility, with percentage change calculated for\r
-mean values. Note that the standard deviation is quite high for a small\r
-number of packet throughput tests, what indicates poor test results\r
-repeatability and makes the relative change of mean throughput value not\r
-fully representative for these tests. The root causes behind poor\r
-results repeatibility vary between the test cases.\r
-\r
-NDR Throughput Changes\r
-~~~~~~~~~~~~~~~~~~~~~~\r
-\r
-NDR small packet throughput changes between releases are available in a CSV and\r
-pretty ASCII formats:\r
-\r
- - `csv format for 1t1c <../_static/vpp/performance-changes-ndr-1t1c-full.csv>`_,\r
- - `csv format for 2t2c <../_static/vpp/performance-changes-ndr-2t2c-full.csv>`_,\r
- - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-ndr-1t1c-full.txt>`_,\r
- - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-ndr-2t2c-full.txt>`_.\r
-\r
-PDR Throughput Changes\r
-~~~~~~~~~~~~~~~~~~~~~~\r
-\r
-NDR small packet throughput changes between releases are available in a CSV and\r
-pretty ASCII formats:\r
-\r
- - `csv format for 1t1c <../_static/vpp/performance-changes-pdr-1t1c-full.csv>`_,\r
- - `csv format for 2t2c <../_static/vpp/performance-changes-pdr-2t2c-full.csv>`_,\r
- - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-pdr-1t1c-full.txt>`_,\r
- - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-pdr-2t2c-full.txt>`_.\r
-\r
-Measured improvements are in line with VPP code optimizations listed in\r
-`VPP-18.01 release notes\r
-<https://docs.fd.io/vpp/18.01/release_notes_1801.html>`_.\r
-\r
-Known Issues\r
-------------\r
-\r
-Here is the list of known issues in CSIT |release| for VPP performance tests:\r
-\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| # | Issue | Jira ID | Description |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 1 | Vic1385 and Vic1227 low performance. | VPP-664 | Low NDR performance. |\r
-| | | | |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 2 | Sporadic (1 in 200) NDR discovery test failures | CSIT-570 | DPDK reporting rx-errors, indicating L1 issue. Suspected issue |\r
-| | on x520. | | with HW combination of X710-X520 in LF testbeds. Not observed |\r
-| | | | outside of LF testbeds. |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 3 | Lower than expected NDR throughput with | CSIT-571 | Suspected NIC firmware or DPDK driver issue affecting NDR and |\r
-| | xl710 and x710 NICs, compared to x520 NICs. | | PDR throughput. Applies to XL710 and X710 NICs. |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 4 | QAT IPSec scale with 1000 tunnels (interfaces) | VPP-1121 | VPP crashes during configuration of 1000 IPsec tunnels. |\r
-| | in 2t2c config, all tests are failing. | | 1t1c tests are not affected |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 5 | rls1801 plugin related performance regression | CSIT-925 | With all plugins loaded NDR, PDR and MaxRates vary |\r
-| | | | intermittently from 3% to 5% across multiple test executions. |\r
-| | | | Requires plugin code bisecting. |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 6 | rls1801 generic small performance regression | CSIT-926 | Generic performance regression of discovered NDR, PDR and |\r
-| | ip4base, l2xcbase, l2bdbase | | MaxRates of -3%..-1% vs. rls1710, affects ip4base, l2xcbase, |\r
-| | | | l2bdbase test suites. Not detected by CSIT performance trending |\r
-| | | | scheme as it was masked out by another issue CSIT-925. |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 7 | rls1801 substantial NDR performance regression | CSIT-927 | Much lower NDR for vhostvr1024 tests, with mean values |\r
-| | for vhost-user vring size of 1024 | | regression of -17%..-42% vs. rls1710, but also very high |\r
-| | | | standard deviation of up to 1.46 Mpps => poor repeatibility. |\r
-| | | | Making mean values not fully representative. |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 8 | rls1801 substantial NDR/PDR regression for | CSIT-928 | NDR regression of -7%..-15%, PDR regression of -3%..-15% |\r
-| | IPSec tunnel scale with HW QAT crypto-dev | | compared to rls1710. |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
+Release Notes
+=============
+
+Changes in |csit-release|
+-------------------------
+
+#. VPP PERFORMANCE TESTS
+
+ - **Intel Xeon 2n-skx, 3n-skx testbeds**: VPP performance test data
+ is now included in this report version. Due to substantial impact
+ of test environment changes (applied during the CSIT-2001
+ development cycle) on the performance of VPP software, a new
+ approach to performance comparison and progression/regression
+ root cause analysis (RCA) has been applied.
+
+ - CSIT test environment is now versioned, with ver. 1 associated
+ with CSIT rls1908 git branch as of 2019-08-21, and ver. 2
+ associated with CSIT rls2001 git branch as of 2020-03-27.
+
+ - To identify performance changes due to CSIT test environment
+ changes from ver. 1 to ver. 2, VPP v19.08.1 has been re-tested in
+ ver. 2 and compared against the past results obtained with
+ testing in ver. 1. Separate RCA1 analysis has been applied to
+ this part. See :ref:`vpp_compare_current_vs_previous_release` and
+ :ref:`vpp_known_issues`.
+
+ - To identify performance changes due to VPP code changes from
+ v19.08.1 to v20.01.0, both have been tested in CSIT environment
+ ver. 2 and compared against each other. Separate RCA2 analysis
+ has been applied to this part. See
+ :ref:`vpp_compare_current_vs_previous_release` and
+ :ref:`vpp_known_issues`.
+
+ - **Intel Xeon 2n-clx testbeds**: VPP performance test data is now
+ included in this report. See :ref:`vpp_known_issues`.
+
+ - **Service density 2n-skx tests**: Added new NF density tests with
+ IPsec encryption between DUTs.
+
+ - **AVF tests**: Full test coveraged based on code changes in CSIT
+ core layer (driver/interface awareness) and generated by suite
+ generator (Intel Fortville NICs only).
+
+ - **Hoststack tests**: Major refactor of VPP Hoststack TCP/IP
+ performance tests using WRK generator talking to the VPP HTTP
+ static server plugin measuring connections per second and
+ requests per second. Added new iperf3 with LDPreload tests,
+ iperf3/LDPreload tests with packet loss induced via the VPP NSIM
+ (Network Simulator) plugin, and QUIC/UDP/IP transport tests.
+ All of the new tests measure goodput through the VPP Hoststack
+ from client to server.
+
+ - **Latency HDRHistogram**: Added High Dynamic Range Histogram
+ latency measurements based on the new capability in TRex traffic
+ generator. HDRH latency data presented in latency packet
+ percentile graphs and in detailed results tables.
+
+ - **Mellanox CX556A-EDAT tests**: Added tests with Mellanox
+ ConnectX5-2p100GE NICs in 2n-clx testbeds using VPP native rdma
+ driver.
+
+ - **IPsec reconfiguration tests**: Added tests measuring the impact
+ of IPsec tunnels creations and removals.
+
+ - **Load Balancer tests**: Added VPP performance tests for Maglev,
+ L3DSR (Direct Server Return), Layer 4 Load Balancing NAT Mode.
+
+#. TEST FRAMEWORK
+
+ - **CSIT Python3 support**: Full migration of CSIT from Python2.7 to
+ Python3.6. This change includes library migration, PIP dependency
+ upgrade, CSIT container images, infrastructure packages
+ ugrade/installation.
+
+ - **CSIT PAPI support**: Finished conversion of CSIT VAT L1 keywords
+ to PAPI L1 KWs in CSIT using VPP Python bindings (VPP PAPI).
+ Redesign of key components of PAPI Socket Executor and PAPI
+ history. Due to issues with PAPI performance, VAT is still used
+ in CSIT for all VPP scale tests. See known issues below.
+
+ - **Test Suite Generator**: Added capability to generate suites for
+ different drivers per NIC model including DPDK, AVF, RDMA.
+ Extended coverage for all tests.
+
+ - **General Code Housekeeping**: Ongoing RF keywords optimizations,
+ removal of redundant RF keywords and aligning of suite/test
+ setup/teardowns.
+
+#. TEST ENVIRONMENT
+
+ - **TRex Fortville NIC Performance**: Received FVL fix from Intel
+ resolving TRex low throughput issue. TRex per FVL NIC throughput
+ increased from ~27 Mpps to the nominal ~37 Mpps. For detail see
+ `CSIT-1503 <https://jira.fd.io/browse/CSIT-1503>`_ and `TRex-519
+ <https://trex-tgn.cisco.com/youtrack/issue/trex-519>`_].
+
+ - **New Intel Xeon Cascadelake Testbeds**: Added performance tests
+ for 2-Node-Cascadelake (2n-clx) testbeds with x710, xxv710 and
+ cx556a-edat NIC cards.
+
+#. PRESENTATION AND ANALYTICS LAYER
+
+ - **Graphs layout improvements**: Improved performance graphs layout
+ for better readibility and maintenance: test grouping, axis
+ labels, descriptions, other informative decoration.
+
+ - **Latency graphs**: Min/Avg/Max group bar latency graphs are
+ replaced with packet latency percentile distributon at different
+ background packet loads based on TRex latency hdrhistogram
+ measurements.
+
+..
+ // Alternative Note for 1st Bullet when bad microcode Skx, Clx results are published
+ - **Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds**: VPP performance
+ test data is included in this report version, but it shows lower
+ performance and behaviour inconsistency of these systems
+ following the upgrade of processor microcode packages (skx ucode
+ 0x2000064, clx ucode 0x500002c) as part of updating Ubuntu 18.04
+ LTS kernel version. Tested VPP and DPDK applications (L3fwd) are
+ affected. Skx and Clx test data will be corrected in subsequent
+ maintenance report version(s) once the issue is resolved. See
+ :ref:`vpp_known_issues`.
+
+.. raw:: latex
+
+ \clearpage
+
+.. _vpp_known_issues:
+
+Known Issues
+------------
+
+List of known issues in |csit-release| for VPP performance tests:
+
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| # | JiraID | Issue Description |
++====+=========================================+===========================================================================================================+
+| 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. |
+| | <https://jira.fd.io/browse/CSIT-570>`_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 2 | `VPP-662 | 9000B packets not supported by NICs VIC1227 and VIC1387. |
+| | <https://jira.fd.io/browse/VPP-662>`_ | |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 3 | `CSIT-1498 | Memif tests are sporadically failing on initialization of memif connection. |
+| | <https://jira.fd.io/browse/CSIT-1498>`_ | |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 4 | `VPP-1677 | 9000B ip4 nat44: VPP crash + coredump. |
+| | <https://jira.fd.io/browse/VPP-1677>`_ | VPP crashes very often in case that NAT44 is configured and it has to process IP4 jumbo frames (9000B). |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 5 | `CSIT-1591 | All CSIT scale tests can not use PAPI due to much slower performance compared to VAT/CLI (it takes much |
+| | <https://jira.fd.io/browse/CSIT-1499>`_ | longer to program VPP). This needs to be addressed on the PAPI side. |
+| +-----------------------------------------+ |
+| | `VPP-1763 | |
+| | <https://jira.fd.io/browse/VPP-1763>`_ | |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 6 | `VPP-1675 | IPv4 IPSEC 9000B packet tests are failing as no packet is forwarded. |
+| | <https://jira.fd.io/browse/VPP-1675>`_ | Reason: chained buffers are not supported. |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 7 | `CSIT-1593 | IPv4 AVF 9000B packet tests are failing on 3n-skx while passing on 2n-skx. |
+| | <https://jira.fd.io/browse/CSIT-1593>`_ | |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 8 | `CSIT-1675 | Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds behaviour and performance became inconsistent following |
+| | <https://jira.fd.io/browse/CSIT-1675>`_ | the upgrade to the latest Ubuntu 18.04 LTS kernel version (4.15.0-72-generic) and associated microcode |
+| | | packages (skx ucode 0x2000064, clx ucode 0x500002c). VPP as well as DPDK L3fwd tests are affected. |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 9 | `CSIT-1679 | All 2n-clx VPP ip4 tests with xxv710 and avf driver are failing. |
+| | <https://jira.fd.io/browse/CSIT-1679>`_ | |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 10 | `CSIT-1680 | Some 2n-clx cx556a rdma tests are failing. |
+| | <https://jira.fd.io/browse/CSIT-1680>`_ | |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 11 | `CSIT-1699 | Root Cause Analysis for CSIT-2001. Investigate high stdev of tests with VPP inside VM. |
+| | <https://jira.fd.io/browse/CSIT-1699>`_ | |
+| +-----------------------------------------+ |
+| | `CSIT-1704 | |
+| | <https://jira.fd.io/browse/CSIT-1704>`_ | |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 12 | `CSIT-1699 | Root Cause Analysis for CSIT-2001. Identify cause of dot1q-l2xcbase progression. |
+| | <https://jira.fd.io/browse/CSIT-1699>`_ | |
+| +-----------------------------------------+ |
+| | `CSIT-1705 | |
+| | <https://jira.fd.io/browse/CSIT-1705>`_ | |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 13 | `CSIT-1699 | Root Cause Analysis for CSIT-2001. Identify cause of avf-ip4scale regression. |
+| | <https://jira.fd.io/browse/CSIT-1699>`_ | |
+| +-----------------------------------------+ |
+| | `CSIT-1706 | |
+| | <https://jira.fd.io/browse/CSIT-1706>`_ | |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 14 | `CSIT-1699 | Root Cause Analysis for CSIT-2001. Identify cause of progression in vhost-user tests with testpmd in VM. |
+| | <https://jira.fd.io/browse/CSIT-1699>`_ | |
+| +-----------------------------------------+ |
+| | `CSIT-1707 | |
+| | <https://jira.fd.io/browse/CSIT-1707>`_ | |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 15 | `CSIT-1699 | Root Cause Analysis for CSIT-2001. Identify cause for avf-ip4base regression. |
+| | <https://jira.fd.io/browse/CSIT-1699>`_ | |
+| +-----------------------------------------+ |
+| | `CSIT-1708 | |
+| | <https://jira.fd.io/browse/CSIT-1708>`_ | |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+