Code Review / csit.git / blobdiff
? search:
summary | shortlog | log | commit | commitdiff | review | tree
raw | inline | side by side
rls2001: report update typos in vpp perf rls notes
[csit.git] / docs / report / vpp_performance_tests / csit_release_notes.rst
diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst
index f79e331..64f6f31 100644 (file)
--- a/docs/report/vpp_performance_tests/csit_release_notes.rst
+++ b/docs/report/vpp_performance_tests/csit_release_notes.rst
@@ -6,15 +6,33 @@ Changes in |csit-release|
 
 #. VPP PERFORMANCE TESTS
 
 
 #. VPP PERFORMANCE TESTS
 
-   - **Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds**: VPP performance
-     test data is not included in this report version. This is due to
-     the lower performance and behaviour inconsistency of these
-     systems following the upgrade of processor microcode packages
-     (skx ucode 0x2000064, clx ucode 0x500002c), done as part of
-     updating Ubuntu 18.04 LTS kernel version. Tested VPP and DPDK
-     applications (L3fwd) are affected. Skx and Clx test data will be
-     added in subsequent maintenance report version(s) once the issue
-     is resolved. See :ref:`vpp_known_issues`.
+   - **Intel Xeon 2n-skx, 3n-skx testbeds**: VPP performance test data
+     is now included in this report version. Due to substantial impact
+     of test environment changes (applied during the CSIT-2001
+     development cycle) on the performance of VPP software, a new
+     approach to performance comparison and progression/regression
+     root cause analysis (RCA) has been applied.
+
+     - CSIT test environment is now versioned, with ver. 1 associated
+       with CSIT rls1908 git branch as of 2019-08-21, and ver. 2
+       associated with CSIT rls2001 git branch as of 2020-03-27.
+
+     - To identify performance changes due to CSIT test environment
+       changes from ver. 1 to ver. 2, VPP v19.08.1 has been re-tested in
+       ver. 2 and compared against the past results obtained with
+       testing in ver. 1. Separate RCA1 analysis has been applied to
+       this part. See :ref:`vpp_compare_current_vs_previous_release` and
+       :ref:`vpp_known_issues`.
+
+     - To identify performance changes due to VPP code changes from
+       v19.08.1 to v20.01.0, both have been tested in CSIT environment
+       ver. 2 and compared against each other. Separate RCA2 analysis
+       has been applied to this part. See
+       :ref:`vpp_compare_current_vs_previous_release` and
+       :ref:`vpp_known_issues`.
+
+   - **Intel Xeon 2n-clx testbeds**: VPP performance test data is now
+     included in this report. See :ref:`vpp_known_issues`.
 
    - **Service density 2n-skx tests**: Added new NF density tests with
      IPsec encryption between DUTs.
 
    - **Service density 2n-skx tests**: Added new NF density tests with
      IPsec encryption between DUTs.
@@ -23,17 +41,21 @@ Changes in |csit-release|
      core layer (driver/interface awareness) and generated by suite
      generator (Intel Fortville NICs only).
 
      core layer (driver/interface awareness) and generated by suite
      generator (Intel Fortville NICs only).
 
-   - **Hoststack TCP/IP tests**: Major refactor of VPP Hoststack TCP/IP
+   - **Hoststack tests**: Major refactor of VPP Hoststack TCP/IP
      performance tests using WRK generator talking to the VPP HTTP
      static server plugin measuring connections per second and
      performance tests using WRK generator talking to the VPP HTTP
      static server plugin measuring connections per second and
-     requests per second. Added iperf3 tests with LDPreload.
+     requests per second. Added new iperf3 with LDPreload tests,
+     iperf3/LDPreload tests with packet loss induced via the VPP NSIM
+     (Network Simulator) plugin, and QUIC/UDP/IP transport tests.
+     All of the new tests measure goodput through the VPP Hoststack
+     from client to server.
 
    - **Latency HDRHistogram**: Added High Dynamic Range Histogram
      latency measurements based on the new capability in TRex traffic
      generator. HDRH latency data presented in latency packet
      percentile graphs and in detailed results tables.
 
 
    - **Latency HDRHistogram**: Added High Dynamic Range Histogram
      latency measurements based on the new capability in TRex traffic
      generator. HDRH latency data presented in latency packet
      percentile graphs and in detailed results tables.
 
-   - **Mellanox MCX556A-EDAT tests**: Added tests with Mellanox
+   - **Mellanox CX556A-EDAT tests**: Added tests with Mellanox
      ConnectX5-2p100GE NICs in 2n-clx testbeds using VPP native rdma
      driver.
 
      ConnectX5-2p100GE NICs in 2n-clx testbeds using VPP native rdma
      driver.
 
@@ -43,18 +65,6 @@ Changes in |csit-release|
    - **Load Balancer tests**: Added VPP performance tests for Maglev,
      L3DSR (Direct Server Return), Layer 4 Load Balancing NAT Mode.
 
    - **Load Balancer tests**: Added VPP performance tests for Maglev,
      L3DSR (Direct Server Return), Layer 4 Load Balancing NAT Mode.
 
-..
-    // Alternative Note for 1st Bullet when bad microcode Skx, Clx results are published
-    - **Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds**: VPP performance
-      test data is included in this report version, but it shows lower
-      performance and behaviour inconsistency of these systems
-      following the upgrade of processor microcode packages (skx ucode
-      0x2000064, clx ucode 0x500002c) as part of updating Ubuntu 18.04
-      LTS kernel version. Tested VPP and DPDK applications (L3fwd) are
-      affected. Skx and Clx test data will be corrected in subsequent
-      maintenance report version(s) once the issue is resolved. See
-      :ref:`vpp_known_issues`.
-
 #. TEST FRAMEWORK
 
    - **CSIT Python3 support**: Full migration of CSIT from Python2.7 to
 #. TEST FRAMEWORK
 
    - **CSIT Python3 support**: Full migration of CSIT from Python2.7 to
@@ -76,6 +86,17 @@ Changes in |csit-release|
      removal of redundant RF keywords and aligning of suite/test
      setup/teardowns.
 
      removal of redundant RF keywords and aligning of suite/test
      setup/teardowns.
 
+#. TEST ENVIRONMENT
+
+   - **TRex Fortville NIC Performance**: Received FVL fix from Intel
+     resolving TRex low throughput issue. TRex per FVL NIC throughput
+     increased from ~27 Mpps to the nominal ~37 Mpps. For detail see
+     `CSIT-1503 <https://jira.fd.io/browse/CSIT-1503>`_ and `TRex-519
+     <https://trex-tgn.cisco.com/youtrack/issue/trex-519>`_].
+
+   - **New Intel Xeon Cascadelake Testbeds**: Added performance tests
+     for 2-Node-Cascadelake (2n-clx) testbeds with x710, xxv710 and
+     cx556a-edat NIC cards.
 
 #. PRESENTATION AND ANALYTICS LAYER
 
 
 #. PRESENTATION AND ANALYTICS LAYER
 
@@ -88,6 +109,18 @@ Changes in |csit-release|
      background packet loads based on TRex latency hdrhistogram
      measurements.
 
      background packet loads based on TRex latency hdrhistogram
      measurements.
 
+..
+    // Alternative Note for 1st Bullet when bad microcode Skx, Clx results are published
+    - **Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds**: VPP performance
+      test data is included in this report version, but it shows lower
+      performance and behaviour inconsistency of these systems
+      following the upgrade of processor microcode packages (skx ucode
+      0x2000064, clx ucode 0x500002c) as part of updating Ubuntu 18.04
+      LTS kernel version. Tested VPP and DPDK applications (L3fwd) are
+      affected. Skx and Clx test data will be corrected in subsequent
+      maintenance report version(s) once the issue is resolved. See
+      :ref:`vpp_known_issues`.
+
 .. raw:: latex
 
     \clearpage
 .. raw:: latex
 
     \clearpage
@@ -130,4 +163,39 @@ List of known issues in |csit-release| for VPP performance tests:
 |    | <https://jira.fd.io/browse/CSIT-1675>`_ | the upgrade to the latest Ubuntu 18.04 LTS kernel version (4.15.0-72-generic) and associated microcode    |
 |    |                                         | packages (skx ucode 0x2000064, clx ucode 0x500002c). VPP as well as DPDK L3fwd tests are affected.        |
 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 |    | <https://jira.fd.io/browse/CSIT-1675>`_ | the upgrade to the latest Ubuntu 18.04 LTS kernel version (4.15.0-72-generic) and associated microcode    |
 |    |                                         | packages (skx ucode 0x2000064, clx ucode 0x500002c). VPP as well as DPDK L3fwd tests are affected.        |
 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
-
+| 9  | `CSIT-1679                              | All 2n-clx VPP ip4 tests with xxv710 and avf driver are failing.                                          |
+|    | <https://jira.fd.io/browse/CSIT-1679>`_ |                                                                                                           |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 10 | `CSIT-1680                              | Some 2n-clx cx556a rdma tests are failing.                                                                |
+|    | <https://jira.fd.io/browse/CSIT-1680>`_ |                                                                                                           |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 11 | `CSIT-1699                              | Root Cause Analysis for CSIT-2001. Investigate high stdev of tests with VPP inside VM.                    |
+|    | <https://jira.fd.io/browse/CSIT-1699>`_ |                                                                                                           |
+|    +-----------------------------------------+                                                                                                           |
+|    | `CSIT-1704                              |                                                                                                           |
+|    | <https://jira.fd.io/browse/CSIT-1704>`_ |                                                                                                           |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 12 | `CSIT-1699                              | Root Cause Analysis for CSIT-2001. Identify cause of dot1q-l2xcbase progression.                          |
+|    | <https://jira.fd.io/browse/CSIT-1699>`_ |                                                                                                           |
+|    +-----------------------------------------+                                                                                                           |
+|    | `CSIT-1705                              |                                                                                                           |
+|    | <https://jira.fd.io/browse/CSIT-1705>`_ |                                                                                                           |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 13 | `CSIT-1699                              | Root Cause Analysis for CSIT-2001. Identify cause of avf-ip4scale regression.                             |
+|    | <https://jira.fd.io/browse/CSIT-1699>`_ |                                                                                                           |
+|    +-----------------------------------------+                                                                                                           |
+|    | `CSIT-1706                              |                                                                                                           |
+|    | <https://jira.fd.io/browse/CSIT-1706>`_ |                                                                                                           |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 14 | `CSIT-1699                              | Root Cause Analysis for CSIT-2001. Identify cause of progression in vhost-user tests with testpmd in VM.  |
+|    | <https://jira.fd.io/browse/CSIT-1699>`_ |                                                                                                           |
+|    +-----------------------------------------+                                                                                                           |
+|    | `CSIT-1707                              |                                                                                                           |
+|    | <https://jira.fd.io/browse/CSIT-1707>`_ |                                                                                                           |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| 15 | `CSIT-1699                              | Root Cause Analysis for CSIT-2001. Identify cause for avf-ip4base regression.                             |
+|    | <https://jira.fd.io/browse/CSIT-1699>`_ |                                                                                                           |
+|    +-----------------------------------------+                                                                                                           |
+|    | `CSIT-1708                              |                                                                                                           |
+|    | <https://jira.fd.io/browse/CSIT-1708>`_ |                                                                                                           |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
Integration tests