docs/report/vpp_performance_tests/csit_release_notes.rst

   1 Release Notes
   2 =============
   3
   4 Changes in |csit-release|
   5 -------------------------
   6
   7 #. VPP PERFORMANCE TESTS
   8
   9    - **Intel Xeon 2n-skx, 3n-skx testbeds**: VPP performance test data
  10      is now included in this report version. Due to substantial impact
  11      of test environment changes (applied during the CSIT-2001
  12      development cycle) on the performance of VPP software, a new
  13      approach to performance comparison and progression/regression
  14      root cause analysis (RCA) has been applied.
  15
  16      - CSIT test environment is now versioned, with ver. 1 associated
  17        with CSIT rls1908 git branch as of 2019-08-21, and ver. 2
  18        associated with CSIT rls2001 git branch as of 2020-03-27.
  19
  20      - To identify performance changes due to CSIT test environment
  21        changes from ver. 1 to ver. 2, VPP v19.08.1 has been re-tested in
  22        ver. 2 and compared against the past results obtained with
  23        testing in ver. 1. Separate RCA1 analysis has been applied to
  24        this part. See :ref:`vpp_compare_current_vs_previous_release` and
  25        :ref:`vpp_known_issues`.
  26
  27      - To identify performance changes due to VPP code changes from
  28        v19.08.1 to v20.01.0, both have been tested in CSIT environment
  29        ver. 2 and compared against each other. Separate RCA2 analysis
  30        has been applied to this part. See
  31        :ref:`vpp_compare_current_vs_previous_release` and
  32        :ref:`vpp_known_issues`.
  33
  34    - **Intel Xeon 2n-clx testbeds**: VPP performance test data is now
  35      included in this report. See :ref:`vpp_known_issues`.
  36
  37    - **Service density 2n-skx tests**: Added new NF density tests with
  38      IPsec encryption between DUTs.
  39
  40    - **AVF tests**: Full test coveraged based on code changes in CSIT
  41      core layer (driver/interface awareness) and generated by suite
  42      generator (Intel Fortville NICs only).
  43
  44    - **Hoststack tests**: Major refactor of VPP Hoststack TCP/IP
  45      performance tests using WRK generator talking to the VPP HTTP
  46      static server plugin measuring connections per second and
  47      requests per second. Added new iperf3 with LDPreload tests,
  48      iperf3/LDPreload tests with packet loss induced via the VPP NSIM
  49      (Network Simulator) plugin, and QUIC/UDP/IP transport tests.
  50      All of the new tests measure goodput through the VPP Hoststack
  51      from client to server.
  52
  53    - **Latency HDRHistogram**: Added High Dynamic Range Histogram
  54      latency measurements based on the new capability in TRex traffic
  55      generator. HDRH latency data presented in latency packet
  56      percentile graphs and in detailed results tables.
  57
  58    - **Mellanox CX556A-EDAT tests**: Added tests with Mellanox
  59      ConnectX5-2p100GE NICs in 2n-clx testbeds using VPP native rdma
  60      driver.
  61
  62    - **IPsec reconfiguration tests**: Added tests measuring the impact
  63      of IPsec tunnels creations and removals.
  64
  65    - **Load Balancer tests**: Added VPP performance tests for Maglev,
  66      L3DSR (Direct Server Return), Layer 4 Load Balancing NAT Mode.
  67
  68 #. TEST FRAMEWORK
  69
  70    - **CSIT Python3 support**: Full migration of CSIT from Python2.7 to
  71      Python3.6. This change includes library migration, PIP dependency
  72      upgrade, CSIT container images, infrastructure packages
  73      ugrade/installation.
  74
  75    - **CSIT PAPI support**: Finished conversion of CSIT VAT L1 keywords
  76      to PAPI L1 KWs in CSIT using VPP Python bindings (VPP PAPI).
  77      Redesign of key components of PAPI Socket Executor and PAPI
  78      history. Due to issues with PAPI performance, VAT is still used
  79      in CSIT for all VPP scale tests. See known issues below.
  80
  81    - **Test Suite Generator**: Added capability to generate suites for
  82      different drivers per NIC model including DPDK, AVF, RDMA.
  83      Extended coverage for all tests.
  84
  85    - **General Code Housekeeping**: Ongoing RF keywords optimizations,
  86      removal of redundant RF keywords and aligning of suite/test
  87      setup/teardowns.
  88
  89 #. TEST ENVIRONMENT
  90
  91    - **TRex Fortville NIC Performance**: Received FVL fix from Intel
  92      resolving TRex low throughput issue. TRex per FVL NIC throughput
  93      increased from ~27 Mpps to the nominal ~37 Mpps. For detail see
  94      `CSIT-1503 <https://jira.fd.io/browse/CSIT-1503>`_ and `TRex-519
  95      <https://trex-tgn.cisco.com/youtrack/issue/trex-519>`_].
  96
  97    - **New Intel Xeon Cascadelake Testbeds**: Added performance tests
  98      for 2-Node-Cascadelake (2n-clx) testbeds with x710, xxv710 and
  99      cx556a-edat NIC cards.
 100
 101 #. PRESENTATION AND ANALYTICS LAYER
 102
 103    - **Graphs layout improvements**: Improved performance graphs layout
 104      for better readibility and maintenance: test grouping, axis
 105      labels, descriptions, other informative decoration.
 106
 107    - **Latency graphs**: Min/Avg/Max group bar latency graphs are
 108      replaced with packet latency percentile distributon at different
 109      background packet loads based on TRex latency hdrhistogram
 110      measurements.
 111
 112 ..
 113     // Alternative Note for 1st Bullet when bad microcode Skx, Clx results are published
 114     - **Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds**: VPP performance
 115       test data is included in this report version, but it shows lower
 116       performance and behaviour inconsistency of these systems
 117       following the upgrade of processor microcode packages (skx ucode
 118       0x2000064, clx ucode 0x500002c) as part of updating Ubuntu 18.04
 119       LTS kernel version. Tested VPP and DPDK applications (L3fwd) are
 120       affected. Skx and Clx test data will be corrected in subsequent
 121       maintenance report version(s) once the issue is resolved. See
 122       :ref:`vpp_known_issues`.
 123
 124 .. raw:: latex
 125
 126     \clearpage
 127
 128 .. _vpp_known_issues:
 129
 130 Known Issues
 131 ------------
 132
 133 List of known issues in |csit-release| for VPP performance tests:
 134
 135 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 136 | #  | JiraID                                  | Issue Description                                                                                         |
 137 +====+=========================================+===========================================================================================================+
 138 | 1  | `CSIT-570                               | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue.   |
 139 |    | <https://jira.fd.io/browse/CSIT-570>`_  | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds.     |
 140 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 141 | 2  | `VPP-662                                | 9000B packets not supported by NICs VIC1227 and VIC1387.                                                  |
 142 |    | <https://jira.fd.io/browse/VPP-662>`_   |                                                                                                           |
 143 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 144 | 3  | `CSIT-1498                              | Memif tests are sporadically failing on initialization of memif connection.                               |
 145 |    | <https://jira.fd.io/browse/CSIT-1498>`_ |                                                                                                           |
 146 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 147 | 4  | `VPP-1677                               | 9000B ip4 nat44: VPP crash + coredump.                                                                    |
 148 |    | <https://jira.fd.io/browse/VPP-1677>`_  | VPP crashes very often in case that NAT44 is configured and it has to process IP4 jumbo frames (9000B).   |
 149 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 150 | 5  | `CSIT-1591                              | All CSIT scale tests can not use PAPI due to much slower performance compared to VAT/CLI (it takes much   |
 151 |    | <https://jira.fd.io/browse/CSIT-1499>`_ | longer to program VPP). This needs to be addressed on the PAPI side.                                      |
 152 |    +-----------------------------------------+                                                                                                           |
 153 |    | `VPP-1763                               |                                                                                                           |
 154 |    | <https://jira.fd.io/browse/VPP-1763>`_  |                                                                                                           |
 155 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 156 | 6  | `VPP-1675                               | IPv4 IPSEC 9000B packet tests are failing as no packet is forwarded.                                      |
 157 |    | <https://jira.fd.io/browse/VPP-1675>`_  | Reason: chained buffers are not supported.                                                                |
 158 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 159 | 7  | `CSIT-1593                              | IPv4 AVF 9000B packet tests are failing on 3n-skx while passing on 2n-skx.                                |
 160 |    | <https://jira.fd.io/browse/CSIT-1593>`_ |                                                                                                           |
 161 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 162 | 8  | `CSIT-1675                              | Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds behaviour and performance became inconsistent following     |
 163 |    | <https://jira.fd.io/browse/CSIT-1675>`_ | the upgrade to the latest Ubuntu 18.04 LTS kernel version (4.15.0-72-generic) and associated microcode    |
 164 |    |                                         | packages (skx ucode 0x2000064, clx ucode 0x500002c). VPP as well as DPDK L3fwd tests are affected.        |
 165 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 166 | 9  | `CSIT-1679                              | All 2n-clx VPP ip4 tests with xxv710 and avf driver are failing.                                          |
 167 |    | <https://jira.fd.io/browse/CSIT-1679>`_ |                                                                                                           |
 168 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 169 | 10 | `CSIT-1680                              | Some 2n-clx cx556a rdma tests are failing.                                                                |
 170 |    | <https://jira.fd.io/browse/CSIT-1680>`_ |                                                                                                           |
 171 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 172 | 11 | `CSIT-1699                              | Root Cause Analysis for CSIT-2001. Investigate high stdev of tests with VPP inside VM.                    |
 173 |    | <https://jira.fd.io/browse/CSIT-1699>`_ |                                                                                                           |
 174 |    +-----------------------------------------+                                                                                                           |
 175 |    | `CSIT-1704                              |                                                                                                           |
 176 |    | <https://jira.fd.io/browse/CSIT-1704>`_ |                                                                                                           |
 177 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 178 | 12 | `CSIT-1699                              | Root Cause Analysis for CSIT-2001. Identify cause of dot1q-l2xcbase progression.                          |
 179 |    | <https://jira.fd.io/browse/CSIT-1699>`_ |                                                                                                           |
 180 |    +-----------------------------------------+                                                                                                           |
 181 |    | `CSIT-1705                              |                                                                                                           |
 182 |    | <https://jira.fd.io/browse/CSIT-1705>`_ |                                                                                                           |
 183 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 184 | 13 | `CSIT-1699                              | Root Cause Analysis for CSIT-2001. Identify cause of avf-ip4scale regression.                             |
 185 |    | <https://jira.fd.io/browse/CSIT-1699>`_ |                                                                                                           |
 186 |    +-----------------------------------------+                                                                                                           |
 187 |    | `CSIT-1706                              |                                                                                                           |
 188 |    | <https://jira.fd.io/browse/CSIT-1706>`_ |                                                                                                           |
 189 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 190 | 14 | `CSIT-1699                              | Root Cause Analysis for CSIT-2001. Identify cause of progression in vhost-user tests with testpmd in VM.  |
 191 |    | <https://jira.fd.io/browse/CSIT-1699>`_ |                                                                                                           |
 192 |    +-----------------------------------------+                                                                                                           |
 193 |    | `CSIT-1707                              |                                                                                                           |
 194 |    | <https://jira.fd.io/browse/CSIT-1707>`_ |                                                                                                           |
 195 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 196 | 15 | `CSIT-1699                              | Root Cause Analysis for CSIT-2001. Identify cause for avf-ip4base regression.                             |
 197 |    | <https://jira.fd.io/browse/CSIT-1699>`_ |                                                                                                           |
 198 |    +-----------------------------------------+                                                                                                           |
 199 |    | `CSIT-1708                              |                                                                                                           |
 200 |    | <https://jira.fd.io/browse/CSIT-1708>`_ |                                                                                                           |
 201 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+