docs/report/vpp_performance_tests/csit_release_notes.rst

   1 Release Notes
   2 =============
   3
   4 Changes in |csit-release|
   5 -------------------------
   6
   7 #. VPP PERFORMANCE TESTS
   8
   9    - **Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds**: VPP performance
  10      test data is not included in this report version. This is due to
  11      the lower performance and behaviour inconsistency of these
  12      systems following the upgrade of processor microcode packages
  13      (skx ucode 0x2000064, clx ucode 0x500002c), done as part of
  14      updating Ubuntu 18.04 LTS kernel version. Tested VPP and DPDK
  15      applications (L3fwd) are affected. Skx and Clx test data will be
  16      added in subsequent maintenance report version(s) once the issue
  17      is resolved. See :ref:`vpp_known_issues`.
  18
  19    - **Service density 2n-skx tests**: Added new NF density tests with
  20      IPsec encryption between DUTs.
  21
  22    - **AVF tests**: Full test coveraged based on code changes in CSIT
  23      core layer (driver/interface awareness) and generated by suite
  24      generator (Intel Fortville NICs only).
  25
  26    - **Hoststack TCP/IP tests**: Major refactor of VPP Hoststack TCP/IP
  27      performance tests using WRK generator talking to the VPP HTTP
  28      static server plugin measuring connections per second and
  29      requests per second. Added iperf3 tests with LDPreload.
  30
  31    - **Latency HDRHistogram**: Added High Dynamic Range Histogram
  32      latency measurements based on the new capability in TRex traffic
  33      generator. HDRH latency data presented in latency packet
  34      percentile graphs and in detailed results tables.
  35
  36    - **Mellanox MCX556A-EDAT tests**: Added tests with Mellanox
  37      ConnectX5-2p100GE NICs in 2n-clx testbeds using VPP native rdma
  38      driver.
  39
  40    - **IPsec reconfiguration tests**: Added tests measuring the impact
  41      of IPsec tunnels creations and removals.
  42
  43    - **Load Balancer tests**: Added VPP performance tests for Maglev,
  44      L3DSR (Direct Server Return), Layer 4 Load Balancing NAT Mode.
  45
  46 #. TEST FRAMEWORK
  47
  48    - **CSIT Python3 support**: Full migration of CSIT from Python2.7 to
  49      Python3.6. This change includes library migration, PIP dependency
  50      upgrade, CSIT container images, infrastructure packages
  51      ugrade/installation.
  52
  53    - **CSIT PAPI support**: Finished conversion of CSIT VAT L1 keywords
  54      to PAPI L1 KWs in CSIT using VPP Python bindings (VPP PAPI).
  55      Redesign of key components of PAPI Socket Executor and PAPI
  56      history. Due to issues with PAPI performance, VAT is still used
  57      in CSIT for all VPP scale tests. See known issues below.
  58
  59    - **Test Suite Generator**: Added capability to generate suites for
  60      different drivers per NIC model including DPDK, AVF, RDMA.
  61      Extended coverage for all tests.
  62
  63    - **General Code Housekeeping**: Ongoing RF keywords optimizations,
  64      removal of redundant RF keywords and aligning of suite/test
  65      setup/teardowns.
  66
  67
  68 #. PRESENTATION AND ANALYTICS LAYER
  69
  70    - **Graphs layout improvements**: Improved performance graphs layout
  71      for better readibility and maintenance: test grouping, axis
  72      labels, descriptions, other informative decoration.
  73
  74    - **Latency graphs**: Min/Avg/Max group bar latency graphs are
  75      replaced with packet latency percentile distributon at different
  76      background packet loads based on TRex latency hdrhistogram
  77      measurements.
  78
  79 ..
  80     // Alternative Note for 1st Bullet when bad microcode Skx, Clx results are published
  81     - **Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds**: VPP performance
  82       test data is included in this report version, but it shows lower
  83       performance and behaviour inconsistency of these systems
  84       following the upgrade of processor microcode packages (skx ucode
  85       0x2000064, clx ucode 0x500002c) as part of updating Ubuntu 18.04
  86       LTS kernel version. Tested VPP and DPDK applications (L3fwd) are
  87       affected. Skx and Clx test data will be corrected in subsequent
  88       maintenance report version(s) once the issue is resolved. See
  89       :ref:`vpp_known_issues`.
  90
  91 .. raw:: latex
  92
  93     \clearpage
  94
  95 .. _vpp_known_issues:
  96
  97 Known Issues
  98 ------------
  99
 100 List of known issues in |csit-release| for VPP performance tests:
 101
 102 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 103 | #  | JiraID                                  | Issue Description                                                                                         |
 104 +====+=========================================+===========================================================================================================+
 105 | 1  | `CSIT-570                               | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue.   |
 106 |    | <https://jira.fd.io/browse/CSIT-570>`_  | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds.     |
 107 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 108 | 2  | `VPP-662                                | 9000B packets not supported by NICs VIC1227 and VIC1387.                                                  |
 109 |    | <https://jira.fd.io/browse/VPP-662>`_   |                                                                                                           |
 110 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 111 | 3  | `CSIT-1498                              | Memif tests are sporadically failing on initialization of memif connection.                               |
 112 |    | <https://jira.fd.io/browse/CSIT-1498>`_ |                                                                                                           |
 113 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 114 | 4  | `VPP-1677                               | 9000B ip4 nat44: VPP crash + coredump.                                                                    |
 115 |    | <https://jira.fd.io/browse/VPP-1677>`_  | VPP crashes very often in case that NAT44 is configured and it has to process IP4 jumbo frames (9000B).   |
 116 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 117 | 5  | `CSIT-1591                              | All CSIT scale tests can not use PAPI due to much slower performance compared to VAT/CLI (it takes much   |
 118 |    | <https://jira.fd.io/browse/CSIT-1499>`_ | longer to program VPP). This needs to be addressed on the PAPI side.                                      |
 119 |    +-----------------------------------------+                                                                                                           |
 120 |    | `VPP-1763                               |                                                                                                           |
 121 |    | <https://jira.fd.io/browse/VPP-1763>`_  |                                                                                                           |
 122 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 123 | 6  | `VPP-1675                               | IPv4 IPSEC 9000B packet tests are failing as no packet is forwarded.                                      |
 124 |    | <https://jira.fd.io/browse/VPP-1675>`_  | Reason: chained buffers are not supported.                                                                |
 125 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 126 | 7  | `CSIT-1593                              | IPv4 AVF 9000B packet tests are failing on 3n-skx while passing on 2n-skx.                                |
 127 |    | <https://jira.fd.io/browse/CSIT-1593>`_ |                                                                                                           |
 128 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 129 | 8  | `CSIT-1675                              | Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds behaviour and performance became inconsistent following     |
 130 |    | <https://jira.fd.io/browse/CSIT-1675>`_ | the upgrade to the latest Ubuntu 18.04 LTS kernel version (4.15.0-72-generic) and associated microcode    |
 131 |    |                                         | packages (skx ucode 0x2000064, clx ucode 0x500002c). VPP as well as DPDK L3fwd tests are affected.        |
 132 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 133