docs/report/vpp_performance_tests/csit_release_notes.rst

   1 Release Notes
   2 =============
   3
   4 Changes in |csit-release|
   5 -------------------------
   6
   7 #. VPP PERFORMANCE TESTS
   8
   9    - CSIT test environment is versioned, see
  10      :ref:`test_environment_versioning`.
  11
  12    - To identify performance changes due to VPP code changes from
  13      v20.05.0 to v20.09.0, both have been tested in CSIT environment
  14      ver. 5 and compared against each other. All substantial
  15      progressions has been marked up with RCA analysis. See
  16      :ref:`vpp_compare_current_vs_previous_release` and
  17      :ref:`vpp_known_issues`.
  18
  19    - **NAT44 tests**: Adapted existing and added new tests.
  20
  21      - Refactored NAT44 deterministic mode (nat44det) tests to use separate
  22        det44 vpp plugin and to use the same scheme of inside and outside
  23        addresses and ports, as used in new NAT44 endpoint-dependent mode tests.
  24
  25      - Added new NAT44 endpoint-depended mode uni-directional (nat44ed-udir)
  26        tests that measure packet throughput in one direction with usage of TRex
  27        in stateless mode.
  28
  29      - Added new NAT44 endpoint-dependent mode CPS tests that measure
  30        connections per second with usage of TRex in stateful mode.
  31        NOTE: CPS measurement data will be added in two weeks.
  32
  33    - **IPSec async mode tests**: Added VPP performance tests for async crypto
  34      engine.
  35
  36    - **AMD 2n-zn2 testbed**: New physical testbed type installed in FD.io CSIT,
  37      VPP performance data will be added in upcoming weeks.
  38
  39 #. TEST FRAMEWORK
  40
  41    - **TRex ASTF**: Added capability to run TRex in advanced stateful mode.
  42
  43    - **CSIT PAPI support**: Due to issues with PAPI performance, VAT is
  44      still used in CSIT for all VPP scale tests. See known issues below.
  45
  46    - **General Code Housekeeping**: Ongoing RF keywords optimizations,
  47      removal of redundant RF keywords and aligning of suite/test
  48      setup/teardowns.
  49
  50    - **Intel E810CQ 100G NIC**: Added configuration for Intel E810CQ 100G NIC.
  51      No tests run for this NIC as it is not present in FD.io CSIT lab yet.
  52
  53 #. PRESENTATION AND ANALYTICS LAYER
  54
  55    - **Graphs improvements**: Added possibility to use Gbps on Y-axis in
  56      Packet Throughput and Speedup Multi-Core graphs, added unidirectional
  57      mode to the Latency graphs.
  58
  59 .. raw:: latex
  60
  61     \clearpage
  62
  63 .. _vpp_known_issues:
  64
  65 Known Issues
  66 ------------
  67
  68 List of known issues in |csit-release| for VPP performance tests: TODO: check
  69
  70 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  71 | #  | JiraID                                  | Issue Description                                                                                         |
  72 +====+=========================================+===========================================================================================================+
  73 |  1 | `CSIT-570                               | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue.   |
  74 |    | <https://jira.fd.io/browse/CSIT-570>`_  | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds.     |
  75 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  76 |  2 | `VPP-662                                | 9000B packets not supported by NICs VIC1227 and VIC1387.                                                  |
  77 |    | <https://jira.fd.io/browse/VPP-662>`_   |                                                                                                           |
  78 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  79 |  3 | `CSIT-1763                              | Adapt ramp-up phase of nat44 tests for different frame sizes.                                             |
  80 |    | <https://jira.fd.io/browse/CSIT-1763>`_ | Currently ramp-up phase rate and duration values are correctly set for tests with 64B frame size.         |
  81 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  82 |  4 | `CSIT-1671                              | All CSIT scale tests can not use PAPI due to much slower performance compared to VAT/CLI (it takes much   |
  83 |    | <https://jira.fd.io/browse/CSIT-1671>`_ | longer to program VPP). This needs to be addressed on the PAPI side.                                      |
  84 |    +-----------------------------------------+ The usual PAPI library spends too much time parsing arguments, so even with async processing (hundreds of |
  85 |    | `VPP-1763                               | commands in flight over socket), the VPP configuration for large scale tests (millions of messages) takes |
  86 |    | <https://jira.fd.io/browse/VPP-1763>`_  | too long.                                                                                                 |
  87 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  88 |  5 | `VPP-1675                               | IPv4 IPSEC 9000B packet tests are failing as no packet is forwarded.                                      |
  89 |    | <https://jira.fd.io/browse/VPP-1675>`_  | Reason: chained buffers are not supported.                                                                |
  90 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  91 |  6 | `VPP-1934                               | [i40e] Interfaces are not brought up from carrier-down.                                                   |
  92 |    | <https://jira.fd.io/browse/VPP-1934 >`_ | In case of i40e -based interface (e.g Intel x700 series NIC) is bound to kernel driver (i40e) and is in   |
  93 |    |                                         | state "no-carrier" (<NO-CARRIER,BROADCAST,MULTICAST,UP>) because previously it was disabled via           |
  94 |    |                                         | "I40E_AQ_PHY_LINK_ENABLED" call, then VPP during initialization of AVF interface is not re-enabling       |
  95 |    |                                         | interface link via i40e driver to up.                                                                     |
  96 |    |                                         | CSIT implemented `workaround for AVF interface <https://gerrit.fd.io/r/c/csit/+/29086>`_ until fixed.     |
  97 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  98 |  7 | `CSIT-1760                              | All Mellanox / rdma driver tests are failing on LF testbed28 while successfully run on other LF testbeds. |
  99 |    | <https://jira.fd.io/browse/CSIT-1760>`_ |                                                                                                           |
 100 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 101
 102 Root Cause Analysis for Performance Changes
 103 -------------------------------------------
 104
 105 List of RCAs in |csit-release| for VPP performance changes:
 106
 107 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 108 | #  | JiraID                                  | Issue Description                                                                                         |
 109 +====+=========================================+===========================================================================================================+
 110 |  1 |                                         |                                                                                                           |
 111 |    |                                         |                                                                                                           |
 112 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+