docs/report/vpp_performance_tests/csit_release_notes.rst

   1 .. _vpp_performance_tests_release_notes:
   2
   3 Release Notes
   4 =============
   5
   6 Changes in |csit-release|
   7 -------------------------
   8
   9 #. VPP PERFORMANCE TESTS
  10
  11    - **Intel Xeon Ice Lake**: Added test data for these platforms. Current
  12      CSIT-2110 report data for Intel Xeon Ice Lake comes from an external source
  13      (Intel labs running CSIT code on "8360Y D Stepping" and "6338N"
  14      processors). For details about the physical setup see
  15      :ref:`tested_physical_topologies`.
  16
  17    - **MLRsearch improvements**: Added support for multiple packet
  18      throughput rates in a single search, each rate is associated
  19      with a distinct Packet Loss Ratio (PLR) criterion. Previously
  20      only Non Drop Rate (NDR) (PLR=0) and single Partial Drop Rate
  21      (PDR) (PLR<0.5%) were supported. Implemented number of
  22      optimizations improving rate discovery efficiency.
  23
  24    - **Reduction of tests**: Removed obsolete VPP use cases and
  25      superfluous test combinations from continuous and report test
  26      executions, including:
  27
  28      - All vts tests, obsolete use cases.
  29      - dot1q tests apart from dot1q-l2bd, superfluous combinations.
  30      - -100flows, -100kflows in all acl tests.
  31      - nat44 tests
  32
  33        - -pps tests, replaced by -tput tests.
  34        - h1-p1-s1 single session tests, unessential combination.
  35        - h4096-p63-s258048 tests, unessential scale combination.
  36
  37      - ipsec tests
  38
  39        - ethip4ipsectptlispgpe.
  40        - policy-aes128gcm.
  41        - policy-aes128cbc-hmac256sha.
  42        - policy-aes128cbc-hmac512sha.
  43        - int-aes128cbc-hmac256sha.
  44        - scale of
  45
  46          - 400tnlsw.
  47          - 5000tnlsw.
  48          - 20000tnlsw.
  49          - 60000tnlsw.
  50
  51 #. TEST FRAMEWORK
  52
  53    - **CSIT test environment** version has been updated to ver. 8, see
  54      :ref:`test_environment_versioning`.
  55
  56    - **CSIT in AWS environment**: Added CSIT support for AWS c5n
  57      instances environment.
  58
  59    - **CSIT PAPI support**: Due to issues with PAPI performance, VAT is
  60      still used in CSIT for all VPP scale tests. See known issues
  61      below.
  62
  63    - **General Code Housekeeping**: Ongoing code optimizations and bug
  64      fixes.
  65
  66 #. PRESENTATION AND ANALYTICS LAYER
  67
  68    - **Graphs improvements**: Updated Packet Latency graphs,
  69      see :ref:`latency_methodology`.
  70
  71 .. raw:: latex
  72
  73     \clearpage
  74
  75 .. _vpp_known_issues:
  76
  77 Known Issues
  78 ------------
  79
  80 List of known issues in |csit-release| for VPP performance tests:
  81
  82 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  83 | #  | JiraID                                  | Issue Description                                                                                         |
  84 +====+=========================================+===========================================================================================================+
  85 |  1 | `CSIT-1782                              | Multicore AVF tests are failing when trying to create interface. The issue persists after FVL FW upgrade. |
  86 |    | <https://jira.fd.io/browse/CSIT-1782>`_ |                                                                                                           |
  87 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  88 |  2 | `CSIT-1763                              | Adapt ramp-up phase of nat44 tests for different frame sizes.                                             |
  89 |    | <https://jira.fd.io/browse/CSIT-1763>`_ | Currently ramp-up phase rate and duration values are correctly set for tests with 64B frame size.         |
  90 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  91 |  3 | `CSIT-1671                              | All CSIT scale tests can not use PAPI due to much slower performance compared to VAT/CLI (it takes much   |
  92 |    | <https://jira.fd.io/browse/CSIT-1671>`_ | longer to program VPP). This needs to be addressed on the PAPI side.                                      |
  93 |    +-----------------------------------------+ The usual PAPI library spends too much time parsing arguments, so even with async processing (hundreds of |
  94 |    | `VPP-1763                               | commands in flight over socket), the VPP configuration for large scale tests (millions of messages) takes |
  95 |    | <https://jira.fd.io/browse/VPP-1763>`_  | too long.                                                                                                 |
  96 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  97 |  4 | `CSIT-1789                              | AVF driver does not perform RSS in a deterministic way.                                                   |
  98 |    | <https://jira.fd.io/browse/CSIT-1789>`_ | This increases standard deviation of tests with small number of flows (mainly ipsec).                     |
  99 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 100 |  5 | `CSIT-1780                              | IPSEC SW async scheduler MRR tests failing with no traffic forwarded.                                     |
 101 |    | <https://jira.fd.io/browse/CSIT-1780>`_ |                                                                                                           |
 102 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 103 |  6 | `CSIT-1786                              | IP4 and IP6 scale tests failing with no traffic forwarded.                                                |
 104 |    | <https://jira.fd.io/browse/CSIT-1786>`_ | Issue reported to VPP devs.                                                                               |
 105 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 106 |  7 | `CSIT-1785                              | NAT44ED tests failing to establish all TCP sessions.                                                      |
 107 |    | <https://jira.fd.io/browse/CSIT-1785>`_ |                                                                                                           |
 108 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 109 |  8 | `CSIT-1791                              | Performance regression in RDMA tests, due to CSIT environment changes.                                    |
 110 |    | <https://jira.fd.io/browse/CSIT-1791>`_ |                                                                                                           |
 111 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 112
 113 Root Cause Analysis for Performance Changes
 114 -------------------------------------------
 115
 116 List of RCAs in |csit-release| for VPP performance changes:
 117
 118 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 119 | #  | JiraID                                  | Issue Description                                                                                         |
 120 +====+=========================================+===========================================================================================================+
 121 |  1 | `VPP-1972                               | One VPP change has decreased performance of NAT44ed processing, mostly slow path.                         |
 122 |    | <https://jira.fd.io/browse/VPP-1972>`_  |                                                                                                           |
 123 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+