docs/report/vpp_performance_tests/csit_release_notes.rst

   1 .. _vpp_performance_tests_release_notes:
   2
   3 Release Notes
   4 =============
   5
   6 Changes in |csit-release|
   7 -------------------------
   8
   9 #. VPP PERFORMANCE TESTS
  10
  11    - **Intel Xeon Ice Lake**: Added initial test data for these
  12      platforms. Current CSIT-2106 report data for Intel Xeon Ice Lake
  13      comes from an external source (Intel labs running CSIT code on
  14      "8360Y D Stepping" and "6338N" processors). For details about the
  15      physical setup see :ref:`tested_physical_topologies`. Tested
  16      VPP and CSIT versions are pre-release, VPP
  17      21.06-rc0~779-gd640ae52f.
  18
  19    - **MLRsearch improvements**: Added support for multiple packet
  20      throughput rates in a single search, each rate is associated
  21      with a distinct Packet Loss Ratio (PLR) criterion. Previously
  22      only Non Drop Rate (NDR) (PLR=0) and single Partial Drop Rate
  23      (PDR) (PLR<0.5%) were supported. Implemented number of
  24      optimizations improving rate discovery efficiency.
  25
  26    - **Reduction of tests**: Removed obsolete VPP use cases and
  27      superfluous test combinations from continuous and report test
  28      executions, including:
  29
  30      - All vts tests, obsolete use cases.
  31      - dot1q tests apart from dot1q-l2bd, superfluous combinations.
  32      - -100flows, -100kflows in all acl tests.
  33      - nat44 tests
  34
  35        - -pps tests, replaced by -tput tests.
  36        - h1-p1-s1 single session tests, unessential combination.
  37        - h4096-p63-s258048 tests, unessential scale combination.
  38
  39      - ipsec tests
  40
  41        - ethip4ipsectptlispgpe.
  42        - policy-aes128gcm.
  43        - policy-aes128cbc-hmac256sha.
  44        - policy-aes128cbc-hmac512sha.
  45        - int-aes128cbc-hmac256sha.
  46        - scale of
  47
  48          - 400tnlsw.
  49          - 5000tnlsw.
  50          - 20000tnlsw.
  51          - 60000tnlsw.
  52
  53 #. TEST FRAMEWORK
  54
  55    - **CSIT test environment** version has been updated to ver. 8, see
  56      :ref:`test_environment_versioning`.
  57
  58    - **CSIT in AWS environment**: Added CSIT support for AWS c5n
  59      instances environment.
  60
  61    - **CSIT PAPI support**: Due to issues with PAPI performance, VAT is
  62      still used in CSIT for all VPP scale tests. See known issues
  63      below.
  64
  65    - **General Code Housekeeping**: Ongoing code optimizations and bug
  66      fixes.
  67
  68 #. PRESENTATION AND ANALYTICS LAYER
  69
  70    - **Graphs improvements**: Updated Packet Latency graphs,
  71      see :ref:`latency_methodology`.
  72
  73 .. raw:: latex
  74
  75     \clearpage
  76
  77 .. _vpp_known_issues:
  78
  79 Known Issues
  80 ------------
  81
  82 List of known issues in |csit-release| for VPP performance tests:
  83
  84 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  85 | #  | JiraID                                  | Issue Description                                                                                         |
  86 +====+=========================================+===========================================================================================================+
  87 |  1 | `CSIT-1782                              | Multicore AVF tests are failing when trying to create interface.                                          |
  88 |    | <https://jira.fd.io/browse/CSIT-1782>`_ |                                                                                                           |
  89 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  90 |  2 | `CSIT-1763                              | Adapt ramp-up phase of nat44 tests for different frame sizes.                                             |
  91 |    | <https://jira.fd.io/browse/CSIT-1763>`_ | Currently ramp-up phase rate and duration values are correctly set for tests with 64B frame size.         |
  92 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  93 |  3 | `CSIT-1671                              | All CSIT scale tests can not use PAPI due to much slower performance compared to VAT/CLI (it takes much   |
  94 |    | <https://jira.fd.io/browse/CSIT-1671>`_ | longer to program VPP). This needs to be addressed on the PAPI side.                                      |
  95 |    +-----------------------------------------+ The usual PAPI library spends too much time parsing arguments, so even with async processing (hundreds of |
  96 |    | `VPP-1763                               | commands in flight over socket), the VPP configuration for large scale tests (millions of messages) takes |
  97 |    | <https://jira.fd.io/browse/VPP-1763>`_  | too long.                                                                                                 |
  98 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
  99 |  4 | `CSIT-1789                              | AVF driver does not perform RSS in a deterministic way.                                                   |
 100 |    | <https://jira.fd.io/browse/CSIT-1789>`_ | This increases standard deviation of tests with small number of flows (mainly ipsec).                     |
 101 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 102 |  5 | `CSIT-1790                              | Broken TRex latency measurements with TRex v2.88, DPDK 21.02 and FVL FW 6.01.                             |
 103 |    | <https://jira.fd.io/browse/CSIT-1790>`_ | High latency O(5msec) for all VPP and testpmd/l3fwd test cases for FVL NICs with FW 6.01.                 |
 104 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 105 |  6 | `CSIT-1780                              | IPSEC SW async scheduler MRR tests failing with no traffic forwarded.                                     |
 106 |    | <https://jira.fd.io/browse/CSIT-1780>`_ |                                                                                                           |
 107 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 108 |  7 | `CSIT-1786                              | IP4 and IP6 scale tests failing with no traffic forwarded.                                                |
 109 |    | <https://jira.fd.io/browse/CSIT-1786>`_ | Issue reported to VPP devs.                                                                               |
 110 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 111 |  8 | `CSIT-1785                              | NAT44ED tests failing to establish all TCP sessions.                                                      |
 112 |    | <https://jira.fd.io/browse/CSIT-1785>`_ |                                                                                                           |
 113 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 114 |  9 | `CSIT-1791                              | Performance regression in RDMA tests, due to CSIT environment changes.                                    |
 115 |    | <https://jira.fd.io/browse/CSIT-1791>`_ |                                                                                                           |
 116 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 117
 118 Root Cause Analysis for Performance Changes
 119 -------------------------------------------
 120
 121 List of RCAs in |csit-release| for VPP performance changes:
 122
 123 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 124 | #  | JiraID                                  | Issue Description                                                                                         |
 125 +====+=========================================+===========================================================================================================+
 126 |  1 | `VPP-1972                               | One VPP change has decreased performance of NAT44ed processing, mostly slow path.                         |
 127 |    | <https://jira.fd.io/browse/VPP-1972>`_  |                                                                                                           |
 128 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+