docs/report/vpp_performance_tests/csit_release_notes.rst

   1 .. _vpp_performance_tests_release_notes:
   2
   3 Release Notes
   4 =============
   5
   6 Changes in |csit-release|
   7 -------------------------
   8
   9 #. VPP PERFORMANCE TESTS
  10
  11    - **AF_XDP**: Added af_xdp driver support for all test cases. Test
  12      results will be added in sub-sequent CSIT-2106 report.
  13
  14    - **GTPU tunnel**: Added GTPU HW Offload IPv4 routing tests.
  15
  16    - **Intel Xeon Ice Lake**: Added initial test data for these
  17      platforms. Current CSIT-2106 report data for Intel Xeon Ice Lake
  18      comes from an external source (Intel labs running CSIT code on
  19      "8360Y D Stepping" and "6338N" processors). For details about the
  20      physical setup see :ref:`tested_physical_topologies`. Tested
  21      VPP and CSIT versions are pre-release, VPP
  22      21.06-rc0~779-gd640ae52f.
  23
  24    - **MLRsearch improvements**: Added support for multiple packet
  25      throughput rates in a single search, each rate is associated
  26      with a distinct Packet Loss Ratio (PLR) criterion. Previously
  27      only Non Drop Rate (NDR) (PLR=0) and single Partial Drop Rate
  28      (PDR) (PLR<0.5%) were supported. Implemented number of
  29      optimizations improving rate discovery efficiency.
  30
  31    - **Reduction of tests**: Removed obsolete VPP use cases and
  32      superfluous test combinations from continuous and report test
  33      executions, including:
  34
  35      - All vts tests, obsolete use cases.
  36      - dot1q tests apart from dot1q-l2bd, superfluous combinations.
  37      - -100flows, -100kflows in all acl tests.
  38      - nat44 tests
  39
  40        - -pps tests, replaced by -tput tests.
  41        - h1-p1-s1 single session tests, unessential combination.
  42        - h4096-p63-s258048 tests, unessential scale combination.
  43
  44      - ipsec tests
  45
  46        - ethip4ipsectptlispgpe.
  47        - policy-aes128gcm.
  48        - policy-aes128cbc-hmac256sha.
  49        - policy-aes128cbc-hmac512sha.
  50        - int-aes128cbc-hmac256sha.
  51        - scale of
  52
  53          - 400tnlsw.
  54          - 5000tnlsw.
  55          - 20000tnlsw.
  56          - 60000tnlsw.
  57
  58 #. TEST FRAMEWORK
  59
  60    - **Telemetry retouch**: Refactored telemetry retrieval from DUTs and
  61      SUTs. Included VPP perfmon plugin telemetry with all perfmon
  62      bundles available in VPP release.
  63
  64    - **Upgrade to Ubuntu 20.04 LTS**: Re-installed base operating system
  65      to Ubuntu 20.04.2 LTS. Upgrade included also baseline Docker
  66      containers used for spawning topology.
  67
  68    - **TRex upgrade v2.86 to v2.88**: Included move to DPDK 21.02 and
  69      changed the way egress low latency queues are used in FVL NICs.
  70      This broke latency measurements for majority of FVL NICs in
  71      CSIT. Latency values look better after upgrading FVL FW on TRex
  72      servers, but still somewhat higher than before the TRex upgrade.
  73      Tracked by `CSIT-1790 <https://jira.fd.io/browse/CSIT-1790>`_.
  74
  75    - **CSIT test environment** version has been updated to ver. 7, see
  76      :ref:`test_environment_versioning`.
  77
  78    - **CSIT in AWS environment**: Added CSIT support for AWS c5n
  79      instances environment. Test results will be added in sub-sequent
  80      CSIT-2106 report.
  81
  82    - **CSIT PAPI support**: Due to issues with PAPI performance, VAT is
  83      still used in CSIT for all VPP scale tests. See known issues
  84      below.
  85
  86    - **General Code Housekeeping**: Ongoing code optimizations and bug
  87      fixes.
  88
  89 #. PRESENTATION AND ANALYTICS LAYER
  90
  91    - **Graphs improvements**: Updated Packet Latency graphs,
  92      see :ref:`latency_methodology`.
  93
  94 .. raw:: latex
  95
  96     \clearpage
  97
  98 .. _vpp_known_issues:
  99
 100 Known Issues
 101 ------------
 102
 103 List of known issues in |csit-release| for VPP performance tests:
 104
 105 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 106 | #  | JiraID                                  | Issue Description                                                                                         |
 107 +====+=========================================+===========================================================================================================+
 108 |  1 | `CSIT-1782                              | Multicore AVF tests are failing when trying to create interface.                                          |
 109 |    | <https://jira.fd.io/browse/CSIT-1782>`_ |                                                                                                           |
 110 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 111 |  2 | `CSIT-1763                              | Adapt ramp-up phase of nat44 tests for different frame sizes.                                             |
 112 |    | <https://jira.fd.io/browse/CSIT-1763>`_ | Currently ramp-up phase rate and duration values are correctly set for tests with 64B frame size.         |
 113 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 114 |  3 | `CSIT-1671                              | All CSIT scale tests can not use PAPI due to much slower performance compared to VAT/CLI (it takes much   |
 115 |    | <https://jira.fd.io/browse/CSIT-1671>`_ | longer to program VPP). This needs to be addressed on the PAPI side.                                      |
 116 |    +-----------------------------------------+ The usual PAPI library spends too much time parsing arguments, so even with async processing (hundreds of |
 117 |    | `VPP-1763                               | commands in flight over socket), the VPP configuration for large scale tests (millions of messages) takes |
 118 |    | <https://jira.fd.io/browse/VPP-1763>`_  | too long.                                                                                                 |
 119 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 120 |  4 | `CSIT-1789                              | AVF driver does not perform RSS in a deterministic way.                                                   |
 121 |    | <https://jira.fd.io/browse/CSIT-1789>`_ | This increases standard deviation of tests with small number of flows (mainly ipsec).                     |
 122 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 123 |  5 | `CSIT-1790                              | Broken TRex latency measurements with TRex v2.88, DPDK 21.02 and FVL FW 6.01.                             |
 124 |    | <https://jira.fd.io/browse/CSIT-1790>`_ | High latency O(5msec) for all VPP and testpmd/l3fwd test cases for FVL NICs with FW 6.01.                 |
 125 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 126 |  6 | `CSIT-1780                              | IPSEC SW async scheduler MRR tests failing with no traffic forwarded.                                     |
 127 |    | <https://jira.fd.io/browse/CSIT-1780>`_ |                                                                                                           |
 128 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 129 |  7 | `CSIT-1786                              | IP4 and IP6 scale tests failing with no traffic forwarded.                                                |
 130 |    | <https://jira.fd.io/browse/CSIT-1786>`_ | Issue reported to VPP devs.                                                                               |
 131 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 132 |  8 | `CSIT-1785                              | NAT44ED tests failing to establish all TCP sessions.                                                      |
 133 |    | <https://jira.fd.io/browse/CSIT-1785>`_ |                                                                                                           |
 134 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 135 |  9 | `CSIT-1791                              | Performance regression in RDMA tests, due to CSIT environment changes.                                    |
 136 |    | <https://jira.fd.io/browse/CSIT-1791>`_ |                                                                                                           |
 137 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 138
 139 Root Cause Analysis for Performance Changes
 140 -------------------------------------------
 141
 142 List of RCAs in |csit-release| for VPP performance changes:
 143
 144 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
 145 | #  | JiraID                                  | Issue Description                                                                                         |
 146 +====+=========================================+===========================================================================================================+
 147 |  1 | `VPP-1972                               | One VPP change has decreased performance of NAT44ed processing, mostly slow path.                         |
 148 |    | <https://jira.fd.io/browse/VPP-1972>`_  |                                                                                                           |
 149 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+