Report: GSO release notes
[csit.git] / docs / report / vpp_performance_tests / csit_release_notes.rst
index 5086893..eff392d 100644 (file)
@@ -6,45 +6,52 @@ Changes in |csit-release|
 
 #. VPP PERFORMANCE TESTS
 
-   - **Experimental Soak Tests**: Added performamce soak tests framework
-     code for extended time duration tests and throughput discovery
-     at given PLR and at give total test time e.g. minutes, hours,
-     days, weeks, months, years. See updated
-     :ref:`test_methodology` section for more details.
+   - CSIT test environment is versioned, see
+     :ref:`test_environment_versioning`.
 
-#. TEST FRAMEWORK
+   - **GENEVE tests**: Added VPP performance tests for GENEVE tunnels.
+     See :ref:`geneve_methodology` for more details.
 
-   - **Container code optimizations**: Optimized container library allows to
-     run containre_memif tests faster.
 
-   - **CSIT PAPI Support**: Continue converting existing VAT L1 keywords to
-     PAPI L1 KWs in CSIT using VPP Python bindings. Required for migrating away
-     from VAT.
+   - **GSO tests**: Added VPP performance tests for GSOtap and GSOvirtio.
+     All tested topologies are compared with GSO enabled and disabled.
+     In |csit-release| there is only 1t1c tests running.
+     See :ref:`gso_methodology` for more details.
 
-   - **General Code Housekeeping**: Ongoing RF keywords optimizations,
-     removal of redundant RF keywords.
 
-#. PRESENTATION AND ANALYTICS LAYER
+   - **NAT44 tests**: Added new test type, pure throughput tests.
+     They are similar to PPS tests, but they employ ramp-up trials
+     to ensure all sessions are created (and not timing out)
+     for performance trials.
+
+   - **Jumbo for ipsec**: Test cases with 9000 byte frames are re-enabled
+     in ipsec suites.
 
-   - **Graphs Layout Improvements**: Improved performance graphs layout
-     for better readibility and maintenance: test grouping, axis
-     labels, descriptions, other informative decoration.
+   - **Randomized profiles**: Improved repeatability and cycle length.
+     For details, see :ref:`packet_flow_ordering`.
 
-#. MISCELLANEOUS
+   - **Arm 2n-tx2 testbed**: New physical testbed type installed in
+     FD.io CSIT, with VPP and DPDK performance data added to CSIT
+     trending and this report.
 
-   - **2n-dnv Tests (3rd Party)**: Published performance tests for 2n-
-     dnv (2-Node Atom Denverton) from 3rd party testbeds running FD.io
-     |csit-release| automated testing code.
-     Only graphs for Packet Throughput and Speedup Multi-core and not
-     for Packet Latency were published as there are no results for Packet
-     Latency available.
+   - **Framework speedup**: Shortened overall test job duration
+     by using a different test selection mechanism (using --test
+     instead of --include) and by avoiding unnecessary PAPI reconnects.
+
+#. TEST FRAMEWORK
 
-.. note::
+   - **TRex ASTF**: Improved capability to run TRex in advanced stateful mode.
+
+   - **CSIT PAPI support**: Due to issues with PAPI performance, VAT is
+     still used in CSIT for all VPP scale tests. See known issues below.
+
+   - **General Code Housekeeping**: Ongoing code optimizations,
+     speed ups and bug fixes.
+
+#. PRESENTATION AND ANALYTICS LAYER
 
-    The Report rls1901.1 was generated with only 1 run of performance tests,
-    ie. the data are not statistically significant.
-    The NFV tests were not run so this section has been removed from the
-    report.
+   - **Graphs improvements**: Updated Packet Latency graphs,
+     see :ref:`latency_methodology`.
 
 .. raw:: latex
 
@@ -57,24 +64,40 @@ Known Issues
 
 List of known issues in |csit-release| for VPP performance tests:
 
-+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| #  | JiraID                                  | Issue Description                                                                                                               |
-+====+=========================================+=================================================================================================================================+
-| 1  | `CSIT-570                               | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue.                         |
-|    | <https://jira.fd.io/browse/CSIT-570>`_  | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds.                           |
-+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 2  | `VPP-1563                               | AVF L2patch tests are failing for all packet size and core combination. Reason: null-node blackholed packets in show error.     |
-|    | <https://jira.fd.io/browse/VPP-1563>`_  |                                                                                                                                 |
-+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 3  | `CSIT-1234                              | VPP IPSecHW/IPSecsW scale interface mode, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-19.01 vs. CSIT-18.10.      |
-|    | <https://jira.fd.io/browse/CSIT-1234>`_ |                                                                                                                                 |
-+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 4  | `CSIT-1431                              | AVF 4 cores tests are sporadically failing. Under investigation.                                                                |
-|    | <https://jira.fd.io/browse/CSIT-1431>`_ |                                                                                                                                 |
-+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 5  | `CSIT-1465                              | 4c VPP VM vhost tests failing on 3n-skx. Under investigation.                                                                   |
-|    | <https://jira.fd.io/browse/CSIT-1465>`_ |                                                                                                                                 |
-+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 6  | `CSIT-1466                              | IPSecHW scale tests failing due to VPP reset. Fixed in one of subsequent VPP patches. Confirmed by  running tests with VPP      |
-|    | <https://jira.fd.io/browse/CSIT-1466>`_ | build 19.01.1-8~g50a392f~b56.                                                                                                   |
-+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| #  | JiraID                                  | Issue Description                                                                                         |
++====+=========================================+===========================================================================================================+
+|  1 | `CSIT-570                               | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue.   |
+|    | <https://jira.fd.io/browse/CSIT-570>`_  | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds.     |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+|  2 | `VPP-662                                | 9000B packets not supported by NICs VIC1227 and VIC1387.                                                  |
+|    | <https://jira.fd.io/browse/VPP-662>`_   |                                                                                                           |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+|  3 | `CSIT-1763                              | Adapt ramp-up phase of nat44 tests for different frame sizes.                                             |
+|    | <https://jira.fd.io/browse/CSIT-1763>`_ | Currently ramp-up phase rate and duration values are correctly set for tests with 64B frame size.         |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+|  4 | `CSIT-1671                              | All CSIT scale tests can not use PAPI due to much slower performance compared to VAT/CLI (it takes much   |
+|    | <https://jira.fd.io/browse/CSIT-1671>`_ | longer to program VPP). This needs to be addressed on the PAPI side.                                      |
+|    +-----------------------------------------+ The usual PAPI library spends too much time parsing arguments, so even with async processing (hundreds of |
+|    | `VPP-1763                               | commands in flight over socket), the VPP configuration for large scale tests (millions of messages) takes |
+|    | <https://jira.fd.io/browse/VPP-1763>`_  | too long.                                                                                                 |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+|  5 | `VPP-1934                               | [i40e] Interfaces are not brought up from carrier-down.                                                   |
+|    | <https://jira.fd.io/browse/VPP-1934>`_  | In case of i40e -based interface (e.g Intel x700 series NIC) is bound to kernel driver (i40e) and is in   |
+|    |                                         | state "no-carrier" (<NO-CARRIER,BROADCAST,MULTICAST,UP>) because previously it was disabled via           |
+|    |                                         | "I40E_AQ_PHY_LINK_ENABLED" call, then VPP during initialization of AVF interface is not re-enabling       |
+|    |                                         | interface link via i40e driver to up.                                                                     |
+|    |                                         | CSIT implemented `workaround for AVF interface <https://gerrit.fd.io/r/c/csit/+/29086>`_ until fixed.     |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+
+Root Cause Analysis for Performance Changes
+-------------------------------------------
+
+List of RCAs in |csit-release| for VPP performance changes:
+
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
+| #  | JiraID                                  | Issue Description                                                                                         |
++====+=========================================+===========================================================================================================+
+|  1 | `VPP-1972                               | One VPP change has decreased performance of NAT44ed processing, both slow path and fast path.             |
+|    | <https://jira.fd.io/browse/VPP-1972>`   |                                                                                                           |
++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+