docs/report/vpp_performance_tests/csit_release_notes.rst

   1 Release Notes
   2 =============
   3
   4 Changes in |csit-release|
   5 -------------------------
   6
   7 #. VPP PERFORMANCE TESTS
   8
   9    - **Intel Xeon 2n-skx, 3n-skx testbeds**: VPP performance test data
  10      is provided using a different CSIT test environment compared to
  11      CSIT-1908.1 and CSIT-1908. The changes were applied during the
  12      CSIT-2001 development cycle.
  13
  14      - CSIT test environment is now versioned, with ver. 1 associated
  15        with CSIT rls1908 git branch as of 2019-08-21, and ver. 2
  16        associated with CSIT rls2001 git branch as of 2020-03-27.
  17
  18      - To identify performance changes due to VPP code changes from
  19        v19.08.1 to v19.08.2, both have been tested in CSIT
  20        environment ver. 2. See
  21        :ref:`vpp_compare_current_vs_previous_release` and
  22        :ref:`vpp_known_issues`.
  23
  24    - **Intel Xeon 2n-clx testbeds**: VPP performance test data is now
  25      included in this report. See :ref:`vpp_known_issues`.
  26
  27    - **Service density 2n-skx tests**: Added higher NF density tests with
  28      802.1q (vlan) and VXLAN encapsulation from Traffic Generator.
  29
  30    - **GBP tests**: Added GBP (Group Based Policy) routing test cases
  31      with 802.1q (vlan) external traffic.
  32
  33    - **AVF IPv4 scale tests**: Increased coverage of AVF IPv4 base and
  34      scale test cases (Fortville NICs only).
  35
  36    - **2n-skx tests**: Increased coverage of selected (COP, iACL,
  37      Policer) test cases.
  38
  39    - **IPsec scale tests**: Added IPsec interface mode scale tests with
  40      1, 40, 400, 1000, 5000, 10000, 20000, 40000, 60000 IPsec tunnels.
  41      Removed DPDK backend dependency. Major IPsec test code
  42      refactoring.
  43
  44    - **Hoststack TCP/IP tests**: Major refactor of Hoststack TCP
  45      performance tests using WRK generator talking to the VPP HTTP
  46      static server plugin measuring connections per second and
  47      requests per second.
  48
  49    - **Changed methodology of dot1q tests in 2-Node testbeds**: dot1q
  50      encapsulation is now used on both links of SUT. Previously dot1q
  51      was used only on a single link with the other link carrying
  52      untagged Ethernet frames. This change results in slightly lower
  53      throughput in CSIT-1908 for all dot1q tests in all 2-Node
  54      testbeds.
  55
  56    - **KVM VM vhost-user tests**: completed move to Kernel-VM for all
  57      tests. In addition to running DPDK Testpmd as VM workload, new
  58      tests created with VPP as VM workload. VPP in VM is the same
  59      version as the DUT VPP (acting as vSwitch) and its configuration
  60      depends on the test type. For all L2 Ethernet Switching tests
  61      it's vpp-l2xc (L2 cross-connect), for all IPv4 Routing tests it's
  62      vpp-ip4 (VPP IPv4 routing).
  63
  64 #. TEST FRAMEWORK
  65
  66    - **CSIT PAPI Support**: Finished conversion of CSIT VAT L1 keywords
  67      to PAPI L1 KWs in CSIT using VPP Python bindings (VPP PAPI).
  68      Redesign of key components of PAPI Socket Executor and PAPI
  69      history. Due to issues with PAPI performance, VAT is still used
  70      in CSIT for all VPP scale tests. See known issues below.
  71
  72    - **General Code Housekeeping**: Ongoing RF keywords optimizations,
  73      removal of redundant RF keywords and aligning of suite/test
  74      setup/teardowns.
  75
  76
  77 #. PRESENTATION AND ANALYTICS LAYER
  78
  79    - **Graphs Layout Improvements**: Improved performance graphs layout
  80      for better readibility and maintenance: test grouping, axis
  81      labels, descriptions, other informative decoration.
  82
  83 .. raw:: latex
  84
  85     \clearpage
  86
  87 .. _vpp_known_issues:
  88
  89 Known Issues
  90 ------------
  91
  92 List of known issues in |csit-release| for VPP performance tests:
  93
  94 +----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
  95 | #  | JiraID                                  | Issue Description                                                                                        |
  96 +====+=========================================+==========================================================================================================+
  97 | 1  | `CSIT-570                               | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue.  |
  98 |    | <https://jira.fd.io/browse/CSIT-570>`_  | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds.    |
  99 +----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
 100 | 2  | `VPP-662                                | 9000B packets not supported by NICs VIC1227 and VIC1387.                                                 |
 101 |    | <https://jira.fd.io/browse/VPP-662>`_   |                                                                                                          |
 102 +----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
 103 | 3  | `CSIT-1498                              | Memif tests are sporadically failing on initialization of memif connection.                              |
 104 |    | <https://jira.fd.io/browse/CSIT-1498>`_ |                                                                                                          |
 105 +----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
 106 | 4  | `VPP-1676                               | 9000B ip4 memif errors - ip4-input: ip4 length > l2 length.                                              |
 107 |    | <https://jira.fd.io/browse/VPP-1676>`_  | IP4 jumbo frames (9000B) are dropped in case of tests with memif.                                        |
 108 +----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
 109 | 5  | `VPP-1677                               | 9000B ip4 nat44: VPP crash + coredump.                                                                   |
 110 |    | <https://jira.fd.io/browse/VPP-1677>`_  | VPP crashes very often in case that NAT44 is configured and it has to process IP4 jumbo frames (9000B).  |
 111 +----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
 112 | 6  | `CSIT-1591                              | All CSIT scale tests can not use PAPI due to much slower performance compared to VAT/CLI (it takes much  |
 113 |    | <https://jira.fd.io/browse/CSIT-1591>`_ | longer to program VPP). This needs to be addressed on the PAPI side.                                     |
 114 |    +-----------------------------------------+                                                                                                          |
 115 |    | `VPP-1763                               |                                                                                                          |
 116 |    | <https://jira.fd.io/browse/VPP-1763>`_  |                                                                                                          |
 117 +----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
 118 | 7  | `CSIT-1593                              | IPv4 AVF 9000B packet tests are failing on 3n-skx while passing on 2n-skx.                               |
 119 |    | <https://jira.fd.io/browse/CSIT-1593>`_ |                                                                                                          |
 120 +----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+