4 Changes in |csit-release|
5 -------------------------
7 #. VPP PERFORMANCE TESTS
9 - **Intel Xeon 2n-skx, 3n-skx testbeds**: VPP performance test data
10 is not included in this report version. This is due to the lower
11 performance and behaviour inconsistency of these systems
12 following the upgrade of processor microcode packages (skx ucode
13 0x2000064), done as part of updating Ubuntu 18.04 LTS kernel
14 version. Tested VPP and DPDK applications (L3fwd) are affected.
15 Skx test data will be added in subsequent maintenance report
16 version(s) once the issue is resolved. See
17 :ref:`vpp_known_issues`.
19 - **Intel Xeon 2n-clx testbeds**: VPP performance test data is now
20 included in this report, after resolving the issue of lower
21 performance and behaviour inconsistency of these systems due to
22 the Linux kernel driven upgrade of processor microcode packages
23 to 0x500002c. The resolution is to use latest SuperMicro BIOS 3.2
24 (for X11DPG-QT motherboards used) that upgrades processor
25 microcode to 0x500002c, AND NOT kernel provided ucode package as
26 it does put system into sub-optimal state. Subset of 2n-clx VPP
27 tests are failing due to clx system behaviour change: i) all ip4
28 tests with xxv710 and avf driver and ii) some cx556a rdma tests.
29 See :ref:`vpp_known_issues`.
31 - **Service density 2n-skx tests**: Added new NF density tests with
32 IPsec encryption between DUTs.
34 - **AVF tests**: Full test coveraged based on code changes in CSIT
35 core layer (driver/interface awareness) and generated by suite
36 generator (Intel Fortville NICs only).
38 - **Hoststack tests**: Major refactor of VPP Hoststack TCP/IP
39 performance tests using WRK generator talking to the VPP HTTP
40 static server plugin measuring connections per second and
41 requests per second. Added new iperf3 with LDPreload tests,
42 iperf3/LDPreload tests with packet loss induced via the VPP NSIM
43 (Network Simulator) plugin, and QUIC/UDP/IP transport tests.
44 All of the new tests measure goodput through the VPP Hoststack
45 from client to server.
47 - **Latency HDRHistogram**: Added High Dynamic Range Histogram
48 latency measurements based on the new capability in TRex traffic
49 generator. HDRH latency data presented in latency packet
50 percentile graphs and in detailed results tables.
52 - **Mellanox CX556A-EDAT tests**: Added tests with Mellanox
53 ConnectX5-2p100GE NICs in 2n-clx testbeds using VPP native rdma
56 - **IPsec reconfiguration tests**: Added tests measuring the impact
57 of IPsec tunnels creations and removals.
59 - **Load Balancer tests**: Added VPP performance tests for Maglev,
60 L3DSR (Direct Server Return), Layer 4 Load Balancing NAT Mode.
64 - **CSIT Python3 support**: Full migration of CSIT from Python2.7 to
65 Python3.6. This change includes library migration, PIP dependency
66 upgrade, CSIT container images, infrastructure packages
69 - **CSIT PAPI support**: Finished conversion of CSIT VAT L1 keywords
70 to PAPI L1 KWs in CSIT using VPP Python bindings (VPP PAPI).
71 Redesign of key components of PAPI Socket Executor and PAPI
72 history. Due to issues with PAPI performance, VAT is still used
73 in CSIT for all VPP scale tests. See known issues below.
75 - **Test Suite Generator**: Added capability to generate suites for
76 different drivers per NIC model including DPDK, AVF, RDMA.
77 Extended coverage for all tests.
79 - **General Code Housekeeping**: Ongoing RF keywords optimizations,
80 removal of redundant RF keywords and aligning of suite/test
85 - **TRex Fortville NIC Performance**: Received FVL fix from Intel
86 resolving TRex low throughput issue. TRex per FVL NIC throughput
87 increased from ~27 Mpps to the nominal ~37 Mpps. For detail see
88 `CSIT-1503 <https://jira.fd.io/browse/CSIT-1503>`_ and `TRex-519
89 <https://trex-tgn.cisco.com/youtrack/issue/trex-519>`_].
91 - **New Intel Xeon Cascadelake Testbeds**: Added performance tests
92 for 2-Node-Cascadelake (2n-clx) testbeds with x710, xxv710 and
93 cx556a-edat NIC cards.
95 #. PRESENTATION AND ANALYTICS LAYER
97 - **Graphs layout improvements**: Improved performance graphs layout
98 for better readibility and maintenance: test grouping, axis
99 labels, descriptions, other informative decoration.
101 - **Latency graphs**: Min/Avg/Max group bar latency graphs are
102 replaced with packet latency percentile distributon at different
103 background packet loads based on TRex latency hdrhistogram
107 // Alternative Note for 1st Bullet when bad microcode Skx, Clx results are published
108 - **Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds**: VPP performance
109 test data is included in this report version, but it shows lower
110 performance and behaviour inconsistency of these systems
111 following the upgrade of processor microcode packages (skx ucode
112 0x2000064, clx ucode 0x500002c) as part of updating Ubuntu 18.04
113 LTS kernel version. Tested VPP and DPDK applications (L3fwd) are
114 affected. Skx and Clx test data will be corrected in subsequent
115 maintenance report version(s) once the issue is resolved. See
116 :ref:`vpp_known_issues`.
122 .. _vpp_known_issues:
127 List of known issues in |csit-release| for VPP performance tests:
129 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
130 | # | JiraID | Issue Description |
131 +====+=========================================+===========================================================================================================+
132 | 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. |
133 | | <https://jira.fd.io/browse/CSIT-570>`_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. |
134 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
135 | 2 | `VPP-662 | 9000B packets not supported by NICs VIC1227 and VIC1387. |
136 | | <https://jira.fd.io/browse/VPP-662>`_ | |
137 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
138 | 3 | `CSIT-1498 | Memif tests are sporadically failing on initialization of memif connection. |
139 | | <https://jira.fd.io/browse/CSIT-1498>`_ | |
140 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
141 | 4 | `VPP-1677 | 9000B ip4 nat44: VPP crash + coredump. |
142 | | <https://jira.fd.io/browse/VPP-1677>`_ | VPP crashes very often in case that NAT44 is configured and it has to process IP4 jumbo frames (9000B). |
143 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
144 | 5 | `CSIT-1591 | All CSIT scale tests can not use PAPI due to much slower performance compared to VAT/CLI (it takes much |
145 | | <https://jira.fd.io/browse/CSIT-1499>`_ | longer to program VPP). This needs to be addressed on the PAPI side. |
146 | +-----------------------------------------+ |
148 | | <https://jira.fd.io/browse/VPP-1763>`_ | |
149 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
150 | 6 | `VPP-1675 | IPv4 IPSEC 9000B packet tests are failing as no packet is forwarded. |
151 | | <https://jira.fd.io/browse/VPP-1675>`_ | Reason: chained buffers are not supported. |
152 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
153 | 7 | `CSIT-1593 | IPv4 AVF 9000B packet tests are failing on 3n-skx while passing on 2n-skx. |
154 | | <https://jira.fd.io/browse/CSIT-1593>`_ | |
155 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
156 | 8 | `CSIT-1675 | Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds behaviour and performance became inconsistent following |
157 | | <https://jira.fd.io/browse/CSIT-1675>`_ | the upgrade to the latest Ubuntu 18.04 LTS kernel version (4.15.0-72-generic) and associated microcode |
158 | | | packages (skx ucode 0x2000064, clx ucode 0x500002c). VPP as well as DPDK L3fwd tests are affected. |
159 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
160 | 9 | `CSIT-1679 | All 2n-clx VPP ip4 tests with xxv710 and avf driver are failing. |
161 | | <https://jira.fd.io/browse/CSIT-1679>`_ | |
162 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+
163 | 10 | `CSIT-1680 | Some 2n-clx cx556a rdma tests are failing. |
164 | | <https://jira.fd.io/browse/CSIT-1680>`_ | |
165 +----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+