X-Git-Url: https://gerrit.fd.io/r/gitweb?a=blobdiff_plain;f=docs%2Freport%2Fvpp_performance_tests%2Fcsit_release_notes.rst;h=f79e3316531d5b4b12d16700442b592ab7a80b04;hb=d2b27779627f1f3cd92366d262a29df9a8d3e8be;hp=da25ef6bd5ec332065629c2aaeafc4b5c6663e54;hpb=c5bb42a7c4fb7c7fd8a62947250d91f8e81668f7;p=csit.git diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index da25ef6bd5..f79e331653 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -4,221 +4,130 @@ Release Notes Changes in |csit-release| ------------------------- -#. **VPP performance tests** +#. VPP PERFORMANCE TESTS - - *MRR tests* - Maximum Receive Rate tests measure the packet forwarding rate - under the maximum load offered by traffic generator over a set trial - duration, regardless of packet loss. MRR tests are used for continuous - performance trending and for comparison between releases. + - **Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds**: VPP performance + test data is not included in this report version. This is due to + the lower performance and behaviour inconsistency of these + systems following the upgrade of processor microcode packages + (skx ucode 0x2000064, clx ucode 0x500002c), done as part of + updating Ubuntu 18.04 LTS kernel version. Tested VPP and DPDK + applications (L3fwd) are affected. Skx and Clx test data will be + added in subsequent maintenance report version(s) once the issue + is resolved. See :ref:`vpp_known_issues`. - - *MLR tests* - NDR and PDR tests measure the packet forwarding rate using - MLRsearch library by traffic generator. All tests that previously used - binary search were converted to MLRsearch. + - **Service density 2n-skx tests**: Added new NF density tests with + IPsec encryption between DUTs. - - *L2patch tests* - Tests measure performance of L2patch, cross linking - RX and TX of two physical interfaces. + - **AVF tests**: Full test coveraged based on code changes in CSIT + core layer (driver/interface awareness) and generated by suite + generator (Intel Fortville NICs only). - - *2-node tests* - Baseline set of 2-node tests covering base ip4, ip4, - l2patch, l2bd, l2xc. + - **Hoststack TCP/IP tests**: Major refactor of VPP Hoststack TCP/IP + performance tests using WRK generator talking to the VPP HTTP + static server plugin measuring connections per second and + requests per second. Added iperf3 tests with LDPreload. - - *Generated tests* - Simplified and unified test structure, - semi-autogenerated by generator script. Test generator is currently able - to create test combinations with various frame size and cores combinations. - All existing test cases were converted to new format. + - **Latency HDRHistogram**: Added High Dynamic Range Histogram + latency measurements based on the new capability in TRex traffic + generator. HDRH latency data presented in latency packet + percentile graphs and in detailed results tables. - - *Simultaneous multithreading* - SMT-aware automatic detection and - resource configuration including thread affinity, number of rx queues and - DPDK I/O mbufs. Tests are automatically tagged during execution indicating - configuration. + - **Mellanox MCX556A-EDAT tests**: Added tests with Mellanox + ConnectX5-2p100GE NICs in 2n-clx testbeds using VPP native rdma + driver. - - *Intel Skylake* - Topologies consisting of 2-node and 3-node using - SuperMirco servers each equipped with two Intel Xeon Skylake Platinum - processors. + - **IPsec reconfiguration tests**: Added tests measuring the impact + of IPsec tunnels creations and removals. -#. **Presentation and Analytics Layer** + - **Load Balancer tests**: Added VPP performance tests for Maglev, + L3DSR (Direct Server Return), Layer 4 Load Balancing NAT Mode. - - *Performance trending* - Added continuous performance trending and - analysis. New Performance Trending and Performance Analysis jobs - executed regular throughput tests, with results being subsequently - analysed and trend and anomalies summarized and presented in VPP - Performance Dashboard and trendline graphs. +.. + // Alternative Note for 1st Bullet when bad microcode Skx, Clx results are published + - **Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds**: VPP performance + test data is included in this report version, but it shows lower + performance and behaviour inconsistency of these systems + following the upgrade of processor microcode packages (skx ucode + 0x2000064, clx ucode 0x500002c) as part of updating Ubuntu 18.04 + LTS kernel version. Tested VPP and DPDK applications (L3fwd) are + affected. Skx and Clx test data will be corrected in subsequent + maintenance report version(s) once the issue is resolved. See + :ref:`vpp_known_issues`. -#. **Test Framework Optimizations** +#. TEST FRAMEWORK - - *Performance tests efficiency* - Qemu build/install optimizations, - warmup phase handling, vpp restart handling. Resulted in improved - stability and reduced total execution time by 30% for single pkt - size e.g. 64B/78B. + - **CSIT Python3 support**: Full migration of CSIT from Python2.7 to + Python3.6. This change includes library migration, PIP dependency + upgrade, CSIT container images, infrastructure packages + ugrade/installation. - - *General code housekeeping* - ongoing RF keywords optimizations, - removal of redundant RF keywords. + - **CSIT PAPI support**: Finished conversion of CSIT VAT L1 keywords + to PAPI L1 KWs in CSIT using VPP Python bindings (VPP PAPI). + Redesign of key components of PAPI Socket Executor and PAPI + history. Due to issues with PAPI performance, VAT is still used + in CSIT for all VPP scale tests. See known issues below. -Performance Changes -------------------- + - **Test Suite Generator**: Added capability to generate suites for + different drivers per NIC model including DPDK, AVF, RDMA. + Extended coverage for all tests. -Relative performance changes in measured packet throughput in |csit-release| -are calculated against the results from |csit-release-1| -report. Listed mean and standard deviation values are computed based on -a series of the same tests executed against respective VPP releases to -verify test results repeatability, with percentage change calculated for -mean values. Note that the standard deviation is quite high for a small -number of packet throughput tests, what indicates poor test results -repeatability and makes the relative change of mean throughput value not -fully representative for these tests. The root causes behind poor -results repeatability vary between the test cases. + - **General Code Housekeeping**: Ongoing RF keywords optimizations, + removal of redundant RF keywords and aligning of suite/test + setup/teardowns. -NDR Changes -~~~~~~~~~~~ -NDR throughput changes between releases are available in a -CSV and pretty ASCII formats: +#. PRESENTATION AND ANALYTICS LAYER - - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-ndr.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-ndr.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-ndr.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-ndr.txt>`_. + - **Graphs layout improvements**: Improved performance graphs layout + for better readibility and maintenance: test grouping, axis + labels, descriptions, other informative decoration. -.. note:: + - **Latency graphs**: Min/Avg/Max group bar latency graphs are + replaced with packet latency percentile distributon at different + background packet loads based on TRex latency hdrhistogram + measurements. - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - with Robot Framework result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. +.. raw:: latex -PDR Changes -~~~~~~~~~~~ + \clearpage -PDR throughput changes between releases are available in a -CSV and pretty ASCII formats: - - - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-pdr.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-pdr.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-pdr.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-pdr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - with Robot Framework result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -MRR Changes -~~~~~~~~~~~ - -MRR small packet throughput changes between releases are available in a -CSV and pretty ASCII formats: - - - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-mrr.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-mrr.csv>`_, - - `csv format for 4t4c <../_static/vpp/performance-changes-4t4c-mrr.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-mrr.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-mrr.txt>`_, - - `pretty ASCII format for 4t4c <../_static/vpp/performance-changes-4t4c-mrr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - with Robot Framework result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -Comparison Across Testbeds --------------------------- - -Relative performance changes in measured packet throughputon 3-Node Skx testbed -are calculated against the results measured on 3-Node Hsw testbed. - -NDR Changes -~~~~~~~~~~~ - -NDR changes between testbeds are available in a -CSV and pretty ASCII formats: - - - `csv format for ndr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_, - - `pretty ASCII format for ndr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_ and - `FD.io test executor vpp performance job 3n-skx`_ - with Robot Framework result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -PDR Changes -~~~~~~~~~~~ - -PDR throughput changes between testbeds are available in a -CSV and pretty ASCII formats: - - - `csv format for pdr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_, - - `pretty ASCII format for pdr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_ and - `FD.io test executor vpp performance job 3n-skx`_ - with Robot Framework result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -MRR Changes -~~~~~~~~~~~ - -MRR throughput changes between testbeds are available in a -CSV and pretty ASCII formats: - - - `csv format for mrr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_, - - `pretty ASCII format for mrr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_ and - `FD.io test executor vpp performance job 3n-skx`_ - with Robot Framework result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -Throughput Trending -------------------- - -In addition to reporting throughput changes between VPP releases, CSIT -provides continuous performance trending for VPP master branch: - -#. `VPP Performance Dashboard `_ - - per VPP test case throughput trend, trend compliance and summary of - detected anomalies. - -#. `Trending Methodology `_ - - throughput test metrics, trend calculations and anomaly - classification (progression, regression, outlier). - -#. `Trendline Graphs `_ - - per VPP build MRR throughput measurements against the trendline - with anomaly highlights, with associated CSIT test jobs. +.. _vpp_known_issues: Known Issues ------------ List of known issues in |csit-release| for VPP performance tests: -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| # | Issue | Jira ID | Description | -+===+=================================================+============+=================================================================+ -| 1 | Sporadic (1 in 200) NDR discovery test failures | CSIT-570 | DPDK reporting rx-errors, indicating L1 issue. Suspected issue | -| | on x520. | | with HW combination of X710-X520 in LF testbeds. Not observed | -| | | | outside of LF testbeds. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 2 | High failure rate of api call | VPP-1361 | Failure rate: 30-40% of tests failing due to interfaces not | -| | sw_interface_set_flags [admin-up|link-up] | | in link-up state after API call sw_interface_set_flags. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 3 | Scale IPSecHW Interface mode throughput | CSIT-1234 | IPSec throughput regression: NDR -28%..-31%, PDR -28%..-31%, | -| | regression. | | MRR -40%. Affects IPSec HW Scale 1000tnl tests with Int mode. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ ++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+ +| # | JiraID | Issue Description | ++====+=========================================+===========================================================================================================+ +| 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. | +| | `_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. | ++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+ +| 2 | `VPP-662 | 9000B packets not supported by NICs VIC1227 and VIC1387. | +| | `_ | | ++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+ +| 3 | `CSIT-1498 | Memif tests are sporadically failing on initialization of memif connection. | +| | `_ | | ++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+ +| 4 | `VPP-1677 | 9000B ip4 nat44: VPP crash + coredump. | +| | `_ | VPP crashes very often in case that NAT44 is configured and it has to process IP4 jumbo frames (9000B). | ++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+ +| 5 | `CSIT-1591 | All CSIT scale tests can not use PAPI due to much slower performance compared to VAT/CLI (it takes much | +| | `_ | longer to program VPP). This needs to be addressed on the PAPI side. | +| +-----------------------------------------+ | +| | `VPP-1763 | | +| | `_ | | ++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+ +| 6 | `VPP-1675 | IPv4 IPSEC 9000B packet tests are failing as no packet is forwarded. | +| | `_ | Reason: chained buffers are not supported. | ++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+ +| 7 | `CSIT-1593 | IPv4 AVF 9000B packet tests are failing on 3n-skx while passing on 2n-skx. | +| | `_ | | ++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+ +| 8 | `CSIT-1675 | Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds behaviour and performance became inconsistent following | +| | `_ | the upgrade to the latest Ubuntu 18.04 LTS kernel version (4.15.0-72-generic) and associated microcode | +| | | packages (skx ucode 0x2000064, clx ucode 0x500002c). VPP as well as DPDK L3fwd tests are affected. | ++----+-----------------------------------------+-----------------------------------------------------------------------------------------------------------+ +