X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Freport%2Fvpp_performance_tests%2Fcsit_release_notes.rst;h=8f37fed2840aaadce9521f19a4ff0918de3f5f8c;hp=90058d35260b80fdf1db01db7d2e4b5f11fd8036;hb=e0806c8358f38406432d1c6f09de7b90fadb8646;hpb=b37edd789d18a621896b26a42048bd6929742e0a diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index 90058d3526..8f37fed284 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -4,226 +4,88 @@ Release Notes Changes in |csit-release| ------------------------- -#. **VPP performance tests** +#. VPP PERFORMANCE TESTS - - *MRR tests* - Maximum Receive Rate tests measure the packet forwarding rate - under the maximum load offered by traffic generator over a set trial - duration, regardless of packet loss. MRR tests are used for continuous - performance trending and for comparison between releases. + - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now + configurable trial duration and number of consecutive executions. + Coverage of MRR tests has been extended across more test + scenarios. MRR tests are used for continuous performance trending + and for comparison between VPP releases. - - *MLR tests* - NDR and PDR tests measure the packet forwarding rate using - MLRsearch library by traffic generator. All tests that previously used - binary search were converted to MLRsearch. + - **MLRsearch Throughput**: MLRsearch algorithm has been introduced + for all NDR and PDR throughput tests. All tests that previously + used binary search got converted to MLRsearch. Coverage of NDR/PDR + tests has been extended across more test scenarios. - - *L2patch tests* - Tests measure performance of L2patch, cross linking + - **L2patch Tests**: Tests measure performance of VPP L2patch, the + fastest L2 forwarding path implemented in VPP, that cross-links RX and TX of two physical interfaces. - - *2-node tests* - Baseline set of 2-node tests covering base ip4, ip4, - l2patch, l2bd, l2xc. - - - *Generated tests* - Simplified and unified test structure, - semi-autogenerated by generator script. Test generator is currently able - to create test combinations with various frame size and cores combinations. - All existing test cases were converted to new format. - - - *Simultaneous multithreading* - SMT-aware automatic detection and - resource configuration including thread affinity, number of rx queues and - DPDK I/O mbufs. Tests are automatically tagged during execution indicating + - **2-Node Tests**: A new baseline set of 2-node tests covering base + ip4, ip6, l2patch, l2bd, l2xc, running on new Xeon Skylake + testbeds. + + - **Generated tests**: Simplified and unified test structure, semi- + autogenerated by generator script. Test generator is currently + able to create test combinations with various frame size and + cores combinations. All existing test cases were converted to new + format. + + - **Simultaneous Multi-Threading**: SMT-aware detection of server + processor operation mode (HyperThreading enabled/disabled) with + associated compute resource configuration including thread + affinity, number of Rx queues and DPDK I/O mbufs. Tests are + automatically tagged during execution to indicate executed thread configuration. - - *Intel Skylake* - Topologies consisting of 2-node and 3-node using - SuperMirco servers each equipped with two Intel Xeon Skylake Platinum - processors. - -#. **Presentation and Analytics Layer** + - **Intel Xeon Skylake Support**: Support for 2-Node and 3-Node + physical testbed topologies based on the new SuperMirco servers + each with two Intel Xeon Skylake Platinum processors. Full + Ansible playbooks refactor for quick server (re)installation and + reference pointers of configuration. - - *Performance trending* - Added continuous performance trending and - analysis. New Performance Trending and Performance Analysis jobs - executed regular throughput tests, with results being subsequently - analysed and trend and anomalies summarized and presented in VPP - Performance Dashboard and trendline graphs. +#. PRESENTATION AND ANALYTICS LAYER -#. **Test Framework Optimizations** + - **Performance trending**: Further improvements of continuous + performance trending, anomaly detection and analysis. - - *Performance tests efficiency* - Qemu build/install optimizations, - warmup phase handling, vpp restart handling. Resulted in improved - stability and reduced total execution time by 30% for single pkt - size e.g. 64B/78B. +#. TEST FRAMEWORK OPTIMIZATIONS - - *General code housekeeping* - ongoing RF keywords optimizations, + - **General Code Housekeeping**: Ongoing RF keywords optimizations, removal of redundant RF keywords. -Performance Changes -------------------- - -Relative performance changes in measured packet throughput in |csit-release| -are calculated against the results from |csit-release-1| -report. Listed mean and standard deviation values are computed based on -a series of the same tests executed against respective VPP releases to -verify test results repeatability, with percentage change calculated for -mean values. Note that the standard deviation is quite high for a small -number of packet throughput tests, what indicates poor test results -repeatability and makes the relative change of mean throughput value not -fully representative for these tests. The root causes behind poor -results repeatability vary between the test cases. - -NDR Changes -~~~~~~~~~~~ - -NDR throughput changes between releases are available in a -CSV and pretty ASCII formats: - - - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-ndr.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-ndr.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-ndr.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-ndr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - with Robot Framework result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -PDR Changes -~~~~~~~~~~~ - -PDR throughput changes between releases are available in a -CSV and pretty ASCII formats: - - - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-pdr.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-pdr.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-pdr.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-pdr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - with Robot Framework result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -MRR Changes -~~~~~~~~~~~ - -MRR small packet throughput changes between releases are available in a -CSV and pretty ASCII formats: - - - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-mrr.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-mrr.csv>`_, - - `csv format for 4t4c <../_static/vpp/performance-changes-4t4c-mrr.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-mrr.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-mrr.txt>`_, - - `pretty ASCII format for 4t4c <../_static/vpp/performance-changes-4t4c-mrr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - with Robot Framework result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -Comparison Across Testbeds --------------------------- - -Relative performance changes in measured packet throughputon 3-Node Skx testbed -are calculated against the results measured on 3-Node Hsw testbed. - -NDR Changes -~~~~~~~~~~~ - -NDR changes between testbeds are available in a -CSV and pretty ASCII formats: - - - `csv format for ndr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_, - - `pretty ASCII format for ndr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_ and - `FD.io test executor vpp performance job 3n-skx`_ - with Robot Framework result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -PDR Changes -~~~~~~~~~~~ - -PDR throughput changes between testbeds are available in a -CSV and pretty ASCII formats: - - - `csv format for pdr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_, - - `pretty ASCII format for pdr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_ and - `FD.io test executor vpp performance job 3n-skx`_ - with Robot Framework result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -MRR Changes -~~~~~~~~~~~ - -MRR throughput changes between testbeds are available in a -CSV and pretty ASCII formats: - - - `csv format for mrr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_, - - `pretty ASCII format for mrr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_ and - `FD.io test executor vpp performance job 3n-skx`_ - with Robot Framework result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -Throughput Trending -------------------- - -In addition to reporting throughput changes between VPP releases, CSIT -provides continuous performance trending for VPP master branch: - -#. `VPP Performance Dashboard `_ - - per VPP test case throughput trend, trend compliance and summary of - detected anomalies. +.. raw:: latex -#. `Trending Methodology `_ - - throughput test metrics, trend calculations and anomaly - classification (progression, regression, outlier). + \clearpage -#. `Trendline Graphs `_ - - per VPP build MRR throughput measurements against the trendline - with anomaly highlights, with associated CSIT test jobs. +.. _vpp_known_issues: Known Issues ------------ List of known issues in |csit-release| for VPP performance tests: -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| # | Issue | Jira ID | Description | -+===+=================================================+============+=================================================================+ -| 1 | Sporadic (1 in 200) NDR discovery test failures | CSIT-570 | DPDK reporting rx-errors, indicating L1 issue. Suspected issue | -| | on x520. | | with HW combination of X710-X520 in LF testbeds. Not observed | -| | | | outside of LF testbeds. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 2 | Lower than expected NDR throughput of DPDK | CSIT-571 | Suspected NIC firmware or DPDK driver issue affecting NDR and | -| | testpmd and VPP L2 path NDR throughput with | | PDR throughput on XL710 and X710 NICs. | -| | xl710 and x710 NICs, compared to x520 NICs. | | | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 3 | Tagged Ethernet dot1q and dot1ad L2 path | CSIT-1066 | Tagged Ethernet dot1q and dot1ad L2 path throughput regression: | -| | throughput regression. | | NDR -2%..-5%, PDR -2%..-6%, MRR. Affects l2xc and l2bd | -| | | | performance tests. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 4 | IPSec (software, no QAT HW) throughput | CSIT-1064 | IPSec throughput regression: NDR -3%..-8%, PDR -2%..-8%, MRR | -| | regression. | | -3%..-7%. Affects IPSec SW tests, QAT HW tests not affected. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| # | JiraID | Issue Description | ++====+=========================================+=================================================================================================================================+ +| 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. | +| | `_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 2 | `CSIT-1234 | VPP IPSecHW scale interface mode 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | +| | `_ | ip4ipsecscale1000tnl-ip4base-int 1core CSIT-18.07/18.04 relative change: NDR -31%, PDR -32%, MRR -38%. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 3 | `CSIT-1242 | VPP xl710 ip4base test 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | +| | `_ | xl710 ip4base 1core CSIT-18.07/18.04 relative change: NDR -29%, high stdev. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 4 | `CSIT-1243 | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. | +| | `_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 5 | `CSIT-1245 | VPP srv6proxy-stat and srv6proxy-masq, much higher NDR and PDR 78B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | +| | `_ | Due to wrong test suite configuration in dynamic-proxy mode. Artefact of suite code refactoring. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 6 | `CSIT-1246 | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07. | +| | `_ | Past vpp-agent releases are not compatible either. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 7 | `CSIT-1253 | VPP lbdpdk link bonding tests failing due to interfaces not coming up. | +| | `_ | VPP lbdpdk link bonding tests relying on DPDK functionality for bonding fail. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+