X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Freport%2Fvpp_performance_tests%2Fcsit_release_notes.rst;h=50dce6268708ac43bc65e3518636362ccb5833e3;hp=c9df5478bc6def17c73ac00c5836293c97dc9478;hb=4c8b382e0034ed2660d749cbd01d5b304aeb54bc;hpb=908bc54fd36b636382a6c76c351cb85483f80d8e diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index c9df5478bc..50dce62687 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -4,239 +4,78 @@ Release Notes Changes in |csit-release| ------------------------- -#. **VPP Performance Tests** +#. VPP PERFORMANCE TESTS - - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now - configurable trial duration and number of consecutive executions. - Coverage of MRR tests has been extended across more test - scenarios. MRR tests are used for continuous performance trending - and for comparison between VPP releases. + - **Service density 2n-skx tests**: Network Function Virtualization (NFV) + service density tests focus on measuring total per server throughput at + varied NFV service *packing* densities with vswitch providing host + dataplane. The goal is to compare and contrast performance of a shared + vswitch for different network topologies and virtualization technologies, + and their impact on vswitch performance and efficiency in a range of NFV + service configurations. - - **MLRsearch Throughput**: MLRsearch algorithm has been introduced - for all NDR and PDR throughput tests. All tests that previously - used binary search got converted to MLRsearch. Coverage of NDR/PDR - tests has been extended across more test scenarios. + - **Experimental Soak Tests**: Added performamce soak tests framework + code for extended time duration tests and throughput discovery + at given PLR and at give total test time e.g. minutes, hours, + days, weeks, months, years. See updated + :ref:`test_methodology` section for more details. - - **L2patch Tests**: Tests measure performance of VPP L2patch, the - fastest L2 forwarding path implemented in VPP, that cross-links - RX and TX of two physical interfaces. +#. TEST FRAMEWORK - - **2-Node Tests**: A new baseline set of 2-node tests covering base - ip4, ip6, l2patch, l2bd, l2xc, running on new Xeon Skylake - testbeds. + - **Container code optimizations**: Optimized container library allows to + run containre_memif tests faster. - - **Generated tests**: Simplified and unified test structure, semi- - autogenerated by generator script. Test generator is currently - able to create test combinations with various frame size and - cores combinations. All existing test cases were converted to new - format. - - - **Simultaneous Multi-Threading**: SMT-aware detection of server - processor operation mode (HyperThreading enabled/disabled) with - associated compute resource configuration including thread - affinity, number of Rx queues and DPDK I/O mbufs. Tests are - automatically tagged during execution to indicate executed thread - configuration. - - - **Intel Xeon Skylake Support**: Support for 2-Node and 3-Node - physical testbed topologies based on the new SuperMirco servers - each with two Intel Xeon Skylake Platinum processors. Full - Ansible playbooks refactor for quick server (re)installation and - reference pointers of configuration. - -#. **Presentation and Analytics Layer** - - - **Performance trending**: Further improved continuous performance - trending with anomaly detection and analysis. - -#. **Test Framework Optimizations** + - **CSIT PAPI Support**: Continue converting existing VAT L1 keywords to + PAPI L1 KWs in CSIT using VPP Python bindings. Required for migrating away + from VAT. - **General Code Housekeeping**: Ongoing RF keywords optimizations, removal of redundant RF keywords. -Performance Changes -------------------- - -Relative performance changes in measured NDR, PDR and MRR packet -throughput in |csit-release| are calculated against the test results -from |csit-release-1| report, for tests running on 3-Node Intel Xeon -Haswell testbeds (3n-hsw) in 1-core, 2-core and 4-core (MRR only) -configurations. - -Listed mean and standard deviation values are computed based on a series -of the same tests executed against respective VPP releases to verify -test results repeatability, with percentage change calculated for mean -values. Note that the standard deviation is quite high for a small -number of packet throughput tests, what indicates poor test results -repeatability and makes the relative change of mean throughput value not -fully representative for these tests. The root causes behind poor -results repeatability vary between the test cases. - -NDR Changes -~~~~~~~~~~~ - -NDR throughput changes between releases are available in CSV and pretty -ASCII formats: - - - `CSV 1t1c NDR changes <../_static/vpp/performance-changes-1t1c-ndr.csv>`_, - - `CSV 2t2c NDR changes <../_static/vpp/performance-changes-2t2c-ndr.csv>`_, - - `ASCII 1t1c NDR changes <../_static/vpp/performance-changes-1t1c-ndr.txt>`_, - - `ASCII 2t2c NDR changes <../_static/vpp/performance-changes-2t2c-ndr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -PDR Changes -~~~~~~~~~~~ - -PDR throughput changes between releases are available in CSV and pretty -ASCII formats: - - - `CSV 1t1c PDR changes <../_static/vpp/performance-changes-1t1c-pdr.csv>`_, - - `CSV 2t2c PDR changes <../_static/vpp/performance-changes-2t2c-pdr.csv>`_, - - `ASCII 1t1c PDR changes <../_static/vpp/performance-changes-1t1c-pdr.txt>`_, - - `ASCII 2t2c PDR changes <../_static/vpp/performance-changes-2t2c-pdr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -MRR Changes -~~~~~~~~~~~ - -MRR throughput changes between releases are available in CSV and pretty -ASCII formats: - - - `CSV 1t1c MRR changes <../_static/vpp/performance-changes-1t1c-mrr.csv>`_, - - `CSV 2t2c MRR changes <../_static/vpp/performance-changes-2t2c-mrr.csv>`_, - - `CSV 4t4c MRR changes <../_static/vpp/performance-changes-4t4c-mrr.csv>`_, - - `ASCII 1t1c MRR changes <../_static/vpp/performance-changes-1t1c-mrr.txt>`_, - - `ASCII 2t2c MRR changes <../_static/vpp/performance-changes-2t2c-mrr.txt>`_, - - `ASCII 4t4c MRR changes <../_static/vpp/performance-changes-4t4c-mrr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -Skx vs. Hsw Comparison ----------------------- - -Relative performance comparison in measured NDR, PDR and MRR packet -throughput is calculated for tests executed on 3-Node Skylake (3n-skx) -and 3-Node Haswell (3n-hsw) physical testbed types in 1-core -configurations. - -NDR Comparison -~~~~~~~~~~~~~~ - -NDR comparison between testbed types is available in CSV and pretty -ASCII formats: - - - `CSV 1c NDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_, - - `ASCII 1c NDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_ and - `FD.io test executor vpp performance job 3n-skx`_ - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -PDR Comparison -~~~~~~~~~~~~~~ - -PDR comparison between testbed types is available in CSV and pretty -ASCII formats: - - - `CSV 1c PDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_, - - `ASCII 1c PDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_. - -.. note:: - - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_ and - `FD.io test executor vpp performance job 3n-skx`_ - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. - -MRR Comparison -~~~~~~~~~~~~~~ - -MRR comparison between testbed types is available in CSV and pretty -ASCII formats: - - - `CSV 1c MRR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_, - - `ASCII 1c MRR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_. - -.. note:: +#. PRESENTATION AND ANALYTICS LAYER - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_ and - `FD.io test executor vpp performance job 3n-skx`_ - with RF result - files csit-vpp-perf-|srelease|-\*.zip - `archived here <../_static/archive/>`_. + - **Graphs Layout Improvements**: Improved performance graphs layout + for better readibility and maintenance: test grouping, axis + labels, descriptions, other informative decoration. -Throughput Trending -------------------- +#. MISCELLANEOUS -In addition to reporting throughput changes between VPP releases, CSIT -provides continuous performance trending for VPP master branch: + - **3n-dnv Tests (3rd Party)**: Published performance tests for 3n- + dnv (3-Node Atom Denverton) from 3rd party testbeds running FD.io + |csit-release| automated testing code. + Only graphs for Packet Throughput and Speedup Multi-core and not + for Packet Latency were published as there are no results for Packet + Latency available. -#. `VPP Performance Dashboard `_ - - per VPP test case throughput trend, trend compliance and summary of - detected anomalies. +.. raw:: latex -#. `Trending Methodology `_ - - throughput test metrics, trend calculations and anomaly - classification (progression, regression, outlier). + \clearpage -#. `Trendline Graphs `_ - - per VPP build MRR throughput measurements against the trendline - with anomaly highlights, with associated CSIT test jobs. +.. _vpp_known_issues: Known Issues ------------ List of known issues in |csit-release| for VPP performance tests: -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| # | JiraID | Issue Description | -+===+=========================================+=================================================================================================================================+ -| 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. | -| | `_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 2 | `VPP-1361 | High failure rate of api call sw_interface_set_flags [admin-up|link-up]. | -| | `_ | Failure rate: 30-40% of tests failing due to interfaces not in link-up state after API call sw_interface_set_flags. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 3 | `CSIT-1234 | VPP IPSecHW scale interface mode 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | -| | `_ | ip4ipsecscale1000tnl-ip4base-int 1core CSIT-18.07/18.04 relative change: NDR -31%, PDR -32%, MRR -38%. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 4 | `CSIT-1242 | VPP xl710 ip4base test 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | -| | `_ | xl710 ip4base 1core CSIT-18.07/18.04 relative change: NDR -29%, high stdev. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 5 | `CSIT-1243 | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. | -| | `_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 6 | `CSIT-1244 | VPP lispip4 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. | -| | `_ | ip4lispip4-ip4base 2core 3n-skx/3n-hsw relative change: NDR -11%, PDR -18%. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 7 | `CSIT-1245 | VPP srv6proxy-stat and srv6proxy-masq, much higher NDR and PDR 78B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | -| | `_ | Due to wrong test suite configuration in dynamic-proxy mode. Artefact of suite code refactoring. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| # | JiraID | Issue Description | ++====+=========================================+=================================================================================================================================+ +| 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. | +| | `_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 2 | `VPP-1562 | Link bonding (mode LACP, transmit policy l34) test are failing due to VPP crashing producing core dump. | +| | `_ | | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 3 | `CSIT-1427 | Scale HW IPsec are failing due to issue with selecting ipsec backend in VPP. | +| | `_ | | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 4 | `VPP-1563 | AVF L2patch tests are failing for all packet size and core combination. Reason: null-node blackholed packets in show error. | +| | `_ | | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 5 | `CSIT-1234 | VPP IPSecHW/IPSecsW scale interface mode, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-19.01 vs. CSIT-18.10. | +| | `_ | | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 6 | `CSIT-1428 | VM vhost tests with RXQ more than 2 are failing on 3n-skx. Suspecting RXQ configuration issues. | +| | `_ | | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+