X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Freport%2Fvpp_performance_tests%2Fcsit_release_notes.rst;h=580ef4812a538dba552a3d10b841679a9dec5e1a;hp=74e3082f8ea7216099495c40a11bdef3f629ce67;hb=9e00a9702f6b17ff85c597a0915ffb8b693e87ba;hpb=ce1088d88744f2c040801c9852565d522b3feb68 diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index 74e3082f8e..580ef4812a 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -4,36 +4,55 @@ Release Notes Changes in |csit-release| ------------------------- -#. **VPP performance tests** - - - *MRR tests* - New Maximum Receive Rate tests measure the packet - forwarding rate under the maximum load offered by traffic - generator over a set trial duration, regardless of packet loss. - MRR tests are used for continuous performance trending and for - comparison between releases. - - - *Service Chaining with SRv6* - New SRv6 (Segment Routing IPv6) proxy - tests measure performance of SRv6 Endpoint fronting SR-unaware - appliance via masquerading (End.AM), dynamic proxy (End.AD) or - static proxy (End.AS) SR functions. +#. **VPP Performance Tests** + + - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now + configurable trial duration and number of consecutive executions. + Coverage of MRR tests has been extended across more test + scenarios. MRR tests are used for continuous performance trending + and for comparison between VPP releases. + + - **MLRsearch Throughput**: MLRsearch algorithm has been introduced + for all NDR and PDR throughput tests. All tests that previously + used binary search got converted to MLRsearch. Coverage of NDR/PDR + tests has been extended across more test scenarios. + + - **L2patch Tests**: Tests measure performance of VPP L2patch, the + fastest L2 forwarding path implemented in VPP, that cross-links + RX and TX of two physical interfaces. + + - **2-Node Tests**: A new baseline set of 2-node tests covering base + ip4, ip6, l2patch, l2bd, l2xc, running on new Xeon Skylake + testbeds. + + - **Generated tests**: Simplified and unified test structure, semi- + autogenerated by generator script. Test generator is currently + able to create test combinations with various frame size and + cores combinations. All existing test cases were converted to new + format. + + - **Simultaneous Multi-Threading**: SMT-aware detection of server + processor operation mode (HyperThreading enabled/disabled) with + associated compute resource configuration including thread + affinity, number of Rx queues and DPDK I/O mbufs. Tests are + automatically tagged during execution to indicate executed thread + configuration. + + - **Intel Xeon Skylake Support**: Support for 2-Node and 3-Node + physical testbed topologies based on the new SuperMirco servers + each with two Intel Xeon Skylake Platinum processors. Full + Ansible playbooks refactor for quick server (re)installation and + reference pointers of configuration. #. **Presentation and Analytics Layer** - - *Performance trending* - Added continuous performance trending and - analysis. New Performance Trending and Performance Analysis jobs - executed regular throughput tests, with results being subsequently - analysed and trend and anomalies summarized and presented in VPP - Performance Dashboard and trendline graphs. + - **Performance trending**: Further improved continuous performance + trending with anomaly detection and analysis. #. **Test Framework Optimizations** - - *Performance tests efficiency* - Qemu build/install optimizations, - warmup phase handling, vpp restart handling. Resulted in improved - stability and reduced total execution time by 30% for single pkt - size e.g. 64B/78B. - - - *General code housekeeping* - ongoing RF keywords optimizations, - removal of redundant RF keywords. + - **General Code Housekeeping**: Ongoing RF keywords optimizations, + removal of redundant RF keywords. Performance Changes ------------------- @@ -103,9 +122,10 @@ CSV and pretty ASCII formats: .. note:: Test results have been generated by - `FD.io test executor vpp mrr jobs `_ - with Robot Framework result files csit-vpp-perf-mrr-daily-master__*__output.xml.gz - `archived here `_. + `FD.io test executor vpp performance job 3n-hsw`_, + with Robot Framework result + files csit-vpp-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_. Comparison Across Testbeds -------------------------- @@ -149,6 +169,24 @@ CSV and pretty ASCII formats: files csit-vpp-perf-|srelease|-\*.zip `archived here <../_static/archive/>`_. +MRR Changes +~~~~~~~~~~~ + +MRR throughput changes between testbeds are available in a +CSV and pretty ASCII formats: + + - `csv format for mrr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_, + - `pretty ASCII format for mrr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_. + +.. note:: + + Test results have been generated by + `FD.io test executor vpp performance job 3n-hsw`_ and + `FD.io test executor vpp performance job 3n-skx`_ + with Robot Framework result + files csit-vpp-perf-|srelease|-\*.zip + `archived here <../_static/archive/>`_. + Throughput Trending ------------------- @@ -179,18 +217,9 @@ List of known issues in |csit-release| for VPP performance tests: | | on x520. | | with HW combination of X710-X520 in LF testbeds. Not observed | | | | | outside of LF testbeds. | +---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 2 | Lower than expected NDR throughput of DPDK | CSIT-571 | Suspected NIC firmware or DPDK driver issue affecting NDR and | -| | testpmd and VPP L2 path NDR throughput with | | PDR throughput on XL710 and X710 NICs. | -| | xl710 and x710 NICs, compared to x520 NICs. | | | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 3 | Tagged Ethernet dot1q and dot1ad L2 path | CSIT-1066 | Tagged Ethernet dot1q and dot1ad L2 path throughput regression: | -| | throughput regression. | | NDR -2%..-5%, PDR -2%..-6%, MRR. Affects l2xc and l2bd | -| | | | performance tests. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 4 | IPSec (software, no QAT HW) throughput | CSIT-1064 | IPSec throughput regression: NDR -3%..-8%, PDR -2%..-8%, MRR | -| | regression. | | -3%..-7%. Affects IPSec SW tests, QAT HW tests not affected. | +| 2 | High failure rate of api call | VPP-1361 | Failure rate: 30-40% of tests failing due to interfaces not | +| | sw_interface_set_flags [admin-up|link-up] | | in link-up state after API call sw_interface_set_flags. | +---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 5 | High failure rate of creating working container | CSIT-1065 | About 20% of orchestrated container topology tests failing data | -| | topologies with K8s/Ligato orchestration. | | plane verification indicating configuration issue. Suspected | -| | | | issue with Ligato vpp-agent. | +| 3 | Scale IPSecHW Interface mode throughput | CSIT-1234 | IPSec throughput regression: NDR -28%..-31%, PDR -28%..-31%, | +| | regression. | | MRR -40%. Affects IPSec HW Scale 1000tnl tests with Int mode. | +---+-------------------------------------------------+------------+-----------------------------------------------------------------+