X-Git-Url: https://gerrit.fd.io/r/gitweb?a=blobdiff_plain;f=docs%2Freport%2Fvpp_performance_tests%2Fcsit_release_notes.rst;h=e60e50bcf95fbc02b74c2e3f84e91a1d3ba7c813;hb=9d324eab35857dac39015da42ce4e7cd3554ffaf;hp=14ca8c566103a57ce5fc66e3f528087aaca0e738;hpb=bb32c95f445c8481b991b0b3d48e1fbfa352f3c7;p=csit.git diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index 14ca8c5661..e60e50bcf9 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -1,89 +1,60 @@ Release Notes ============= -Changes in |csit-release| -------------------------- +.. + Changes in |csit-release| + ------------------------- -#. VPP PERFORMANCE TESTS + #. VPP PERFORMANCE TESTS - - **BMRR Throughput**: MRR (Maximum Receive Rate) test code has been - updated with a configurable burst MRR parameters: trial duration - and number of trials in a single burst. Enables a new Burst MRR - (BMRR for short) methodology for more precise performance - trending. See updated :ref:`performance_test_methodology` section - for more details. + - **Service density 2n-skx tests**: Network Function Virtualization (NFV) + service density tests focus on measuring total per server throughput at + varied NFV service *packing* densities with vswitch providing host + dataplane. The goal is to compare and contrast performance of a shared + vswitch for different network topologies and virtualization technologies, + and their impact on vswitch performance and efficiency in a range of NFV + service configurations. - - **2n-skx tests**: Added performamce tests for 2n-skx (2-Node Xeon - Skylake) testbeds: focus on baseline and scale tests, including - VM vhost and Container memif tests. + - **Experimental Soak Tests**: Added performamce soak tests framework + code for extended time duration tests and throughput discovery + at given PLR and at give total test time e.g. minutes, hours, + days, weeks, months, years. See updated + :ref:`test_methodology` section for more details. - - **3n-skx tests**: Added performamce tests for 3n-skx (3-Node Xeon - Skylake) testbeds: VM vhost and Container memif tests. + #. TEST FRAMEWORK - - **VXLAN Scale Tests**: Added performamce tests for VXLAN scale with - dot1q and VPP L2BD. + - **Container code optimizations**: Optimized container library allows to + run containre_memif tests faster. - - **AVF Driver Tests**: Added performamce tests for i40e AVF driver - on VPP, no DPDK required. + - **CSIT PAPI Support**: Continue converting existing VAT L1 keywords to + PAPI L1 KWs in CSIT using VPP Python bindings. Required for migrating away + from VAT. - - **QAT**: Fixed reoccuring issues with QAT crypto accelerator cards. + - **General Code Housekeeping**: Ongoing RF keywords optimizations, + removal of redundant RF keywords. - - **VM Vhost Virtio Params Combinations**: Added performance tests - for VM vhost with different virtio parameters combinations: - indirect buffers, mergeable buffers. + #. PRESENTATION AND ANALYTICS LAYER - - **K8s/Ligato in Trending**: Added K8s/Ligato Container memif tests - to daily trending. + - **Graphs Layout Improvements**: Improved performance graphs layout + for better readibility and maintenance: test grouping, axis + labels, descriptions, other informative decoration. -#. TEST FRAMEWORK + #. MISCELLANEOUS - - **Experimental Soak Tests**: Added performamce soak tests framework - code for extended time duration tests and throughput discovery - at given PLR and at give total test time e.g. minutes, hours, - days, weeks, months, years. See updated - :ref:`performance_test_methodology` section for more details. + - **3n-dnv Tests (3rd Party)**: Published performance tests for 3n- + dnv (3-Node Atom Denverton) from 3rd party testbeds running FD.io + |csit-release| automated testing code. + Only graphs for Packet Throughput and Speedup Multi-core and not + for Packet Latency were published as there are no results for Packet + Latency available. - - **Trending Tests BMRR**: Used new Burst MRR (BMRR) tests for daily - trending. +.. note:: - - **Per VPP Patch Performance Checks**: Per VPP gerrit patch vs. - parent performance tests, anomaly detection and no verify voting - (-1/0/+1) yet. Manual trigger only. Not "marketed" to FD.io - community yet to avoid excessive LFN FD.io physical performance - testbed blocking. - - - **Patch-on-Patch Infra**: Added capability to run performance tests - using CSIT gerrit patch code testing VPP gerrit patch code, i.e. - before any VPP and/or CSIT code is merged into git branch. - - - **CSIT PAPI Support**: Initial implementation of PAPI L1 KWs in - CSIT using VPP Python bindings. Required for migraing away from - VAT. Very few L1 KWs implemented ("show version", "show - interfaces"). - - - **General Code Housekeeping**: Ongoing RF keywords optimizations, - removal of redundant RF keywords. - -#. PRESENTATION AND ANALYTICS LAYER - - - **Graphs Layout Improvements**: Improved performance graphs layout - for better readibility and maintenance: test grouping, axis - labels, descriptions, other informative decoration. Master report - generated. 744 graphs(!) - - - - **Performance Trending**: Further improvements of continuous - performance trending, anomaly detection and analysis. - -#. MISCELLANEOUS - - - **3n-dnv Tests (3rd Party)**: Published performance tests for 3n- - dnv (3-Node Atom Denverton) from 3rd party testbeds running FD.io - |csit-release| automated testing code. - -.. raw:: latex - - \clearpage + |csit-release| report was generated with a single run of selected + 64B frame performance tests on 3n-skx test bed + in order to spot-check the main data plane paths. + NFV service density and soak tests were not run and do not feature in + this report. .. _vpp_known_issues: @@ -98,16 +69,9 @@ List of known issues in |csit-release| for VPP performance tests: | 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. | | | `_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. | +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 2 | `CSIT-1234 | VPP IPSecHW scale interface mode 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | -| | `_ | ip4ipsecscale1000tnl-ip4base-int 1core CSIT-18.07/18.04 relative change: NDR -31%, PDR -32%, MRR -38%. | +| 2 | `VPP-1563 | AVF L2patch tests are failing for all packet size and core combination. Reason: null-node blackholed packets in show error. | +| | `_ | | +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 3 | `CSIT-1242 | VPP xl710 ip4base test 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | -| | `_ | xl710 ip4base 1core CSIT-18.07/18.04 relative change: NDR -29%, high stdev. | +| 3 | `CSIT-1465 | 4c VPP VM vhost tests failiing on 3n-skx | +| | `_ | | +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 4 | `CSIT-1243 | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. | -| | `_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%. | -+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 5 | `CSIT-1246 | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07. | -| | `_ | Past vpp-agent releases are not compatible either. | -+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -