X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Freport%2Fvpp_performance_tests%2Fcsit_release_notes.rst;h=0318e7a01e9893cb557b50e67c77e5f4932bba84;hp=17003bc85ac6aaddca84136062a505b94f2c8b77;hb=637eb126bb4ed30df61a293e03d36bd8ad5265e5;hpb=cddac498bafc7a6092dade5e183e5c7a95cff64d diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index 17003bc85a..0318e7a01e 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -1,125 +1,85 @@ -CSIT Release Notes -================== - -Changes in CSIT |release| -------------------------- - -#. Added VPP performance tests - - - **Container Service Chain Topologies Orchestrated by K8s with VPP Memif** - - - Added tests with VPP vswitch in container connecting a number of VPP- - in-container service chain topologies with L2 Cross-Connect and L2 - Bridge-Domain configurations, orchestrated by Kubernetes. Added - following forwarding topologies: i) "Parallel" with packets flowing from - NIC via VPP to container and back to VPP and NIC; ii) "Chained" (a.k.a. - "Snake") with packets flowing via VPP to container, back to VPP, to next - container, back to VPP and so on until the last container in a chain, - then back to VPP and NIC; iii) "Horizontal" with packets flowing via VPP - to container, then via "horizontal" memif to next container, and so on - until the last container, then back to VPP and NIC; - - - **VPP TCP/IP stack** - - - Added tests for VPP TCP/IP stack using VPP built-in HTTP server. - WRK traffic generator is used as a client-side; - - - **SRv6** - - - Initial SRv6 (Segment Routing IPv6) tests verifying performance of - IPv6 and SRH (Segment Routing Header) encapsulation, decapsulation, - lookups and rewrites based on configured End and End.DX6 SRv6 egress - functions; - - - **IPSecSW** - - - SW computed IPSec encryption with AES-GCM, CBC-SHA1 ciphers, in - combination with IPv4 routed-forwarding; - -#. Presentation and Analytics Layer - - - Added throughput speedup analysis for multi-core and multi-thread - VPP tests into Presentation and Analytics Layer (PAL) for automated - CSIT test results analysis; - -#. Other changes - - - **Framework optimizations** - - - Ability to run CSIT framework on ARM architecture; - - - Overall stability improvements; - - - **NDR and PDR throughput binary search change** - - - Increased binary search resolution by reducing final step from - 100kpps to 50kpps; - - - **VPP plugin loaded as needed by tests** - - - From this release only plugins required by tests are loaded at - VPP initialization time. Previously all plugins were loaded for - all tests; - -Performance Changes -------------------- - -Relative performance changes in measured packet throughput in CSIT -|release| are calculated against the results from CSIT |release-1| -report. Listed mean and standard deviation values are computed based on -a series of the same tests executed against respective VPP releases to -verify test results repeatibility, with percentage change calculated for -mean values. Note that the standard deviation is quite high for a small -number of packet throughput tests, what indicates poor test results -repeatability and makes the relative change of mean throughput value not -fully representative for these tests. The root causes behind poor -results repeatibility vary between the test cases. - -NDR Throughput Changes -~~~~~~~~~~~~~~~~~~~~~~ - -NDR small packet throughput changes between releases are available in a CSV and -pretty ASCII formats: - - - `csv format for 1t1c <../_static/vpp/performance-changes-ndr-1t1c-full.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-ndr-2t2c-full.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-ndr-1t1c-full.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-ndr-2t2c-full.txt>`_. - -PDR Throughput Changes -~~~~~~~~~~~~~~~~~~~~~~ - -NDR small packet throughput changes between releases are available in a CSV and -pretty ASCII formats: - - - `csv format for 1t1c <../_static/vpp/performance-changes-pdr-1t1c-full.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-pdr-2t2c-full.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-pdr-1t1c-full.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-pdr-2t2c-full.txt>`_. - -Measured improvements are in line with VPP code optimizations listed in -`VPP-18.01 release notes -`_. - -Known Issues ------------- - -Here is the list of known issues in CSIT |release| for VPP performance tests: - -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| # | Issue | Jira ID | Description | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 1 | Vic1385 and Vic1227 low performance. | VPP-664 | Low NDR performance. | -| | | | | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 2 | Sporadic (1 in 200) NDR discovery test failures | CSIT-570 | DPDK reporting rx-errors, indicating L1 issue. Suspected issue | -| | on x520. | | with HW combination of X710-X520 in LF testbeds. Not observed | -| | | | outside of LF testbeds. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 3 | Lower than expected NDR throughput with | CSIT-571 | Suspected NIC firmware or DPDK driver issue affecting NDR and | -| | xl710 and x710 NICs, compared to x520 NICs. | | PDR throughput. Applies to XL710 and X710 NICs. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 4 | QAT IPSec scale with 1000 tunnels (interfaces) | VPP-1121 | VPP crashes during configuration of 1000 IPsec tunnels. | -| | in 2t2c config, all tests are failing. | | 1t1c tests are not affected | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ - +Release Notes +============= + +Changes in |csit-release| +------------------------- + +#. **VPP Performance Tests** + + - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now + configurable trial duration and number of consecutive executions. + Coverage of MRR tests has been extended across more test + scenarios. MRR tests are used for continuous performance trending + and for comparison between VPP releases. + + - **MLRsearch Throughput**: MLRsearch algorithm has been introduced + for all NDR and PDR throughput tests. All tests that previously + used binary search got converted to MLRsearch. Coverage of NDR/PDR + tests has been extended across more test scenarios. + + - **L2patch Tests**: Tests measure performance of VPP L2patch, the + fastest L2 forwarding path implemented in VPP, that cross-links + RX and TX of two physical interfaces. + + - **2-Node Tests**: A new baseline set of 2-node tests covering base + ip4, ip6, l2patch, l2bd, l2xc, running on new Xeon Skylake + testbeds. + + - **Generated tests**: Simplified and unified test structure, semi- + autogenerated by generator script. Test generator is currently + able to create test combinations with various frame size and + cores combinations. All existing test cases were converted to new + format. + + - **Simultaneous Multi-Threading**: SMT-aware detection of server + processor operation mode (HyperThreading enabled/disabled) with + associated compute resource configuration including thread + affinity, number of Rx queues and DPDK I/O mbufs. Tests are + automatically tagged during execution to indicate executed thread + configuration. + + - **Intel Xeon Skylake Support**: Support for 2-Node and 3-Node + physical testbed topologies based on the new SuperMirco servers + each with two Intel Xeon Skylake Platinum processors. Full + Ansible playbooks refactor for quick server (re)installation and + reference pointers of configuration. + +#. **Presentation and Analytics Layer** + + - **Performance trending**: Further improved continuous performance + trending with anomaly detection and analysis. + +#. **Test Framework Optimizations** + + - **General Code Housekeeping**: Ongoing RF keywords optimizations, + removal of redundant RF keywords. + +Known Issues +------------ + +List of known issues in |csit-release| for VPP performance tests: + ++---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| # | JiraID | Issue Description | ++===+=========================================+=================================================================================================================================+ +| 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. | +| | `_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. | ++---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 2 | `VPP-1361 | High failure rate of api call sw_interface_set_flags [admin-up|link-up]. | +| | `_ | Failure rate: 30-40% of tests failing due to interfaces not in link-up state after API call sw_interface_set_flags. | ++---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 3 | `CSIT-1234 | VPP IPSecHW scale interface mode 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | +| | `_ | ip4ipsecscale1000tnl-ip4base-int 1core CSIT-18.07/18.04 relative change: NDR -31%, PDR -32%, MRR -38%. | ++---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 4 | `CSIT-1242 | VPP xl710 ip4base test 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | +| | `_ | xl710 ip4base 1core CSIT-18.07/18.04 relative change: NDR -29%, high stdev. | ++---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 5 | `CSIT-1243 | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. | +| | `_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%. | ++---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 6 | `CSIT-1244 | VPP lispip4 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. | +| | `_ | ip4lispip4-ip4base 2core 3n-skx/3n-hsw relative change: NDR -11%, PDR -18%. | ++---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 7 | `CSIT-1245 | VPP srv6proxy-stat and srv6proxy-masq, much higher NDR and PDR 78B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | +| | `_ | Due to wrong test suite configuration in dynamic-proxy mode. Artefact of suite code refactoring. | ++---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+