X-Git-Url: https://gerrit.fd.io/r/gitweb?a=blobdiff_plain;f=docs%2Freport%2Fvpp_performance_tests%2Fcsit_release_notes.rst;h=2aeb09f11e4d1a8d0cad7625dd3a547aa9de4292;hb=HEAD;hp=fb5737c4fc86a4cf8767dd9bf6703c4031e85a62;hpb=995f2727d6306ad05e8ea089632a552ebbf23b21;p=csit.git diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst deleted file mode 100644 index fb5737c4fc..0000000000 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ /dev/null @@ -1,131 +0,0 @@ -CSIT Release Notes -================== - -Changes in CSIT |release| -------------------------- - -#. **VPP performance tests** - - - *MRR tests* - New Maximum Receive Rate tests measure the packet - forwarding rate under the maximum load offered by traffic - generator over a set trial duration, regardless of packet loss. - MRR tests are used for continuous performance trending and for - comparison between releases. - - - *Service Chaining with SRv6* - New SRv6 (Segment Routing IPv6) proxy - tests measure performance of SRv6 Endpoint fronting SR-unaware - appliance via masquerading (End.AM), dynamic proxy (End.AD) or - static proxy (End.AS) SR functions. - -#. **Presentation and Analytics Layer** - - - *Performance trending* - Added continuous performance trending and - analysis. New Performance Trending and Performance Analysis jobs - executed regular throughput tests, with results being subsequently - analysed and trend and anomalies summarized and presented in VPP - Performance Dashboard and trendline graphs. - -#. **Test Framework Optimizations** - - - *Performance tests efficiency* - Qemu build/install optimizations, - warmup phase handling, vpp restart handling. Resulted in improved - stability and reduced total execution time by 30% for single pkt - size e.g. 64B/78B. - - - *General code housekeeping* - ongoing RF keywords optimizations, - removal of redundant RF keywords. - -Performance Changes -------------------- - -Relative performance changes in measured packet throughput in CSIT -|release| are calculated against the results from CSIT |release-1| -report. Listed mean and standard deviation values are computed based on -a series of the same tests executed against respective VPP releases to -verify test results repeatibility, with percentage change calculated for -mean values. Note that the standard deviation is quite high for a small -number of packet throughput tests, what indicates poor test results -repeatability and makes the relative change of mean throughput value not -fully representative for these tests. The root causes behind poor -results repeatibility vary between the test cases. - -NDR Changes -~~~~~~~~~~~ - -NDR small packet throughput changes between releases are available in a -CSV and pretty ASCII formats: - - - `csv format for 1t1c <../_static/vpp/performance-changes-ndr-1t1c-full.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-ndr-2t2c-full.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-ndr-1t1c-full.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-ndr-2t2c-full.txt>`_. - -PDR Changes -~~~~~~~~~~~ - -NDR small packet throughput changes between releases are available in a -CSV and pretty ASCII formats: - - - `csv format for 1t1c <../_static/vpp/performance-changes-pdr-1t1c-full.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-pdr-2t2c-full.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-pdr-1t1c-full.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-pdr-2t2c-full.txt>`_. - -MRR Changes -~~~~~~~~~~~ - -MRR small packet throughput changes between releases are available in a -CSV and pretty ASCII formats: - - - `csv format for 1t1c <../_static/vpp/performance-changes-mrr-1t1c-full.csv>`_, - - `csv format for 2t2c <../_static/vpp/performance-changes-mrr-2t2c-full.csv>`_, - - `csv format for 4t4c <../_static/vpp/performance-changes-mrr-4t4c-full.csv>`_, - - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-mrr-1t1c-full.txt>`_, - - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-mrr-2t2c-full.txt>`_, - - `pretty ASCII format for 4t4c <../_static/vpp/performance-changes-mrr-4t4c-full.txt>`_. - -Throughput Trending -------------------- - -In addition to reporting throughput changes between VPP releases, CSIT -provides continuous performance trending for VPP master branch: - -#. `VPP Performance Dashboard `_ - - per VPP test case throughput trend, trend compliance and summary of - detected anomalies. - -#. `Trending Methodology `_ - - throughput test metrics, trend calculations and anomaly - classification (progression, regression, outlier). - -#. `Trendline Graphs `_ - - per VPP build MRR throughput measurements against the trendline - with anomaly highlights, with associated CSIT test jobs. - -Known Issues ------------- - -List of known issues in CSIT |release| for VPP performance tests: - -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| # | Issue | Jira ID | Description | -+===+=================================================+============+=================================================================+ -| 1 | Sporadic (1 in 200) NDR discovery test failures | CSIT-570 | DPDK reporting rx-errors, indicating L1 issue. Suspected issue | -| | on x520. | | with HW combination of X710-X520 in LF testbeds. Not observed | -| | | | outside of LF testbeds. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 2 | Lower than expected NDR throughput of DPDK | CSIT-571 | Suspected NIC firmware or DPDK driver issue affecting NDR and | -| | testpmd and VPP L2 path NDR throughput with | | PDR throughput on XL710 and X710 NICs. | -| | xl710 and x710 NICs, compared to x520 NICs. | | | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 3 | Tagged Ethernet dot1q and dot1ad L2 path | CSIT-1066 | Tagged Ethernet dot1q and dot1ad L2 path throughput regression: | -| | throughput regression. | | NDR -2%..-5%, PDR -2%..-6%, MRR. Affects l2xc and l2bd | -| | | | performance tests. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 4 | IPSec (software, no QAT HW) throughput | CSIT-1064 | IPSec throughput regression: NDR -3%..-8%, PDR -2%..-8%, MRR | -| | regression. | | -3%..-7%. Affects IPSec SW tests, QAT HW tests not affected. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 5 | High failure rate of creating working container | CSIT-1065 | Orchestrated container topology tests failing data plane | -| | topologies with K8s/Ligato orchestration. | | verification indicating configuration issue. Suspected issue | -| | | | with Ligato vpp-agent. | -+---+-------------------------------------------------+------------+-----------------------------------------------------------------+