Code Review / csit.git / blobdiff
? search:
summary | shortlog | log | commit | commitdiff | review | tree
raw | inline | side by side
report 1807: editorial nit-picking and formatting all release_notes sections.
[csit.git] / docs / report / vpp_performance_tests / csit_release_notes.rst
diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst
index 74e3082..580ef48 100644 (file)
--- a/docs/report/vpp_performance_tests/csit_release_notes.rst
+++ b/docs/report/vpp_performance_tests/csit_release_notes.rst
@@ -4,36 +4,55 @@ Release Notes
 Changes in |csit-release|
 -------------------------
 
 Changes in |csit-release|
 -------------------------
 
-#. **VPP performance tests**
-
-   - *MRR tests* - New Maximum Receive Rate tests measure the packet
-     forwarding rate under the maximum load offered by traffic
-     generator over a set trial duration, regardless of packet loss.
-     MRR tests are used for continuous performance trending and for
-     comparison between releases.
-
-   - *Service Chaining with SRv6* - New SRv6 (Segment Routing IPv6) proxy
-     tests measure performance of SRv6 Endpoint fronting SR-unaware
-     appliance via masquerading (End.AM), dynamic proxy (End.AD) or
-     static proxy (End.AS) SR functions.
+#. **VPP Performance Tests**
+
+  - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now
+    configurable trial duration and number of consecutive executions.
+    Coverage of MRR tests has been extended across more test
+    scenarios. MRR tests are used for continuous performance trending
+    and for comparison between VPP releases.
+
+  - **MLRsearch Throughput**: MLRsearch algorithm has been introduced
+    for all NDR and PDR throughput tests. All tests that previously
+    used binary search got converted to MLRsearch. Coverage of NDR/PDR
+    tests has been extended across more test scenarios.
+
+  - **L2patch Tests**: Tests measure performance of VPP L2patch, the
+    fastest L2 forwarding path implemented in VPP, that cross-links
+    RX and TX of two physical interfaces.
+
+  - **2-Node Tests**: A new baseline set of 2-node tests covering base
+    ip4, ip6, l2patch, l2bd, l2xc, running on new Xeon Skylake
+    testbeds.
+
+  - **Generated tests**: Simplified and unified test structure, semi-
+    autogenerated by generator script. Test generator is currently
+    able to create test combinations with various frame size and
+    cores combinations. All existing test cases were converted to new
+    format.
+
+  - **Simultaneous Multi-Threading**: SMT-aware detection of server
+    processor operation mode (HyperThreading enabled/disabled) with
+    associated compute resource configuration including thread
+    affinity, number of Rx queues and DPDK I/O mbufs. Tests are
+    automatically tagged during execution to indicate executed thread
+    configuration.
+
+  - **Intel Xeon Skylake Support**: Support for 2-Node and 3-Node
+    physical testbed topologies based on the new SuperMirco servers
+    each with two Intel Xeon Skylake Platinum processors. Full
+    Ansible playbooks refactor for quick server (re)installation and
+    reference pointers of configuration.
 
 #. **Presentation and Analytics Layer**
 
 
 #. **Presentation and Analytics Layer**
 
-   - *Performance trending* - Added continuous performance trending and
-     analysis. New Performance Trending and Performance Analysis jobs
-     executed regular throughput tests, with results being subsequently
-     analysed and trend and anomalies summarized and presented in VPP
-     Performance Dashboard and trendline graphs.
+  - **Performance trending**: Further improved continuous performance
+    trending with anomaly detection and analysis.
 
 #. **Test Framework Optimizations**
 
 
 #. **Test Framework Optimizations**
 
-   - *Performance tests efficiency* - Qemu build/install optimizations,
-     warmup phase handling, vpp restart handling. Resulted in improved
-     stability and reduced total execution time by 30% for single pkt
-     size e.g. 64B/78B.
-
-   - *General code housekeeping* - ongoing RF keywords optimizations,
-     removal of redundant RF keywords.
+  - **General Code Housekeeping**: Ongoing RF keywords optimizations,
+    removal of redundant RF keywords.
 
 Performance Changes
 -------------------
 
 Performance Changes
 -------------------
@@ -103,9 +122,10 @@ CSV and pretty ASCII formats:
 .. note::
 
     Test results have been generated by
 .. note::
 
     Test results have been generated by
-    `FD.io test executor vpp mrr jobs <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master/>`_
-    with Robot Framework result files csit-vpp-perf-mrr-daily-master__*__output.xml.gz
-    `archived here <https://docs.fd.io/csit/master/trending/_static/archive/>`_.
+    `FD.io test executor vpp performance job 3n-hsw`_,
+    with Robot Framework result
+    files csit-vpp-perf-|srelease|-\*.zip
+    `archived here <../_static/archive/>`_.
 
 Comparison Across Testbeds
 --------------------------
 
 Comparison Across Testbeds
 --------------------------
@@ -149,6 +169,24 @@ CSV and pretty ASCII formats:
     files csit-vpp-perf-|srelease|-\*.zip
     `archived here <../_static/archive/>`_.
 
     files csit-vpp-perf-|srelease|-\*.zip
     `archived here <../_static/archive/>`_.
 
+MRR Changes
+~~~~~~~~~~~
+
+MRR throughput changes between testbeds are available in a
+CSV and pretty ASCII formats:
+
+  - `csv format for mrr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_,
+  - `pretty ASCII format for mrr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_.
+
+.. note::
+
+    Test results have been generated by
+    `FD.io test executor vpp performance job 3n-hsw`_ and
+    `FD.io test executor vpp performance job 3n-skx`_
+    with Robot Framework result
+    files csit-vpp-perf-|srelease|-\*.zip
+    `archived here <../_static/archive/>`_.
+
 Throughput Trending
 -------------------
 
 Throughput Trending
 -------------------
 
@@ -179,18 +217,9 @@ List of known issues in |csit-release| for VPP performance tests:
 |   | on x520.                                        |            | with HW combination of X710-X520 in LF testbeds. Not observed   |
 |   |                                                 |            | outside of LF testbeds.                                         |
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 |   | on x520.                                        |            | with HW combination of X710-X520 in LF testbeds. Not observed   |
 |   |                                                 |            | outside of LF testbeds.                                         |
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
-| 2 | Lower than expected NDR throughput of DPDK      | CSIT-571   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |
-|   | testpmd and VPP L2 path NDR throughput with     |            | PDR throughput on XL710 and X710 NICs.                          |
-|   | xl710 and x710 NICs, compared to x520 NICs.     |            |                                                                 |
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+
-| 3 | Tagged Ethernet dot1q and dot1ad L2 path        | CSIT-1066  | Tagged Ethernet dot1q and dot1ad L2 path throughput regression: |
-|   | throughput regression.                          |            | NDR -2%..-5%, PDR -2%..-6%, MRR. Affects l2xc and l2bd          |
-|   |                                                 |            | performance tests.                                              |
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+
-| 4 | IPSec (software, no QAT HW) throughput          | CSIT-1064  | IPSec throughput regression: NDR -3%..-8%, PDR -2%..-8%, MRR    |
-|   | regression.                                     |            | -3%..-7%. Affects IPSec SW tests, QAT HW tests not affected.    |
+| 2 | High failure rate of api call                   | VPP-1361   | Failure rate: 30-40% of tests failing due to interfaces not     |
+|   | sw_interface_set_flags [admin-up|link-up]       |            | in link-up state after API call sw_interface_set_flags.         |
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
-| 5 | High failure rate of creating working container | CSIT-1065  | About 20% of orchestrated container topology tests failing data |
-|   | topologies with K8s/Ligato orchestration.       |            | plane verification indicating configuration issue. Suspected    |
-|   |                                                 |            | issue with Ligato vpp-agent.                                    |
+| 3 | Scale IPSecHW Interface mode throughput         | CSIT-1234  | IPSec throughput regression: NDR -28%..-31%, PDR -28%..-31%,    |
+|   | regression.                                     |            | MRR -40%. Affects IPSec HW Scale 1000tnl tests with Int mode.   |
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+
Integration tests