report 18.07: moved performance change/compare sections out of rls notes to separate...

[csit.git] / docs / report / dpdk_performance_tests / csit_release_notes.rst

diff --git a/docs/report/dpdk_performance_tests/csit_release_notes.rst b/docs/report/dpdk_performance_tests/csit_release_notes.rst
index 363c2e7..e0e2028 100644 (file)
--- a/docs/report/dpdk_performance_tests/csit_release_notes.rst
+++ b/docs/report/dpdk_performance_tests/csit_release_notes.rst
@@ -4,114 +4,40 @@ Release Notes
 Changes in |csit-release|
 -------------------------
 
 Changes in |csit-release|
 -------------------------
 

-No code changes apart from bug fixes.
+#. **DPDK release version change**

 
 

-Performance Changes
--------------------
+   - |csit-release| tested DPDK-18.05, as used by |vpp-release|.

 
 

-Relative performance changes in measured packet throughput in |csit-release|
-are calculated against the results from |csit-release-1|
-report. Listed mean and standard deviation values are computed based on
-a series of the same tests executed against respective VPP releases to
-verify test results repeatability, with percentage change calculated for
-mean values. Note that the standard deviation is quite high for a small
-number of packet throughput tests, what indicates poor test results
-repeatability and makes the relative change of mean throughput value not
-fully representative for these tests. The root causes behind poor
-results repeatability vary between the test cases.
+#. **DPDK performance tests**

 
 

-NDR Changes
-~~~~~~~~~~~
+   - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now
+     configurable trial duration and number of consecutive executions.
+     Coverage of MRR tests has been extended across more test
+     scenarios. MRR tests are used for continuous performance trending.

 
 

-NDR throughput changes between releases are available in a
-CSV and pretty ASCII formats:
+   - **MLRsearch Throughput**: MLRsearch algorithm has been introduced
+     for all NDR and PDR throughput tests. All tests that previously
+     used binary search got converted to MLRsearch. Coverage of NDR/PDR
+     tests has been extended across more test scenarios.

 
 

-  - `csv format for 1t1c <../_static/dpdk/performance-changes-1t1c-ndr.csv>`_,
-  - `csv format for 2t2c <../_static/dpdk/performance-changes-2t2c-ndr.csv>`_,
-  - `pretty ASCII format for 1t1c <../_static/dpdk/performance-changes-1t1c-ndr.txt>`_,
-  - `pretty ASCII format for 2t2c <../_static/dpdk/performance-changes-2t2c-ndr.txt>`_.
+   - **2-Node Tests**: A new set of 2-node tests covering DPDK Testpmd
+     and L3fwd, running on new Xeon Skylake testbeds.

 
 

-.. note::
+   - **More NICs**: Increased coverage of NIC models tested: Intel-
+     xxv710-da2, Intel-x710-da4.

 
 

-    Test results have been generated by
-    `FD.io test executor dpdk performance job 3n-hsw`_
-    with Robot Framework result
-    files csit-vpp-perf-|srelease|-\*.zip
-    `archived here <../_static/archive/>`_.
+   - **Generated tests**: Simplified and unified test structure, semi-
+     autogenerated by generator script. Test generator is currently
+     able to create test combinations with various frame size and
+     cores combinations. All existing test cases were converted to new
+     format.

 
 

-PDR Changes
-~~~~~~~~~~~
+#. **Presentation and Analytics Layer**

 
 

-PDR throughput changes between releases are available in a
-CSV and pretty ASCII formats:
-
-  - `csv format for 1t1c <../_static/dpdk/performance-changes-1t1c-pdr.csv>`_,
-  - `csv format for 2t2c <../_static/dpdk/performance-changes-2t2c-pdr.csv>`_,
-  - `pretty ASCII format for 1t1c <../_static/dpdk/performance-changes-1t1c-pdr.txt>`_,
-  - `pretty ASCII format for 2t2c <../_static/dpdk/performance-changes-2t2c-pdr.txt>`_.
-
-.. note::
-
-    Test results have been generated by
-    `FD.io test executor dpdk performance job 3n-hsw`_
-    with Robot Framework result
-    files csit-vpp-perf-|srelease|-\*.zip
-    `archived here <../_static/archive/>`_.
-
-Comparison Across Testbeds
---------------------------
-
-Relative performance changes in measured packet throughputon 3-Node Skx testbed
-are calculated against the results measured on 3-Node Hsw testbed.
-
-NDR Changes
-~~~~~~~~~~~
-
-NDR changes between testbeds are available in a
-CSV and pretty ASCII formats:
-
-  - `csv format for ndr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_,
-  - `pretty ASCII format for ndr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_.
-
-.. note::
-
-    Test results have been generated by
-    `FD.io test executor dpdk performance job 3n-hsw`_ and
-    `FD.io test executor dpdk performance job 3n-skx`_
-    with Robot Framework result
-    files csit-vpp-perf-|srelease|-\*.zip
-    `archived here <../_static/archive/>`_.
-
-PDR Changes
-~~~~~~~~~~~
-
-PDR throughput changes between testbeds are available in a
-CSV and pretty ASCII formats:
-
-  - `csv format for pdr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_,
-  - `pretty ASCII format for pdr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_.
-
-.. note::
-
-    Test results have been generated by
-    `FD.io test executor dpdk performance job 3n-hsw`_ and
-    `FD.io test executor dpdk performance job 3n-skx`_
-    with Robot Framework result
-    files csit-vpp-perf-|srelease|-\*.zip
-    `archived here <../_static/archive/>`_.
+   - **Performance trending**: Added continuous performance trending with
+     anomaly detection and analysis for DPDK.

 
 Known Issues
 ------------
 
 
 Known Issues
 ------------
 

-Here is the list of known issues in |csit-release| for Testpmd performance tests:
-
-+---+---------------------------------------------------+------------+-----------------------------------------------------------------+
-| # | Issue                                             | Jira ID    | Description                                                     |
-+---+---------------------------------------------------+------------+-----------------------------------------------------------------+
-| 1 | Testpmd in 1t1c and 2t2c setups - large variation | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR       |
-|   | of discovered NDR throughput values across        |            | throughput. Applies to XL710 and X710 NICs, no issues observed  |
-|   | multiple test runs with xl710 and x710 NICs.      |            | on x520 NICs.                                                   |
-+---+---------------------------------------------------+------------+-----------------------------------------------------------------+
-| 2 | Lower than expected NDR throughput with xl710     | CSIT-571   | Suspected NIC firmware or DPDK driver issue affecting NDR       |
-|   | and x710 NICs, compared to x520 NICs.             |            | throughput. Applies to XL710 and X710 NICs.                     |
-+---+---------------------------------------------------+------------+-----------------------------------------------------------------+
+No known issues.