+++ /dev/null
-Release Notes
-=============
-
-Changes in |csit-release|
--------------------------
-
-#. **VPP Performance Tests**
-
- - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now
- configurable trial duration and number of consecutive executions.
- Coverage of MRR tests has been extended across more test
- scenarios. MRR tests are used for continuous performance trending
- and for comparison between VPP releases.
-
- - **MLRsearch Throughput**: MLRsearch algorithm has been introduced
- for all NDR and PDR throughput tests. All tests that previously
- used binary search got converted to MLRsearch. Coverage of NDR/PDR
- tests has been extended across more test scenarios.
-
- - **L2patch Tests**: Tests measure performance of VPP L2patch, the
- fastest L2 forwarding path implemented in VPP, that cross-links
- RX and TX of two physical interfaces.
-
- - **2-Node Tests**: A new baseline set of 2-node tests covering base
- ip4, ip6, l2patch, l2bd, l2xc, running on new Xeon Skylake
- testbeds.
-
- - **Generated tests**: Simplified and unified test structure, semi-
- autogenerated by generator script. Test generator is currently
- able to create test combinations with various frame size and
- cores combinations. All existing test cases were converted to new
- format.
-
- - **Simultaneous Multi-Threading**: SMT-aware detection of server
- processor operation mode (HyperThreading enabled/disabled) with
- associated compute resource configuration including thread
- affinity, number of Rx queues and DPDK I/O mbufs. Tests are
- automatically tagged during execution to indicate executed thread
- configuration.
-
- - **Intel Xeon Skylake Support**: Support for 2-Node and 3-Node
- physical testbed topologies based on the new SuperMirco servers
- each with two Intel Xeon Skylake Platinum processors. Full
- Ansible playbooks refactor for quick server (re)installation and
- reference pointers of configuration.
-
-#. **Presentation and Analytics Layer**
-
- - **Performance trending**: Further improved continuous performance
- trending with anomaly detection and analysis.
-
-#. **Test Framework Optimizations**
-
- - **General Code Housekeeping**: Ongoing RF keywords optimizations,
- removal of redundant RF keywords.
-
-Performance Changes
--------------------
-
-Relative performance changes in measured packet throughput in |csit-release|
-are calculated against the results from |csit-release-1|
-report. Listed mean and standard deviation values are computed based on
-a series of the same tests executed against respective VPP releases to
-verify test results repeatability, with percentage change calculated for
-mean values. Note that the standard deviation is quite high for a small
-number of packet throughput tests, what indicates poor test results
-repeatability and makes the relative change of mean throughput value not
-fully representative for these tests. The root causes behind poor
-results repeatability vary between the test cases.
-
-NDR Changes
-~~~~~~~~~~~
-
-NDR throughput changes between releases are available in a
-CSV and pretty ASCII formats:
-
- - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-ndr.csv>`_,
- - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-ndr.csv>`_,
- - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-ndr.txt>`_,
- - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-ndr.txt>`_.
-
-.. note::
-
- Test results have been generated by
- `FD.io test executor vpp performance job 3n-hsw`_,
- with Robot Framework result
- files csit-vpp-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
-
-PDR Changes
-~~~~~~~~~~~
-
-PDR throughput changes between releases are available in a
-CSV and pretty ASCII formats:
-
- - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-pdr.csv>`_,
- - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-pdr.csv>`_,
- - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-pdr.txt>`_,
- - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-pdr.txt>`_.
-
-.. note::
-
- Test results have been generated by
- `FD.io test executor vpp performance job 3n-hsw`_,
- with Robot Framework result
- files csit-vpp-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
-
-MRR Changes
-~~~~~~~~~~~
-
-MRR small packet throughput changes between releases are available in a
-CSV and pretty ASCII formats:
-
- - `csv format for 1t1c <../_static/vpp/performance-changes-1t1c-mrr.csv>`_,
- - `csv format for 2t2c <../_static/vpp/performance-changes-2t2c-mrr.csv>`_,
- - `csv format for 4t4c <../_static/vpp/performance-changes-4t4c-mrr.csv>`_,
- - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-1t1c-mrr.txt>`_,
- - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-2t2c-mrr.txt>`_,
- - `pretty ASCII format for 4t4c <../_static/vpp/performance-changes-4t4c-mrr.txt>`_.
-
-.. note::
-
- Test results have been generated by
- `FD.io test executor vpp performance job 3n-hsw`_,
- with Robot Framework result
- files csit-vpp-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
-
-Comparison Across Testbeds
---------------------------
-
-Relative performance changes in measured packet throughputon 3-Node Skx testbed
-are calculated against the results measured on 3-Node Hsw testbed.
-
-NDR Changes
-~~~~~~~~~~~
-
-NDR changes between testbeds are available in a
-CSV and pretty ASCII formats:
-
- - `csv format for ndr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_,
- - `pretty ASCII format for ndr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_.
-
-.. note::
-
- Test results have been generated by
- `FD.io test executor vpp performance job 3n-hsw`_ and
- `FD.io test executor vpp performance job 3n-skx`_
- with Robot Framework result
- files csit-vpp-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
-
-PDR Changes
-~~~~~~~~~~~
-
-PDR throughput changes between testbeds are available in a
-CSV and pretty ASCII formats:
-
- - `csv format for pdr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_,
- - `pretty ASCII format for pdr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_.
-
-.. note::
-
- Test results have been generated by
- `FD.io test executor vpp performance job 3n-hsw`_ and
- `FD.io test executor vpp performance job 3n-skx`_
- with Robot Framework result
- files csit-vpp-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
-
-MRR Changes
-~~~~~~~~~~~
-
-MRR throughput changes between testbeds are available in a
-CSV and pretty ASCII formats:
-
- - `csv format for mrr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_,
- - `pretty ASCII format for mrr <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_.
-
-.. note::
-
- Test results have been generated by
- `FD.io test executor vpp performance job 3n-hsw`_ and
- `FD.io test executor vpp performance job 3n-skx`_
- with Robot Framework result
- files csit-vpp-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
-
-Throughput Trending
--------------------
-
-In addition to reporting throughput changes between VPP releases, CSIT
-provides continuous performance trending for VPP master branch:
-
-#. `VPP Performance Dashboard <https://docs.fd.io/csit/master/trending/introduction/index.html>`_
- - per VPP test case throughput trend, trend compliance and summary of
- detected anomalies.
-
-#. `Trending Methodology <https://docs.fd.io/csit/master/trending/methodology/index.html>`_
- - throughput test metrics, trend calculations and anomaly
- classification (progression, regression, outlier).
-
-#. `Trendline Graphs <https://docs.fd.io/csit/master/trending/trending/index.html>`_
- - per VPP build MRR throughput measurements against the trendline
- with anomaly highlights, with associated CSIT test jobs.
-
-Known Issues
-------------
-
-List of known issues in |csit-release| for VPP performance tests:
-
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+
-| # | Issue | Jira ID | Description |
-+===+=================================================+============+=================================================================+
-| 1 | Sporadic (1 in 200) NDR discovery test failures | CSIT-570 | DPDK reporting rx-errors, indicating L1 issue. Suspected issue |
-| | on x520. | | with HW combination of X710-X520 in LF testbeds. Not observed |
-| | | | outside of LF testbeds. |
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+
-| 2 | High failure rate of api call | VPP-1361 | Failure rate: 30-40% of tests failing due to interfaces not |
-| | sw_interface_set_flags [admin-up|link-up] | | in link-up state after API call sw_interface_set_flags. |
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+
-| 3 | Scale IPSecHW Interface mode throughput | CSIT-1234 | IPSec throughput regression: NDR -28%..-31%, PDR -28%..-31%, |
-| | regression. | | MRR -40%. Affects IPSec HW Scale 1000tnl tests with Int mode. |
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+
Code Review / csit.git / blobdiff