Release Notes
=============

Changes in |csit-release|
-------------------------

#. **DPDK performance tests**

  - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now
    configurable trial duration and number of consecutive executions.
    Coverage of MRR tests has been extended across more test
    scenarios. MRR tests are used for continuous performance trending
    and for comparison between DPDK releases.

  - **MLRsearch Throughput**: MLRsearch algorithm has been introduced
    for all NDR and PDR throughput tests. All tests that previously
    used binary search got converted to MLRsearch. Coverage of NDR/PDR
    tests has been extended across more test scenarios.

  - **2-Node Tests**: A new set of 2-node tests covering DPDK Testpmd
    and L3fwd, running on new Xeon Skylake testbeds.

  - **More NICs**: Increased coverage of NIC models tested: Intel-
    xxv710-da2, Intel-x710-da4.

  - **Generated tests**: Simplified and unified test structure, semi-
    autogenerated by generator script. Test generator is currently
    able to create test combinations with various frame size and
    cores combinations. All existing test cases were converted to new
    format.

#. **Presentation and Analytics Layer**

  - **Performance trending**: Added continuous performance trending with
    anomaly detection and analysis for DPDK.

Performance Changes
-------------------

Relative performance changes in measured packet throughput in |csit-release|
are calculated against the results from |csit-release-1|
report. Listed mean and standard deviation values are computed based on
a series of the same tests executed against respective DPDK releases to
verify test results repeatability, with percentage change calculated for
mean values.

NDR Changes
~~~~~~~~~~~

NDR throughput changes between releases are available in a CSV and pretty ASCII
formats:

  - `csv format for 1t1c <../_static/dpdk/performance-changes-1t1c-ndr.csv>`_,
  - `csv format for 2t2c <../_static/dpdk/performance-changes-2t2c-ndr.csv>`_,
  - `pretty ASCII format for 1t1c <../_static/dpdk/performance-changes-1t1c-ndr.txt>`_,
  - `pretty ASCII format for 2t2c <../_static/dpdk/performance-changes-2t2c-ndr.txt>`_.

.. note::

    Test results have been generated by
    `FD.io test executor dpdk performance job 3n-hsw`_
    with Robot Framework result
    files csit-dpdk-perf-|srelease|-\*.zip
    `archived here <../_static/archive/>`_.

PDR Changes
~~~~~~~~~~~

PDR throughput changes between releases are available in a CSV and pretty ASCII
formats:

  - `csv format for 1t1c <../_static/dpdk/performance-changes-1t1c-pdr.csv>`_,
  - `csv format for 2t2c <../_static/dpdk/performance-changes-2t2c-pdr.csv>`_,
  - `pretty ASCII format for 1t1c <../_static/dpdk/performance-changes-1t1c-pdr.txt>`_,
  - `pretty ASCII format for 2t2c <../_static/dpdk/performance-changes-2t2c-pdr.txt>`_.

.. note::

    Test results have been generated by
    `FD.io test executor dpdk performance job 3n-hsw`_
    with Robot Framework result
    files csit-dpdk-perf-|srelease|-\*.zip
    `archived here <../_static/archive/>`_.

MRR Changes
~~~~~~~~~~~

MRR small packet throughput changes between releases are available in a
CSV and pretty ASCII formats:

  - `csv format for 1t1c <../_static/dpdk/performance-changes-1t1c-mrr.csv>`_,
  - `csv format for 2t2c <../_static/dpdk/performance-changes-2t2c-mrr.csv>`_,
  - `csv format for 4t4c <../_static/dpdk/performance-changes-4t4c-mrr.csv>`_,
  - `pretty ASCII format for 1t1c <../_static/dpdk/performance-changes-1t1c-mrr.txt>`_,
  - `pretty ASCII format for 2t2c <../_static/dpdk/performance-changes-2t2c-mrr.txt>`_,
  - `pretty ASCII format for 4t4c <../_static/dpdk/performance-changes-4t4c-mrr.txt>`_.

.. note::

    Test results have been generated by
    `FD.io test executor dpdk performance job 3n-hsw`_,
    with Robot Framework result
    files csit-vpp-perf-|srelease|-\*.zip
    `archived here <../_static/archive/>`_.

Comparison Across Testbeds
--------------------------

Relative performance changes in measured packet throughput on 3-Node Skx testbed
are calculated against the results measured on 3-Node Hsw testbed.

NDR Changes
~~~~~~~~~~~

NDR throughput changes between testbeds are available in a CSV and pretty ASCII
formats:

  - `csv format for ndr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_,
  - `pretty ASCII format for ndr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_.

.. note::

    Test results have been generated by
    `FD.io test executor dpdk performance job 3n-hsw`_ and
    `FD.io test executor dpdk performance job 3n-skx`_
    with Robot Framework result
    files csit-dpdk-perf-|srelease|-\*.zip
    `archived here <../_static/archive/>`_.

PDR Changes
~~~~~~~~~~~

PDR throughput changes between testbeds are available in a CSV and pretty ASCII
formats:

  - `csv format for pdr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_,
  - `pretty ASCII format for pdr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_.

.. note::

    Test results have been generated by
    `FD.io test executor dpdk performance job 3n-hsw`_ and
    `FD.io test executor dpdk performance job 3n-skx`_
    with Robot Framework result
    files csit-dpdk-perf-|srelease|-\*.zip
    `archived here <../_static/archive/>`_.

MRR Changes
~~~~~~~~~~~

MRR throughput changes between testbeds are available in a
CSV and pretty ASCII formats:

  - `csv format for mrr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_,
  - `pretty ASCII format for mrr <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_.

.. note::

    Test results have been generated by
    `FD.io test executor dpdk performance job 3n-hsw`_ and
    `FD.io test executor dpdk performance job 3n-skx`_
    with Robot Framework result
    files csit-vpp-perf-|srelease|-\*.zip
    `archived here <../_static/archive/>`_.

Known Issues
------------

Here is the list of known issues in |csit-release| for Testpmd performance tests:

+---+---------------------------------------------------+------------+-----------------------------------------------------------------+
| # | Issue                                             | Jira ID    | Description                                                     |
+---+---------------------------------------------------+------------+-----------------------------------------------------------------+
|   | No known issues                                   |            |                                                                 |
+---+---------------------------------------------------+------------+-----------------------------------------------------------------+