Changes in |csit-release|
-------------------------
-#. **DMM Functional Tests**
+#. DMM FUNCTIONAL TESTS
- Added DMM base vs. epoll test case.
on 3-Node Skylake (3n-skx) and 3-Node Haswell (3n-hsw) physical testbed
types, in 1-core, 2-core and 4-core configurations.
-NDR Comparison
-~~~~~~~~~~~~~~
-
-NDR throughput comparison between testbeds is available in CSV and
-pretty ASCII formats:
-
- - `CSV NDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_,
- - `ASCII NDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_.
-
.. note::
Test results have been generated by
files csit-dpdk-perf-|srelease|-\*.zip
`archived here <../_static/archive/>`_.
-PDR Comparison
+NDR Comparison
~~~~~~~~~~~~~~
-PDR throughput comparison between testbeds is available in CSV and
-pretty ASCII formats:
+Comparison tables in ASCII and CSV formats:
- - `CSV PDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_,
- - `ASCII PDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_.
+ - `ASCII NDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_
+ - `CSV NDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_
-.. note::
+PDR Comparison
+~~~~~~~~~~~~~~
- Test results have been generated by
- `FD.io test executor dpdk performance job 3n-hsw`_ and
- `FD.io test executor dpdk performance job 3n-skx`_
- with RF result
- files csit-dpdk-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
+Comparison tables in ASCII and CSV formats:
+
+ - `ASCII PDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_
+ - `CSV PDR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_
MRR Comparison
~~~~~~~~~~~~~~
-MRR throughput comparison between testbeds is available in CSV and
-pretty ASCII formats:
-
- - `CSV MRR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_,
- - `ASCII MRR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_.
+Comparison tables in ASCII and CSV formats:
-.. note::
-
- Test results have been generated by
- `FD.io test executor dpdk performance job 3n-hsw`_ and
- `FD.io test executor dpdk performance job 3n-skx`_
- with RF result
- files csit-vpp-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
\ No newline at end of file
+ - `ASCII MRR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_
+ - `CSV MRR comparison <../_static/dpdk/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_
----------------------------
Relative comparison of DPDK Testpmd and L3fwd packet throughput (NDR,
-PDR and MRR) between |dpdk-release| and DPDK-18.02 (measured for |csit-
-release| and |csit-release-1| respectively) is calculated from results
-of tests running on 3-Node Intel Xeon Haswell testbeds (3n-hsw) in
-1-core, 2-core and 4-core (MRR only) configurations.
+PDR and MRR) between |dpdk-release| and DPDK-18.02 (measured for
+|csit-release| and |csit-release-1| respectively) is calculated from
+results of tests running on 3-Node Intel Xeon Haswell testbeds (3n-hsw)
+in 1-core and 2-core configurations.
Listed mean and standard deviation values are computed based on a series
of the same tests executed against respective DPDK releases to verify
test results repeatability, with percentage change calculated for mean
values.
-NDR Comparison
-~~~~~~~~~~~~~~
-
-NDR throughput comparison between releases is available in CSV and pretty
-ASCII formats:
-
- - `CSV 1t1c NDR changes <../_static/dpdk/performance-changes-1t1c-ndr.csv>`_,
- - `CSV 2t2c NDR changes <../_static/dpdk/performance-changes-2t2c-ndr.csv>`_,
- - `ASCII 1t1c NDR changes <../_static/dpdk/performance-changes-1t1c-ndr.txt>`_,
- - `ASCII 2t2c NDR changes <../_static/dpdk/performance-changes-2t2c-ndr.txt>`_.
-
.. note::
Test results have been generated by
files csit-dpdk-perf-|srelease|-\*.zip
`archived here <../_static/archive/>`_.
-PDR Comparison
+NDR Comparison
~~~~~~~~~~~~~~
-PDR throughput comparison between releases is available in CSV and pretty
-ASCII formats:
+Comparison tables in ASCII and CSV formats:
- - `CSV 1t1c PDR changes <../_static/dpdk/performance-changes-1t1c-pdr.csv>`_,
- - `CSV 2t2c PDR changes <../_static/dpdk/performance-changes-2t2c-pdr.csv>`_,
- - `ASCII 1t1c PDR changes <../_static/dpdk/performance-changes-1t1c-pdr.txt>`_,
- - `ASCII 2t2c PDR changes <../_static/dpdk/performance-changes-2t2c-pdr.txt>`_.
+ - `ASCII 1t1c NDR comparison <../_static/dpdk/performance-changes-1t1c-ndr.txt>`_
+ - `ASCII 2t2c NDR comparison <../_static/dpdk/performance-changes-2t2c-ndr.txt>`_
+ - `CSV 1t1c NDR comparison <../_static/dpdk/performance-changes-1t1c-ndr.csv>`_
+ - `CSV 2t2c NDR comparison <../_static/dpdk/performance-changes-2t2c-ndr.csv>`_
-.. note::
+PDR Comparison
+~~~~~~~~~~~~~~
- Test results have been generated by
- `FD.io test executor dpdk performance job 3n-hsw`_
- with RF result
- files csit-dpdk-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
+Comparison tables in ASCII and CSV formats:
+
+ - `ASCII 1t1c PDR comparison <../_static/dpdk/performance-changes-1t1c-pdr.txt>`_
+ - `ASCII 2t2c PDR comparison <../_static/dpdk/performance-changes-2t2c-pdr.txt>`_
+ - `CSV 1t1c PDR comparison <../_static/dpdk/performance-changes-1t1c-pdr.csv>`_
+ - `CSV 2t2c PDR comparison <../_static/dpdk/performance-changes-2t2c-pdr.csv>`_
Changes in |csit-release|
-------------------------
-#. **DPDK release version change**
+#. DPDK RELEASE VERSION CHANGE
- - |csit-release| tested DPDK-18.05, as used by |vpp-release|.
+ - |csit-release| tested |dpdk-release|, as used by |vpp-release|.
-#. **DPDK performance tests**
+#. DPDK PERFORMANCE TESTS
- **MRR Throughput**: MRR (Maximum Receive Rate) test code has now
configurable trial duration and number of consecutive executions.
cores combinations. All existing test cases were converted to new
format.
-#. **Presentation and Analytics Layer**
+#. PRESENTATION AND ANALYTICS LAYER
- - **Performance trending**: Added continuous performance trending with
+ - **Performance trending**: Added regular performance trending,
anomaly detection and analysis for DPDK.
Known Issues
- throughput test metrics, trend calculations and anomaly
classification (progression, regression).
-#. `DPDK Apps Trendline Graphs <https://docs.fd.io/csit/master/trending/trending/dpdk.html>`_
+#. `DPDK Trendline Graphs <https://docs.fd.io/csit/master/trending/trending/dpdk.html>`_
- weekly DPDK Testpmd and L3fwd MRR throughput measurements against
the trendline with anomaly highlights and associated CSIT test jobs.
\ No newline at end of file
Introduction
============
-This is FD.io CSIT (Fast Data i/o Continuous System Integration and
-Testing) project report for |csit-release| system performance and
-functional testing of |vpp-release|.
+FD.io |csit-release| report contains system performance and functional
+testing data for |vpp-release|.
-There is also a downloadable `PDF version of this report`_.
+`PDF version of this report`_ is also available for download.
-This report describes CSIT performance and functional tests delivered in
-|csit-release|. A high-level overview is provided for each CSIT test
-environment running in :abbr:`LF (Linux Foundation)` FD.io Continuous
-Performance Labs. This is followed by summary of all executed tests
-against the |vpp-release| and associated FD.io projects and sub-systems
-(Honeycomb, DPDK, NSH_SFC, DMM), |csit-release| release notes, result
-highlights and known issues discovered in CSIT. More detailed
-description of each environment, pointers to CSIT test code
-documentation and detailed test resuls with links to the source data
-files are also provided.
+|csit-release| report is structured as follows:
-|csit-release| report contains following main sections and sub-sections:
-
-#. **Introduction**: General introduction to CSIT project.
+#. INTRODUCTION: General introduction to CSIT project.
- **Introduction**: This section.
- **Test Scenarios Overview**: A brief overview of test scenarios
- **Physical Testbeds**: Description of physical testbeds.
- **Performance Test Methodology**: Benchmarking methodologies.
-#. **VPP Performance**: VPP performance tests executed in physical
+#. VPP PERFORMANCE: VPP performance tests executed in physical
FD.io testbeds.
- **Overview**: Tested logical topologies, test coverage and naming
in CSIT and pointers to CSIT source code documentation for VPP
performance tests.
-#. **DPDK Performance**: DPDK performance tests executed in physical
+#. DPDK PERFORMANCE: DPDK performance tests executed in physical
FD.io testbeds.
- **Overview**: Tested logical topologies, test coverage.
- **Documentation**: Pointers to CSIT source code documentation for
DPDK performance tests.
-#. **VPP Functional**: VPP functional tests executed in virtual FD.io
+#. VPP FUNCTIONAL: VPP functional tests executed in virtual FD.io
testbeds.
- **Overview**: Tested virtual topologies, test coverage and naming
- **Documentation**: Pointers to CSIT source code documentation for
VPP functional tests.
-#. **Honeycomb Functional**: Honeycomb functional tests executed in
+#. HONEYCOMBE FUNCTIONAL: Honeycomb functional tests executed in
virtual FD.io testbeds.
- **Overview**: Tested virtual topologies, test coverage and naming
- **Documentation**: Pointers to CSIT source code documentation for
Honeycomb functional tests.
-#. **NSH_SFC Functional**: NSH_SFC functional tests executed in
+#. NSH_SFC FUNCTIONAL: NSH_SFC functional tests executed in
virtual FD.io testbeds.
- **Overview**: Tested virtual topologies, test coverage and naming
- **Documentation**: Pointers to CSIT source code documentation for
NSH_SFC functional tests.
-#. **DMM Functional**: DMM functional tests executed in
+#. DMM FUNCTIONAL: DMM functional tests executed in
virtual FD.io testbeds.
- **Overview**: Tested virtual topologies, test coverage and naming
- **Documentation**: Pointers to CSIT source code documentation for
DMM functional tests.
-#. **Detailed Results**: Detailed result tables auto-generated from CSIT
+#. DETAILED RESULTS: Detailed result tables auto-generated from CSIT
test job executions using RF (Robot Framework) output files as
sources.
- **NSH_SFC Functional**: Detailed nsh-plugin functional results.
- **DMM Functional**: Detailed DMM functional results.
-#. **Test Configuration**: VPP DUT configuration data based on VPP API
+#. TEST CONFIGURATION: VPP DUT configuration data based on VPP API
Test (VAT) Commands History auto-generated from CSIT test job
executions using RF output files as sources.
- **VPP K8s Container Memif**: Configuration data.
- **VPP Functional**: Configuration data.
-#. **Test Operational Data**: VPP DUT operational data auto-generated
+#. TEST OPERATIONAL DATA: VPP DUT operational data auto-generated
from CSIT test job executions using RFoutput files as sources.
- **VPP Performance NDR/PDR**: VPP `show run` outputs under test
load.
-#. **CSIT Framework Documentation**: Description of the overall FD.io
+#. CSIT FRAMEWORK DOCUMENTATION: Description of the overall FD.io
CSIT framework.
- **Design**: Framework modular design hierarchy.
Test Scenarios Overview
=======================
-FD.io CSIT report includes multiple test scenarios of a number of VPP
+FD.io |csit-release| report includes multiple test scenarios of VPP
centric applications, topologies and use cases. In addition it also
-covers baseline tests of DPDK sample applications. Tests are executed
-executed in physical (performance tests) and virtual environments
-(functional tests).
+covers baseline tests of DPDK sample applications. Tests are executed in
+physical (performance tests) and virtual environments (functional
+tests).
Following list provides a brief overview of test scenarios covered in
this report:
Changes in |csit-release|
-------------------------
-#. **CSIT Framework**
+#. CSIT FRAMEWORK
- Few test case bug fixes.
-#. **Test Case Porting to VPP_make_test**
+#. TEST CASE PORTING TO VPP_MAKE_TEST
- Implementation of VIRL functional integration tests in
VPP_make_test.
skx) and 3-Node Haswell (3n-hsw) physical testbed types, in 1-core,
2-core and 4-core configurations.
-NDR Comparison
-~~~~~~~~~~~~~~
-
-NDR comparison between testbed types is available in CSV and pretty
-ASCII formats:
-
- - `CSV NDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_,
- - `ASCII NDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_.
-
.. note::
Test results have been generated by
files csit-vpp-perf-|srelease|-\*.zip
`archived here <../_static/archive/>`_.
-PDR Comparison
+NDR Comparison
~~~~~~~~~~~~~~
-PDR comparison between testbed types is available in CSV and pretty
-ASCII formats:
+Comparison tables in ASCII and CSV formats:
- - `CSV PDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_,
- - `ASCII PDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_.
+ - `ASCII NDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.txt>`_
+ - `CSV NDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-ndr.csv>`_
-.. note::
+PDR Comparison
+~~~~~~~~~~~~~~
- Test results have been generated by
- `FD.io test executor vpp performance job 3n-hsw`_ and
- `FD.io test executor vpp performance job 3n-skx`_
- with RF result
- files csit-vpp-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
+Comparison tables in ASCII and CSV formats:
+
+ - `ASCII PDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.txt>`_
+ - `CSV PDR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-pdr.csv>`_
MRR Comparison
~~~~~~~~~~~~~~
-MRR comparison between testbed types is available in CSV and pretty
-ASCII formats:
-
- - `CSV MRR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_,
- - `ASCII MRR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_.
-
-.. note::
+Comparison tables in ASCII and CSV formats:
- Test results have been generated by
- `FD.io test executor vpp performance job 3n-hsw`_ and
- `FD.io test executor vpp performance job 3n-skx`_
- with RF result
- files csit-vpp-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
+ - `ASCII MRR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.txt>`_
+ - `CSV MRR comparison <../_static/vpp/performance-compare-testbeds-3n-hsw-3n-skx-mrr.csv>`_
fully representative for these tests. The root causes behind poor
results repeatability vary between the test cases.
-NDR Comparison
-~~~~~~~~~~~~~~
-
-NDR throughput comparison between releases is available in CSV and pretty
-ASCII formats:
-
- - `CSV 1t1c NDR changes <../_static/vpp/performance-changes-1t1c-ndr.csv>`_,
- - `CSV 2t2c NDR changes <../_static/vpp/performance-changes-2t2c-ndr.csv>`_,
- - `ASCII 1t1c NDR changes <../_static/vpp/performance-changes-1t1c-ndr.txt>`_,
- - `ASCII 2t2c NDR changes <../_static/vpp/performance-changes-2t2c-ndr.txt>`_.
-
.. note::
Test results have been generated by
files csit-vpp-perf-|srelease|-\*.zip
`archived here <../_static/archive/>`_.
-PDR Comparison
+NDR Comparison
~~~~~~~~~~~~~~
-PDR throughput comparison between releases is available in CSV and pretty
-ASCII formats:
+Comparison tables in ASCII and CSV formats:
- - `CSV 1t1c PDR changes <../_static/vpp/performance-changes-1t1c-pdr.csv>`_,
- - `CSV 2t2c PDR changes <../_static/vpp/performance-changes-2t2c-pdr.csv>`_,
- - `ASCII 1t1c PDR changes <../_static/vpp/performance-changes-1t1c-pdr.txt>`_,
- - `ASCII 2t2c PDR changes <../_static/vpp/performance-changes-2t2c-pdr.txt>`_.
+ - `ASCII 1t1c NDR comparison <../_static/vpp/performance-changes-1t1c-ndr.txt>`_
+ - `ASCII 2t2c NDR comparison <../_static/vpp/performance-changes-2t2c-ndr.txt>`_
+ - `CSV 1t1c NDR comparison <../_static/vpp/performance-changes-1t1c-ndr.csv>`_
+ - `CSV 2t2c NDR comparison <../_static/vpp/performance-changes-2t2c-ndr.csv>`_
-.. note::
+PDR Comparison
+~~~~~~~~~~~~~~
- Test results have been generated by
- `FD.io test executor vpp performance job 3n-hsw`_,
- with RF result
- files csit-vpp-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
+Comparison tables in ASCII and CSV formats:
+
+ - `ASCII 1t1c PDR comparison <../_static/vpp/performance-changes-1t1c-pdr.txt>`_
+ - `ASCII 2t2c PDR comparison <../_static/vpp/performance-changes-2t2c-pdr.txt>`_
+ - `CSV 1t1c PDR comparison <../_static/vpp/performance-changes-1t1c-pdr.csv>`_
+ - `CSV 2t2c PDR comparison <../_static/vpp/performance-changes-2t2c-pdr.csv>`_
MRR Comparison
~~~~~~~~~~~~~~
-MRR throughput comparison between releases is available in CSV and pretty
-ASCII formats:
-
- - `CSV 1t1c MRR changes <../_static/vpp/performance-changes-1t1c-mrr.csv>`_,
- - `CSV 2t2c MRR changes <../_static/vpp/performance-changes-2t2c-mrr.csv>`_,
- - `CSV 4t4c MRR changes <../_static/vpp/performance-changes-4t4c-mrr.csv>`_,
- - `ASCII 1t1c MRR changes <../_static/vpp/performance-changes-1t1c-mrr.txt>`_,
- - `ASCII 2t2c MRR changes <../_static/vpp/performance-changes-2t2c-mrr.txt>`_,
- - `ASCII 4t4c MRR changes <../_static/vpp/performance-changes-4t4c-mrr.txt>`_.
-
-.. note::
+Comparison tables in ASCII and CSV formats:
- Test results have been generated by
- `FD.io test executor vpp performance job 3n-hsw`_,
- with RF result
- files csit-vpp-perf-|srelease|-\*.zip
- `archived here <../_static/archive/>`_.
+ - `ASCII 1t1c MRR comparison <../_static/vpp/performance-changes-1t1c-mrr.txt>`_
+ - `ASCII 2t2c MRR comparison <../_static/vpp/performance-changes-2t2c-mrr.txt>`_
+ - `ASCII 4t4c MRR comparison <../_static/vpp/performance-changes-4t4c-mrr.txt>`_
+ - `CSV 1t1c MRR comparison <../_static/vpp/performance-changes-1t1c-mrr.csv>`_
+ - `CSV 2t2c MRR comparison <../_static/vpp/performance-changes-2t2c-mrr.csv>`_
+ - `CSV 4t4c MRR comparison <../_static/vpp/performance-changes-4t4c-mrr.csv>`_
Changes in |csit-release|
-------------------------
-#. **VPP Performance Tests**
+#. VPP PERFORMANCE TESTS
- **MRR Throughput**: MRR (Maximum Receive Rate) test code has now
configurable trial duration and number of consecutive executions.
Ansible playbooks refactor for quick server (re)installation and
reference pointers of configuration.
-#. **Presentation and Analytics Layer**
+#. PRESENTATION AND ANALYTICS LAYER
- - **Performance trending**: Further improved continuous performance
- trending with anomaly detection and analysis.
+ - **Performance trending**: Further improvements of continuous
+ performance trending, anomaly detection and analysis.
-#. **Test Framework Optimizations**
+#. TEST FRAMEWORK OPTIMIZATIONS
- **General Code Housekeeping**: Ongoing RF keywords optimizations,
removal of redundant RF keywords.
In addition to reporting throughput comparison between VPP releases,
CSIT provides continuous performance trending for VPP master branch:
-#. `VPP Performance Dashboard <https://docs.fd.io/csit/master/trending/introduction/index.html>`_
+#. `Performance Dashboard <https://docs.fd.io/csit/master/trending/introduction/index.html>`_
- per VPP test case throughput trend, trend compliance and summary of
detected anomalies.
- throughput test metrics, trend calculations and anomaly
classification (progression, regression).
-#. `Trendline Graphs <https://docs.fd.io/csit/master/trending/trending/index.html>`_
+#. `VPP Trendline Graphs <https://docs.fd.io/csit/master/trending/trending/index.html>`_
- per VPP build MRR throughput measurements against the trendline
with anomaly highlights and associated CSIT test jobs.
\ No newline at end of file