-CSIT Release Notes\r
-==================\r
-\r
-Changes in CSIT |release|\r
--------------------------\r
-\r
-#. Added VPP performance tests\r
-\r
- - **Container Service Chain Topologies Orchestrated by K8s with VPP Memif**\r
-\r
- - Added tests with VPP vswitch in container connecting a number of VPP-\r
- in-container service chain topologies with L2 Cross-Connect and L2\r
- Bridge-Domain configurations, orchestrated by Kubernetes. Added\r
- following forwarding topologies: i) "Parallel" with packets flowing from\r
- NIC via VPP to container and back to VPP and NIC; ii) "Chained" (a.k.a.\r
- "Snake") with packets flowing via VPP to container, back to VPP, to next\r
- container, back to VPP and so on until the last container in a chain,\r
- then back to VPP and NIC; iii) "Horizontal" with packets flowing via VPP\r
- to container, then via "horizontal" memif to next container, and so on\r
- until the last container, then back to VPP and NIC;\r
-\r
- - **VPP TCP/IP stack**\r
-\r
- - Added tests for VPP TCP/IP stack using VPP built-in HTTP server.\r
- WRK traffic generator is used as a client-side;\r
-\r
- - **SRv6**\r
-\r
- - Initial SRv6 (Segment Routing IPv6) tests verifying performance of\r
- IPv6 and SRH (Segment Routing Header) encapsulation, decapsulation,\r
- lookups and rewrites based on configured End and End.DX6 SRv6 egress\r
- functions;\r
-\r
- - **IPSecSW**\r
-\r
- - SW computed IPSec encryption with AES-GCM, CBC-SHA1 ciphers, in\r
- combination with IPv4 routed-forwarding;\r
-\r
-#. Presentation and Analytics Layer\r
-\r
- - Added throughput speedup analysis for multi-core and multi-thread\r
- VPP tests into Presentation and Analytics Layer (PAL) for automated\r
- CSIT test results analysis;\r
-\r
-#. Other changes\r
-\r
- - **Framework optimizations**\r
-\r
- - Ability to run CSIT framework on ARM architecture;\r
-\r
- - Overall stability improvements;\r
-\r
- - **NDR and PDR throughput binary search change**\r
-\r
- - Increased binary search resolution by reducing final step from\r
- 100kpps to 50kpps;\r
-\r
- - **VPP plugin loaded as needed by tests**\r
-\r
- - From this release only plugins required by tests are loaded at\r
- VPP initialization time. Previously all plugins were loaded for\r
- all tests;\r
-\r
-Performance Changes\r
--------------------\r
-\r
-Relative performance changes in measured packet throughput in CSIT\r
-|release| are calculated against the results from CSIT |release-1|\r
-report. Listed mean and standard deviation values are computed based on\r
-a series of the same tests executed against respective VPP releases to\r
-verify test results repeatibility, with percentage change calculated for\r
-mean values. Note that the standard deviation is quite high for a small\r
-number of packet throughput tests, what indicates poor test results\r
-repeatability and makes the relative change of mean throughput value not\r
-fully representative for these tests. The root causes behind poor\r
-results repeatibility vary between the test cases.\r
-\r
-NDR Throughput Changes\r
-~~~~~~~~~~~~~~~~~~~~~~\r
-\r
-NDR small packet throughput changes between releases are available in a CSV and\r
-pretty ASCII formats:\r
-\r
- - `csv format for 1t1c <../_static/vpp/performance-changes-ndr-1t1c-full.csv>`_,\r
- - `csv format for 2t2c <../_static/vpp/performance-changes-ndr-2t2c-full.csv>`_,\r
- - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-ndr-1t1c-full.txt>`_,\r
- - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-ndr-2t2c-full.txt>`_.\r
-\r
-PDR Throughput Changes\r
-~~~~~~~~~~~~~~~~~~~~~~\r
-\r
-NDR small packet throughput changes between releases are available in a CSV and\r
-pretty ASCII formats:\r
-\r
- - `csv format for 1t1c <../_static/vpp/performance-changes-pdr-1t1c-full.csv>`_,\r
- - `csv format for 2t2c <../_static/vpp/performance-changes-pdr-2t2c-full.csv>`_,\r
- - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-pdr-1t1c-full.txt>`_,\r
- - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-pdr-2t2c-full.txt>`_.\r
-\r
-Measured improvements are in line with VPP code optimizations listed in\r
-`VPP-18.01 release notes\r
-<https://docs.fd.io/vpp/18.01/release_notes_1801.html>`_.\r
-\r
-MRR Throughput Changes\r
-~~~~~~~~~~~~~~~~~~~~~~\r
-\r
-MRR changes between releases are available in a CSV and\r
-pretty ASCII formats:\r
-\r
- - `csv format for 1t1c <../_static/vpp/performance-changes-mrr-1t1c-full.csv>`_,\r
- - `csv format for 2t2c <../_static/vpp/performance-changes-mrr-2t2c-full.csv>`_,\r
- - `csv format for 4t4c <../_static/vpp/performance-changes-mrr-4t4c-full.csv>`_,\r
- - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-mrr-1t1c-full.txt>`_,\r
- - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-mrr-2t2c-full.txt>`_,\r
- - `pretty ASCII format for 4t4c <../_static/vpp/performance-changes-mrr-4t4c-full.txt>`_.\r
-\r
-Known Issues\r
-------------\r
-\r
-Here is the list of known issues in CSIT |release| for VPP performance tests:\r
-\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| # | Issue | Jira ID | Description |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 1 | Vic1385 and Vic1227 low performance. | VPP-664 | Low NDR performance. |\r
-| | | | |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 2 | Sporadic (1 in 200) NDR discovery test failures | CSIT-570 | DPDK reporting rx-errors, indicating L1 issue. Suspected issue |\r
-| | on x520. | | with HW combination of X710-X520 in LF testbeds. Not observed |\r
-| | | | outside of LF testbeds. |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 3 | Lower than expected NDR throughput with | CSIT-571 | Suspected NIC firmware or DPDK driver issue affecting NDR and |\r
-| | xl710 and x710 NICs, compared to x520 NICs. | | PDR throughput. Applies to XL710 and X710 NICs. |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 4 | QAT IPSec scale with 1000 tunnels (interfaces) | VPP-1121 | VPP crashes during configuration of 1000 IPsec tunnels. |\r
-| | in 2t2c config, all tests are failing. | | 1t1c tests are not affected |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 5 | rls1801 plugin related performance regression | CSIT-925 | With all plugins loaded NDR, PDR and MaxRates vary |\r
-| | | | intermittently from 3% to 5% across multiple test executions. |\r
-| | | | Requires plugin code bisecting. |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 6 | rls1801 generic small performance regression | CSIT-926 | Generic performance regression of discovered NDR, PDR and |\r
-| | ip4base, l2xcbase, l2bdbase | | MaxRates of -3%..-1% vs. rls1710, affects ip4base, l2xcbase, |\r
-| | | | l2bdbase test suites. Not detected by CSIT performance trending |\r
-| | | | scheme as it was masked out by another issue CSIT-925. |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 7 | rls1801 substantial NDR performance regression | CSIT-927 | Much lower NDR for vhostvr1024 tests, with mean values |\r
-| | for vhost-user vring size of 1024 | | regression of -17%..-42% vs. rls1710, but also very high |\r
-| | | | standard deviation of up to 1.46 Mpps => poor repeatibility. |\r
-| | | | Making mean values not fully representative. |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 8 | rls1801 substantial NDR/PDR regression for | CSIT-928 | NDR regression of -7%..-15%, PDR regression of -3%..-15% |\r
-| | IPSec tunnel scale with HW QAT crypto-dev | | compared to rls1710. |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
+Release Notes
+=============
+
+Changes in |csit-release|
+-------------------------
+
+#. VPP PERFORMANCE TESTS
+
+ - **Service density 2n-skx tests**: Added higher NF density tests with two
+ NFs' data-plane threads sharing a physical core. VPP IPv4 routing is now
+ used as a VNF payload similar to CNF tests.
+
+ - **Soak Tests**: Optimized performamce soak tests framework
+ code for extended time duration tests and throughput discovery
+ at given PLR and at give total test time e.g. minutes, hours,
+ days, weeks. See updated
+ :ref:`test_methodology` section for more details.
+
+#. TEST FRAMEWORK
+
+ - **Qemu code refactor**: Complete code refactor of the key components of
+ QemuUtil.py and QemuManager.py (L1 and L2 KW counterparts). Added
+ implementation of kernel-image-kvm based VM replacing the previously used
+ NestedVM images. Added ability to run VPP as a payload in VNF.
+
+ - **CSIT PAPI Support**: Continued conversion of CSIT VAT L1 keywords to
+ PAPI L1 KWs in CSIT using VPP Python bindings. Redesign of key components
+ of PAPI Executor and PAPI history.
+
+ - **General Code Housekeeping**: Ongoing RF keywords optimizations,
+ removal of redundant RF keywords.
+
+ - **Test suite generator**: Added capability to generate suites for
+ different NIC models as well as throughput search algorithm types. Uses
+ base tests suites as source.
+
+ - **TOX verification**: Added verifications for test suite generator.
+
+#. PRESENTATION AND ANALYTICS LAYER
+
+ - **Graphs Layout Improvements**: Improved performance graphs layout
+ for better readibility and maintenance: test grouping, axis
+ labels, descriptions, other informative decoration.
+
+..
+ #. MISCELLANEOUS
+
+ - **2n-dnv Tests (3rd Party)**: Published performance tests for 2n-
+ dnv (2-Node Atom Denverton) from 3rd party testbeds running FD.io
+ |csit-release| automated testing code.
+ Only graphs for Packet Throughput and Speedup Multi-core and not
+ for Packet Latency were published as there are no results for Packet
+ Latency available.
+
+.. raw:: latex
+
+ \clearpage
+
+.. _vpp_known_issues:
+
+Known Issues
+------------
+
+List of known issues in |csit-release| for VPP performance tests:
+
++----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
+| # | JiraID | Issue Description |
++====+=========================================+==========================================================================================================+
+| 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. |
+| | <https://jira.fd.io/browse/CSIT-570>`_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. |
++----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
+| 2 | `CSIT-1503 | [`TRex-519 <https://trex-tgn.cisco.com/youtrack/issue/trex-519>`_] XL710/XXV710 with FW 6.0.1 will have |
+| | <https://jira.fd.io/browse/CSIT-1503>`_ | Rx drop rate of 27MPPS. |
++----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
+| 3 | `CSIT-1501 | Sporadic crypto backend fails loading `VPP-1670 <https://jira.fd.io/browse/VPP-1670>`_ |
+| | <https://jira.fd.io/browse/CSIT-1501>`_ | |
++----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
+| 4 | `CSIT-1427 | Sporadic HW aes-128-cbc-sha1 tunnel-interface tests are failing. |
+| | <https://jira.fd.io/browse/CSIT-1427>`_ | `VPP-1671 <https://jira.fd.io/browse/VPP-1671>`_ |
++----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
+| 5 | `CSIT-1498 | Memif tests are sporadically failing on initialization of memif connection. |
+| | <https://jira.fd.io/browse/CSIT-1498>`_ | |
++----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
+| 6 | `CSIT-1499 | AVF tests are sporadically failing on initialization of AVF interface. |
+| | <https://jira.fd.io/browse/CSIT-1499>`_ | |
++----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
+| 7 | `VPP-1676 | 9000B ip4 memif errors - ip4-input: ip4 length > l2 length. |
+| | <https://jira.fd.io/browse/VPP-1676>`_ | IP4 jumbo frames (9000B) are dropped in case of tests with memif. |
++----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
+| 8 | `VPP-1677 | 9000B ip4 nat44: VPP crash + coredump. |
+| | <https://jira.fd.io/browse/VPP-1677>`_ | VPP crashes very often in case that NAT44 is configured and it has to process IP4 jumbo frames (9000B). |
++----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+
Code Review / csit.git / blobdiff