Changes in CSIT |release|\r
-------------------------\r
\r
-#. Added VPP performance tests\r
+#. **Added VPP performance tests**\r
\r
- - **Container Service Chain Topologies Orchestrated by K8s with VPP Memif**\r
+ - *MRR tests :* New MRR tests measure the packet forwarding rate\r
+ under the maximum load offered by traffic generator over a set\r
+ trial duration, regardless of packet loss. Maximum load for\r
+ specified Ethernet frame size is set to the bi-directional link\r
+ rate. MRR tests are used for continuous performance trending and\r
+ for comparison between releases.\r
\r
- - Added tests with VPP vswitch in container connecting a number of VPP-\r
- in-container service chain topologies with L2 Cross-Connect and L2\r
- Bridge-Domain configurations, orchestrated by Kubernetes. Added\r
- following forwarding topologies: i) "Parallel" with packets flowing from\r
- NIC via VPP to container and back to VPP and NIC; ii) "Chained" (a.k.a.\r
- "Snake") with packets flowing via VPP to container, back to VPP, to next\r
- container, back to VPP and so on until the last container in a chain,\r
- then back to VPP and NIC; iii) "Horizontal" with packets flowing via VPP\r
- to container, then via "horizontal" memif to next container, and so on\r
- until the last container, then back to VPP and NIC;\r
+ - *Service Chaining with SRv6 :* SRv6 (Segment Routing IPv6) proxy tests\r
+ verifying performance of Endpoint to SR-unaware appliance via\r
+ masquerading (End.AM), dynamic proxy (End.AD) or static proxy (End.AS)\r
+ functions.\r
\r
- - **VPP TCP/IP stack**\r
+#. **Presentation and Analytics Layer (PAL)**\r
\r
- - Added tests for VPP TCP/IP stack using VPP built-in HTTP server.\r
- WRK traffic generator is used as a client-side;\r
+ - Added continuous performance measuring, trending and anomaly\r
+ detection. Includes new PAL code and Jenkins jobs for Performance\r
+ Trending (PT) and Performance Analysis (PA) producing performance\r
+ trending dashboard and trendline graphs with summary and drill-\r
+ down views across all specified tests that can be reviewed and\r
+ inspected regularly by FD.io developers and users community.\r
\r
- - **SRv6**\r
+#. **Test Framework Optimizations**\r
\r
- - Initial SRv6 (Segment Routing IPv6) tests verifying performance of\r
- IPv6 and SRH (Segment Routing Header) encapsulation, decapsulation,\r
- lookups and rewrites based on configured End and End.DX6 SRv6 egress\r
- functions;\r
+ - *Performance tests efficiency :* Qemu build/install\r
+ optimizations, warmup phase handling, vpp restart handling.\r
+ Resulted in improved stability and reduced total execution time by\r
+ 30% for single pkt size e.g. 64B/78B.\r
\r
- - **IPSecSW**\r
-\r
- - SW computed IPSec encryption with AES-GCM, CBC-SHA1 ciphers, in\r
- combination with IPv4 routed-forwarding;\r
-\r
-#. Presentation and Analytics Layer\r
-\r
- - Added throughput speedup analysis for multi-core and multi-thread\r
- VPP tests into Presentation and Analytics Layer (PAL) for automated\r
- CSIT test results analysis;\r
-\r
-#. Other changes\r
-\r
- - **Framework optimizations**\r
-\r
- - Ability to run CSIT framework on ARM architecture;\r
-\r
- - Overall stability improvements;\r
-\r
- - **NDR and PDR throughput binary search change**\r
-\r
- - Increased binary search resolution by reducing final step from\r
- 100kpps to 50kpps;\r
-\r
- - **VPP plugin loaded as needed by tests**\r
-\r
- - From this release only plugins required by tests are loaded at\r
- VPP initialization time. Previously all plugins were loaded for\r
- all tests;\r
+ - *General code housekeeping :* ongoing RF keywords\r
+ optimizations, removal of redundant RF keywords.\r
\r
Performance Changes\r
-------------------\r
- `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-pdr-1t1c-full.txt>`_,\r
- `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-pdr-2t2c-full.txt>`_.\r
\r
-Measured improvements are in line with VPP code optimizations listed in\r
-`VPP-18.01 release notes\r
-<https://docs.fd.io/vpp/18.01/release_notes_1801.html>`_.\r
+MRR Throughput Changes\r
+~~~~~~~~~~~~~~~~~~~~~~\r
+\r
+MRR changes between releases are available in a CSV and\r
+pretty ASCII formats:\r
+\r
+ - `csv format for 1t1c <../_static/vpp/performance-changes-mrr-1t1c-full.csv>`_,\r
+ - `csv format for 2t2c <../_static/vpp/performance-changes-mrr-2t2c-full.csv>`_,\r
+ - `csv format for 4t4c <../_static/vpp/performance-changes-mrr-4t4c-full.csv>`_,\r
+ - `pretty ASCII format for 1t1c <../_static/vpp/performance-changes-mrr-1t1c-full.txt>`_,\r
+ - `pretty ASCII format for 2t2c <../_static/vpp/performance-changes-mrr-2t2c-full.txt>`_,\r
+ - `pretty ASCII format for 4t4c <../_static/vpp/performance-changes-mrr-4t4c-full.txt>`_.\r
+\r
+Throughput Trending\r
+-------------------\r
+\r
+In addition to reporting throughput changes between VPP releases, CSIT provides\r
+continuous performance trending for VPP master branch:\r
+\r
+#. `VPP Performance Dashboard <https://docs.fd.io/csit/master/trending/introduction/index.html>`_\r
+- per VPP test case throughput trend, trend compliance and summary of detected\r
+anomalies.\r
+\r
+#. `Trending Methodology <https://docs.fd.io/csit/master/trending/methodology/index.html>`_\r
+- throughput test metrics, trend calculations and anomaly classification\r
+(progression, regression, outlier).\r
+\r
+#. `Trendline Graphs <https://docs.fd.io/csit/master/trending/trending/index.html>`_\r
+- per VPP build MRR throughput measurements against the trendline with anomaly\r
+highlights, with associated CSIT test jobs.\r
\r
Known Issues\r
------------\r
+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
| # | Issue | Jira ID | Description |\r
+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 1 | Vic1385 and Vic1227 low performance. | VPP-664 | Low NDR performance. |\r
-| | | | |\r
-+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 2 | Sporadic (1 in 200) NDR discovery test failures | CSIT-570 | DPDK reporting rx-errors, indicating L1 issue. Suspected issue |\r
+| 1 | Sporadic (1 in 200) NDR discovery test failures | CSIT-570 | DPDK reporting rx-errors, indicating L1 issue. Suspected issue |\r
| | on x520. | | with HW combination of X710-X520 in LF testbeds. Not observed |\r
| | | | outside of LF testbeds. |\r
+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 3 | Lower than expected NDR throughput with | CSIT-571 | Suspected NIC firmware or DPDK driver issue affecting NDR and |\r
-| | xl710 and x710 NICs, compared to x520 NICs. | | PDR throughput. Applies to XL710 and X710 NICs. |\r
+| 2 | Lower than expected NDR throughput of DPDK | CSIT-571 | Suspected NIC firmware or DPDK driver issue affecting NDR and |\r
+| | testpmd and VPP L2 path NDR throughput with | | PDR throughput on XL710 and X710 NICs. |\r
+| | xl710 and x710 NICs, compared to x520 NICs. | | |\r
+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 4 | QAT IPSec scale with 1000 tunnels (interfaces) | VPP-1121 | VPP crashes during configuration of 1000 IPsec tunnels. |\r
-| | in 2t2c config, all tests are failing. | | 1t1c tests are not affected |\r
+| 3 | Tagged Ethernet dot1q and dot1ad L2 path | CSIT-1066 | Tagged Ethernet dot1q and dot1ad L2 path throughput regression: |\r
+| | throughput regression. | | NDR -2%..-5%, PDR -2%..-6%, MRR. Affects l2xc and l2bd |\r
+| | | | performance tests. |\r
++---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
+| 4 | IPSec (software, no QAT HW) throughput | CSIT-1064 | IPSec throughput regression: NDR -3%..-8%, PDR -2%..-8%, MRR |\r
+| | regression. | | -3%..-7%. |\r
++---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
+| 5 | High failure rate of creating working container | CSIT-1065 | Orchestrated container topology tests failing data plane |\r
+| | topologies with K8s/Ligato orchestration. | | verification indicating configuration issue. Suspected issue |\r
+| | | | with Ligato vpp-agent. |\r
+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-\r
Code Review / csit.git / blobdiff