Changes in CSIT |release|\r
-------------------------\r
\r
-#. VPP performance test environment changes\r
+#. Test environment changes in VPP data plane performance tests:\r
\r
- - Further optimizations of VM and vhost-user test environment - various\r
- Qemu virtio queue size testing with value of 256 and 1024. Applied\r
- Linux CFS optimization to run VPP worker threads and Qemu worker threads\r
- with highest priority.\r
+ - Further characterization and optimizations of VPP vhost-user and VM\r
+ test methodology and test environment;\r
\r
-#. VPP performance test framework changes\r
+ - Tests with varying Qemu virtio queue (a.k.a. vring) sizes:\r
+ [vr256] default 256 descriptors, [vr1024] 1024 descriptors to\r
+ optimize for packet throughput;\r
\r
- - Full code review, optimization and refactor.\r
+ - Tests with varying Linux CFS (Completely Fair Scheduler)\r
+ settings: [cfs] default settings, [cfsrr1] CFS RoundRobin(1)\r
+ policy applied to all data plane threads handling test packet\r
+ path including all VPP worker threads and all Qemu testpmd\r
+ poll-mode threads;\r
\r
-#. T-rex changes\r
+ - Resulting test cases are all combinations with [vr256,vr1024] and\r
+ [cfs,cfsrr1] settings;\r
\r
- - Full refactor of T-rex driver and introduce of traffic profiles that\r
- improves readability, manageability of traffic profiles for various\r
- test scenarios.\r
+ - For more detail see performance results observations section in\r
+ this report;\r
+\r
+#. Code updates and optimizations in CSIT performance framework:\r
+\r
+ - Complete CSIT framework code revision and optimizations as descried\r
+ on CSIT wiki page\r
+ `Design_Optimizations <https://wiki.fd.io/view/CSIT/Design_Optimizations>`_.\r
+\r
+ - For more detail see the CSIT Framework Design section in this\r
+ report;\r
+\r
+#. Changes to CSIT driver for TRex Traffic Generator:\r
+\r
+ - Complete refactor of TRex CSIT driver;\r
+\r
+ - Introduction of packet traffic profiles to improve usability and\r
+ manageability of traffic profiles for a growing number of test\r
+ scenarios.\r
+\r
+ - Support for packet traffic profiles to test IPv4/IPv6 stateful and\r
+ stateless DUT data plane features;\r
\r
#. Added VPP performance tests\r
\r
- - **LXC memif**\r
+ - **Linux Container VPP memif virtual interface tests**\r
\r
- - Memif interface tests interconnecting two VPP instances on single SUT.\r
- Master VPP instance running on native OS with Intel x520 NIC and guest\r
- VPP instance running in Linux Container (LXC) doing the L2 cross\r
- connect loop. LXC running in privileged mode is pinned to dedicated\r
- cores. All VPP instances are same version.\r
+ - VPP Memif virtual interface (shared memory interface) tests\r
+ interconnecting VPP instances over memif. VPP vswitch\r
+ instance runs in bare-metal user-mode handling Intel x520 NIC\r
+ 10GbE interfaces and connecting over memif (Master side) virtual\r
+ interfaces to another instance of VPP running in bare-metal Linux\r
+ Container (LXC) with memif virtual interfaces (Slave side). LXC\r
+ runs in a priviliged mode with VPP data plane worker threads\r
+ pinned to dedicated physical CPU cores per usual CSIT practice.\r
+ Both VPP run the same version of software. This test topology is\r
+ equivalent to existing tests with vhost-user and VMs.\r
\r
- **Stateful Security Groups**\r
\r
- - **VM vhost use cases**\r
+ - New tests of VPP stateful security-groups a.k.a. acl-plugin\r
+ functionally compatible with networking-vpp OpenStack;\r
+\r
+ - New tested security-groups access-control-lists (acl)\r
+ configuration variants include: [iaclNsl] input acl stateless,\r
+ [oaclNsl] output acl stateless, [iaclNsf] input acl stateful\r
+ a.k.a. reflect, [oaclNsf] output acl stateful a.k.a. reflect,\r
+ where N is number of access-control-entries (ace) in the acl.\r
+\r
+ - Testing packet flows transmitted by TG: 100, 10k, 100k, always\r
+ hitting the last permit entry in acl.\r
+\r
+ - **VPP vhost and VM tests**\r
+\r
+ - New VPP vhost-user and VM test cases to benchmark performance of\r
+ VPP and VM topologies with Qemu and CFS policy combinations of\r
+ [vr256,vr1024] x [cfs,cfsrr1];\r
+\r
+ - Statistical analysis of repeatibility of results;\r
\r
Performance Improvements\r
------------------------\r
\r
Substantial improvements in measured packet throughput have been\r
observed in a number of CSIT |release| tests listed below, with relative\r
-increase of double-digit percentage points. Relative improvements are\r
-calculated against the test results listed in CSIT |release-1| report.\r
-VPP-16.09 numbers are provided for reference.\r
+increase of double-digit percentage points. Relative improvements for\r
+this release are calculated against the test results listed in CSIT\r
+|release-1| report. The comparison is calculated between the mean values\r
+based on collected and archived test results' samples for involved VPP\r
+releases. Standard deviation has been also listed for CSIT |release|.\r
+VPP-16.09 and VPP-17.01 numbers are provided for reference.\r
\r
NDR Throughput\r
~~~~~~~~~~~~~~\r
+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
| # | Issue | Jira ID | Description |\r
+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 1 | NDR discovery test failures 1518B frame size | VPP-663 | VPP reporting errors: dpdk-input Rx ip checksum errors. |\r
+| 1 | Security-groups acl-plugin scale tests failure | CSIT-xxx | VPP with 2 worker threads crashes during security-groups |\r
+| | with stateful acls if VPP with 2 worker threads | VPP-912 | iaclNsf and oaclNsf tests with 100k flows. |\r
++---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
+| 2 | VPP fails memif tests in 4 worker 2 core setup | CSIT-xxx | VPP with 4 worker threads running on 2 physical cores crashes |\r
+| | | VPP-xxx | during memif tests. Initial debugging points to DPDK code |\r
++---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
+| X | NDR discovery test failures 1518B frame size | VPP-663 | VPP reporting errors: dpdk-input Rx ip checksum errors. |\r
| | for ip4scale200k, ip4scale2m scale IPv4 routed- | | Observed frequency: all test runs. |\r
| | forwarding tests. ip4scale20k tests are fine. | | |\r
+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 2 | Vic1385 and Vic1227 low performance. | VPP-664 | Low NDR performance. |\r
+| X | Vic1385 and Vic1227 low performance. | VPP-664 | Low NDR performance. |\r
| | | | |\r
+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 3 | Sporadic NDR discovery test failures on x520. | CSIT-750 | Suspected issue with HW settings (BIOS, FW) in LF |\r
+| X | Sporadic NDR discovery test failures on x520. | CSIT-750 | Suspected issue with HW settings (BIOS, FW) in LF |\r
| | | | infrastructure. Issue can't be replicated outside LF. |\r
+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 4 | VPP in 2t2c setups - large variation | CSIT-568 | Suspected NIC firmware or DPDK driver issue affecting NDR |\r
+| X | VPP in 2t2c setups - large variation | CSIT-568 | Suspected NIC firmware or DPDK driver issue affecting NDR |\r
| | of discovered NDR throughput values across | | throughput. Applies to XL710 and X710 NICs, x520 NICs are fine. |\r
| | multiple test runs with xl710 and x710 NICs. | | |\r
+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 5 | Lower than expected NDR and PDR throughput with | CSIT-569 | Suspected NIC firmware or DPDK driver issue affecting NDR and |\r
+| X | Lower than expected NDR and PDR throughput with | CSIT-569 | Suspected NIC firmware or DPDK driver issue affecting NDR and |\r
| | xl710 and x710 NICs, compared to x520 NICs. | | PDR throughput. Applies to XL710 and X710 NICs. |\r
+---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
\r
Code Review / csit.git / commitdiff