rls1707 report: edits of vpp_performance_tests/rls_notes
[csit.git] / docs / report / vpp_performance_tests / csit_release_notes.rst
index ff992df..782fa42 100644 (file)
@@ -4,97 +4,115 @@ CSIT Release Notes
 Changes in CSIT |release|\r
 -------------------------\r
 \r
-#. VPP performance test environment changes\r
+#. Test environment changes in VPP data plane performance tests:\r
 \r
-   - Further optimizations of VM and vhost-user test environment - various\r
-     Qemu virtio queue size testing with value of 256 and 1024. Applied\r
-     Linux CFS optimization to run VPP worker threads and Qemu worker threads\r
-     with highest priority.\r
+   - Further characterization and optimizations of VPP vhost-user and VM\r
+     test methodology and test environment;\r
 \r
-#. VPP performance test framework changes\r
+     - Tests with varying Qemu virtio queue (a.k.a. vring) sizes:\r
+       [vr256] default 256 descriptors, [vr1024] 1024 descriptors to\r
+       optimize for packet throughput;\r
 \r
-   - Full code review, optimization and refactor.\r
+     - Tests with varying Linux CFS (Completely Fair Scheduler)\r
+       settings: [cfs] default settings, [cfsrr1] CFS RoundRobin(1)\r
+       policy applied to all data plane threads handling test packet\r
+       path including all VPP worker threads and all Qemu testpmd\r
+       poll-mode threads;\r
 \r
-#. T-rex changes\r
+     - Resulting test cases are all combinations with [vr256,vr1024] and\r
+       [cfs,cfsrr1] settings;\r
 \r
-   - Full refactor of T-rex driver and introduce of traffic profiles that\r
-     improves readability, manageability of traffic profiles for various\r
-     test scenarios.\r
+     - For more detail see performance results observations section in\r
+       this report;\r
+\r
+#. Code updates and optimizations in CSIT performance framework:\r
+\r
+   - Complete CSIT framework code revision and optimizations as descried\r
+     on CSIT wiki page\r
+     `Design_Optimizations <https://wiki.fd.io/view/CSIT/Design_Optimizations>`_.\r
+\r
+   - For more detail see the CSIT Framework Design section in this\r
+     report;\r
+\r
+#. Changes to CSIT driver for TRex Traffic Generator:\r
+\r
+   - Complete refactor of TRex CSIT driver;\r
+\r
+   - Introduction of packet traffic profiles to improve usability and\r
+     manageability of traffic profiles for a growing number of test\r
+     scenarios.\r
+\r
+   - Support for packet traffic profiles to test IPv4/IPv6 stateful and\r
+     stateless DUT data plane features;\r
 \r
 #. Added VPP performance tests\r
 \r
-   - **LXC memif**\r
+   - **Linux Container VPP memif virtual interface tests**\r
 \r
-     - Memif interface tests interconnecting two VPP instances on single SUT.\r
-       Master VPP instance running on native OS with Intel x520 NIC and guest\r
-       VPP instance running in Linux Container (LXC) doing the L2 cross\r
-       connect loop. LXC running in privileged mode is pinned to dedicated\r
-       cores. All VPP instances are same version.\r
+     - VPP Memif virtual interface (shared memory interface) tests\r
+       interconnecting VPP instances over memif. VPP vswitch\r
+       instance runs in bare-metal user-mode handling Intel x520 NIC\r
+       10GbE interfaces and connecting over memif (Master side) virtual\r
+       interfaces to another instance of VPP running in bare-metal Linux\r
+       Container (LXC) with memif virtual interfaces (Slave side). LXC\r
+       runs in a priviliged mode with VPP data plane worker threads\r
+       pinned to dedicated physical CPU cores per usual CSIT practice.\r
+       Both VPP run the same version of software. This test topology is\r
+       equivalent to existing tests with vhost-user and VMs.\r
 \r
    - **Stateful Security Groups**\r
 \r
-   - **VM vhost use cases**\r
+     - New tests of VPP stateful security-groups a.k.a. acl-plugin\r
+       functionally compatible with networking-vpp OpenStack;\r
+\r
+     - New tested security-groups access-control-lists (acl)\r
+       configuration variants include: [iaclNsl] input acl stateless,\r
+       [oaclNsl] output acl stateless, [iaclNsf] input acl stateful\r
+       a.k.a. reflect, [oaclNsf] output acl stateful a.k.a. reflect,\r
+       where N is number of access-control-entries (ace) in the acl.\r
+\r
+     - Testing packet flows transmitted by TG: 100, 10k, 100k, always\r
+       hitting the last permit entry in acl.\r
+\r
+   - **VPP vhost and VM tests**\r
+\r
+     - New VPP vhost-user and VM test cases to benchmark performance of\r
+       VPP and VM topologies with Qemu and CFS policy combinations of\r
+       [vr256,vr1024] x [cfs,cfsrr1];\r
+\r
+     - Statistical analysis of repeatibility of results;\r
 \r
 Performance Improvements\r
 ------------------------\r
 \r
 Substantial improvements in measured packet throughput have been\r
 observed in a number of CSIT |release| tests listed below, with relative\r
-increase  of double-digit percentage points. Relative improvements are\r
-calculated against the test results listed in CSIT |release-1| report.\r
-VPP-16.09 numbers are provided for reference.\r
+increase  of double-digit percentage points. Relative improvements for\r
+this release are calculated against the test results listed in CSIT\r
+|release-1| report. The comparison is calculated between the mean values\r
+based on collected and archived test results' samples for involved VPP\r
+releases. Standard deviation has been also listed for CSIT |release|.\r
+VPP-16.09 and VPP-17.01 numbers are provided for reference.\r
 \r
 NDR Throughput\r
 ~~~~~~~~~~~~~~\r
 \r
 Non-Drop Rate Throughput discovery tests:\r
 \r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01 | VPP-17.04 | VPP-17.07 | 17.04 to 17.07  |\r
-|                   |                                                                 | [Mpps]    | [Mpps]    | [Mpps]    | [Mpps]    | Relative Change |\r
-+===================+=================================================================+===========+===========+===========+===========+=================+\r
-| L2XC-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-eth-2vhost-1vm-ndrdisc       | 0.5       | 2.8       | 3.4       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| L2BD-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc | 0.4       | 2.7       | 3.1       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 vhost        | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-ndrdisc     | 0.3       | 2.6       | 3.0       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 LISP         | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-ndrdisc             | 4.4       | 4.8       | 5.5       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv6              | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-ndrdisc                    | 3.0       | 7.3       | 8.1       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv6 COP          | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-ndrdisc     | 6.1       | 6.1       | 6.9       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv6 iAcl         | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-ndrdisc        | 6.5       | 6.1       | 6.9       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv6 FIB 2M       | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-ndrdisc                 | 5.3       | 4.2       | 4.6       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
+.. csv-table::\r
+    :align: center\r
+    :header-rows: 1\r
+    :file: ../../../docs/report/vpp_performance_tests/performance_improvements/ndr_throughput.csv\r
 \r
 PDR Throughput\r
 ~~~~~~~~~~~~~~\r
 \r
 Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
 \r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01 | VPP-17.04 | VPP-17.07 | 17.04 to 17.07  |\r
-|                   |                                                                 | [Mpps]    | [Mpps]    | [Mpps]    | [Mpps]    | Relative Change |\r
-+===================+=================================================================+===========+===========+===========+===========+=================+\r
-| L2XC-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-eth-2vhost-1vm-pdrdisc       | 2.6       | 3.2       | 3.7       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| L2BD-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-pdrdisc | 2.1       | 2.9       | 3.3       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 vhost        | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-pdrdisc     | 2.0       | 2.7       | 3.0       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 LISP         | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-pdrdisc             | 4.6       | 4.8       | 5.5       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv6              | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-pdrdisc                    | 7.7       | 7.3       | 8.1       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv6 COP          | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-pdrdisc     | 6.1       | 6.1       | 6.9       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv6 iAcl         | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-pdrdisc        | 6.5       | 6.1       | 6.9       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv6 FIB 2M       | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-pdrdisc                 | 5.3       | 4.2       | 4.6       |           | ??%             |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
+.. csv-table::\r
+    :align: center\r
+    :header-rows: 1\r
+    :file: ../../../docs/report/vpp_performance_tests/performance_improvements/pdr_throughput.csv\r
 \r
 Measured improvements are in line with VPP code optimizations listed in\r
 `VPP-17.07 release notes\r
@@ -113,78 +131,21 @@ NDR Throughput
 \r
 Non-Drop Rate Throughput discovery tests:\r
 \r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01 | VPP-17.04 | VPP-17.07 | 17.04 to 17.07  |\r
-|                   |                                                                 | [Mpps]    | [Mpps]    | [Mpps]    | [Mpps]    | Relative Change |\r
-+===================+=================================================================+===========+===========+===========+===========+=================+\r
-| L2XC              | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-ndrdisc                      | 9.4       | 12.7      | 13.1      |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| L2XC              | 10ge2p1xl710: 64B-1t1c-eth-l2xcbase-ndrdisc                     | 9.5       | 12.2      | 12.4      |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| L2XC dot1ad       | 10ge2p1x520: 64B-1t1c-dot1ad-l2xcbase-ndrdisc                   | 7.4       | 8.8       | 9.3       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| L2XC dot1q        | 10ge2p1x520: 64B-1t1c-dot1q-l2xcbase-ndrdisc                    | 7.5       | 8.8       | 9.2       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| L2XC VxLAN        | 10ge2p1x520: 64B-1t1c-ethip4vxlan-l2xcbase-ndrdisc              | 5.4       | 6.5       | 6.8       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| L2BD              | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-ndrdisc                | 7.8       | 10.4      | 10.8      |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4              | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ndrdisc                    | 8.7       | 9.7       | 10.6      |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 COP          | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-ndrdisc     | 7.1       | 8.3       | 9.0       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 iAcl         | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-iacldstbase-ndrdisc        | 6.9       | 7.6       | 8.3       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 FIB 200k     | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-ndrdisc               | 8.5       | 9.0       | 9.7       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 FIB 20k      | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-ndrdisc                | 8.5       | 9.0       | 9.7       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 FIB 2M       | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-ndrdisc                 | 8.5       | 7.8       | 8.1       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 Policer      | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-ndrdisc    | 6.9       | 7.4       | 8.1       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv6 FIB 200k     | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-ndrdisc               | 6.5       | 5.3       | 5.3       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv6 FIB 20k      | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-ndrdisc                | 6.9       | 6.5       | 6.9       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
+.. csv-table::\r
+    :align: center\r
+    :header-rows: 1\r
+    :file: ../../../docs/report/vpp_performance_tests/performance_improvements/ndr_throughput_others.csv\r
 \r
 PDR Throughput\r
 ~~~~~~~~~~~~~~\r
 \r
 Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
 \r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01 | VPP-17.04 | VPP-17.07 | 17.04 to 17.07  |\r
-|                   |                                                                 | [Mpps]    | [Mpps]    | [Mpps]    | [Mpps]    | Relative Change |\r
-+===================+=================================================================+===========+===========+===========+===========+=================+\r
-| L2XC              | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-pdrdisc                      | 9.4       | 12.7      | 13.4      |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| L2XC dot1ad       | 10ge2p1x520: 64B-1t1c-dot1ad-l2xcbase-pdrdisc                   | 7.4       | 8.8       | 9.3       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| L2XC dot1q        | 10ge2p1x520: 64B-1t1c-dot1q-l2xcbase-pdrdisc                    | 7.5       | 8.8       | 9.2       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| L2XC VxLAN        | 10ge2p1x520: 64B-1t1c-ethip4vxlan-l2xcbase-pdrdisc              | 5.4       | 6.5       | 6.8       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| L2BD              | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-pdrdisc                | 7.8       | 10.6      | 10.8      |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4              | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-pdrdisc                    | 8.7       | 9.7       | 10.6      |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 COP          | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-pdrdisc     | 7.1       | 8.3       | 9.2       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 iAcl         | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-iacldstbase-pdrdisc        | 7.1       | 7.6       | 8.3       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 FIB 200k     | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-pdrdisc               | 8.5       | 9.0       | 9.7       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 FIB 20k      | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-pdrdisc                | 8.5       | 9.0       | 9.7       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 FIB 2M       | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-pdrdisc                 | 8.3       | 8.1       | 8.1       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv4 Policer      | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-pdrdisc    | 7.1       | 7.4       | 8.1       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv6 FIB 200k     | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-pdrdisc               | 6.9       | 5.3       | 5.3       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
-| IPv6 FIB 20k      | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-pdrdisc                | 6.9       | 6.5       | 6.9       |           | ?%              |\r
-+-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------+-----------------+\r
+.. csv-table::\r
+    :align: center\r
+    :header-rows: 1\r
+    :file: ../../../docs/report/vpp_performance_tests/performance_improvements/pdr_throughput_others.csv\r
+\r
 \r
 Known Issues\r
 ------------\r
@@ -194,21 +155,27 @@ Here is the list of known issues in CSIT |release| for VPP performance tests:
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 | # | Issue                                           | Jira ID    | Description                                                     |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 1 | NDR discovery test failures 1518B frame size    | VPP-663    | VPP reporting errors: dpdk-input Rx ip checksum errors.         |\r
+| 1 | Security-groups acl-plugin scale tests failure  | CSIT-xxx   | VPP with 2 worker threads crashes during security-groups        |\r
+|   | with stateful acls if VPP with 2 worker threads | VPP-912    | iaclNsf and oaclNsf tests with 100k flows.                      |\r
++---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
+| 2 | VPP fails memif tests in 4 worker 2 core setup  | CSIT-xxx   | VPP with 4 worker threads running on 2 physical cores crashes   |\r
+|   |                                                 | VPP-xxx    | during memif tests. Initial debugging points to DPDK code       |\r
++---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
+| X | NDR discovery test failures 1518B frame size    | VPP-663    | VPP reporting errors: dpdk-input Rx ip checksum errors.         |\r
 |   | for ip4scale200k, ip4scale2m scale IPv4 routed- |            | Observed frequency: all test runs.                              |\r
 |   | forwarding tests. ip4scale20k tests are fine.   |            |                                                                 |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 2 | Vic1385 and Vic1227 low performance.            | VPP-664    | Low NDR performance.                                            |\r
+| X | Vic1385 and Vic1227 low performance.            | VPP-664    | Low NDR performance.                                            |\r
 |   |                                                 |            |                                                                 |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 3 | Sporadic NDR discovery test failures on x520.   | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |\r
+| X | Sporadic NDR discovery test failures on x520.   | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |\r
 |   |                                                 |            | infrastructure. Issue can't be replicated outside LF.           |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 4 | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |\r
+| X | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |\r
 |   | of discovered NDR throughput values across      |            | throughput. Applies to XL710 and X710 NICs, x520 NICs are fine. |\r
 |   | multiple test runs with xl710 and x710 NICs.    |            |                                                                 |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
-| 5 | Lower than expected NDR and PDR throughput with | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |\r
+| X | Lower than expected NDR and PDR throughput with | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |\r
 |   | xl710 and x710 NICs, compared to x520 NICs.     |            | PDR throughput. Applies to XL710 and X710 NICs.                 |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 \r