docs/report/vpp_performance_tests/csit_release_notes.rst

   1 CSIT Release Notes\r
   2 ==================\r
   3 \r
   4 Changes in CSIT |release|\r
   5 -------------------------\r
   6 \r
   7 #. VPP performance test environment changes\r
   8 \r
   9     - Further optimizations of VM and vhost-user test environment - Qemu virtio\r
  10       queue size increased from default value of 256 to 1024.\r
  11     - Addition of HW cryptodev devices - Intel QAT 8950 50G - in all three\r
  12       LF FD.io physical testbeds.\r
  13 \r
  14 #. VPP performance test framework changes\r
  15 \r
  16     - Added VAT command history collection for every test case as part of teardown.\r
  17 \r
  18 #. Added VPP performance tests\r
  19 \r
  20     - **CGNAT**\r
  21 \r
  22       - Carrier Grade Network Address Translation tests with varying number\r
  23         of users and ports per user: 1u-15p, 10u-15p, 100u-15p, 1000u-15p,\r
  24         2000u-15p, 4000u-15p - with Intel x520 NIC.\r
  25 \r
  26     - **vhost-user tests with one VM**\r
  27 \r
  28       - L2 Bridge Domain switched-forwarding with Intel x710 NIC, Intel x520 NIC,\r
  29         Intel xl710 NIC.\r
  30       - VXLAN and L2 Bridge Domain switched-forwarding with Intel x520 NIC.\r
  31 \r
  32     - **vhost-user tests with two VMs service chain**\r
  33 \r
  34       - L2 cross-connect switched-forwarding with Intel x520 NIC, Intel xl710 NIC.\r
  35       - L2 Bridge Domain switched-forwarding with Intel x520 NIC, Intel xl710 NIC.\r
  36       - IPv4 routed-forwarding with Intel x520 NIC, Intel xl710 NIC.\r
  37 \r
  38     - **IPSec encryption with**\r
  39 \r
  40       - AES-GCM, CBC-SHA1 ciphers, in combination with IPv4 routed-forwarding\r
  41         with Intel xl710 NIC.\r
  42       - CBC-SHA1 ciphers, in combination with LISP-GPE overlay tunneling for\r
  43         IPv4-over-IPv4 with Intel xl710 NIC.\r
  44 \r
  45 Performance Improvements\r
  46 ------------------------\r
  47 \r
  48 Substantial improvements in measured packet throughput have been\r
  49 observed in a number of CSIT |release| tests listed below, with relative\r
  50 increase  of double-digit percentage points. Relative improvements are\r
  51 calculated against the test results listed in CSIT |release-1| report.\r
  52 VPP-16.09 numbers are provided for reference.\r
  53 \r
  54 NDR Throughput\r
  55 ~~~~~~~~~~~~~~\r
  56 \r
  57 Non-Drop Rate Throughput discovery tests:\r
  58 \r
  59 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
  60 | VPP Functionality | Test Name                                                       | VPP-16.09  | VPP-17.01 | VPP-17.04 | 17.01 to 17.04  |\r
  61 |                   |                                                                 | [Mpps]     | [Mpps]    | [Mpps]    | Relative Change |\r
  62 +===================+=================================================================+============+===========+===========+=================+\r
  63 | L2XC-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-eth-2vhost-1vm-ndrdisc       | 0.5        | 2.8       | 3.4       | 21%             |\r
  64 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
  65 | L2BD-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc | 0.4        | 2.7       | 3.1       | 15%             |\r
  66 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
  67 | IPv4 vhost        | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-ndrdisc     | 0.3        | 2.6       | 3.0       | 15%             |\r
  68 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
  69 | IPv4 LISP         | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-ndrdisc             | 4.4        | 4.8       | 5.5       | 15%             |\r
  70 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
  71 | IPv6              | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-ndrdisc                    | 3.0        | 7.3       | 8.1       | 11%             |\r
  72 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
  73 | IPv6 COP          | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-ndrdisc     | 6.1        | 6.1       | 6.9       | 13%             |\r
  74 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
  75 | IPv6 iAcl         | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-ndrdisc        | 6.5        | 6.1       | 6.9       | 13%             |\r
  76 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
  77 | IPv6 FIB 2M       | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-ndrdisc                 | 5.3        | 4.2       | 4.6       | 10%             |\r
  78 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
  79 \r
  80 PDR Throughput\r
  81 ~~~~~~~~~~~~~~\r
  82 \r
  83 Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
  84 \r
  85 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
  86 | VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01 | VPP-17.04 | 17.01 to 17.04  |\r
  87 |                   |                                                                 | [Mpps]    | [Mpps]    | [Mpps]    | Relative Change |\r
  88 +===================+=================================================================+===========+===========+===========+=================+\r
  89 | L2XC-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-eth-2vhost-1vm-pdrdisc       | 2.6       | 3.2       | 3.7       | 15%             |\r
  90 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
  91 | L2BD-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-pdrdisc | 2.1       | 2.9       | 3.3       | 14%             |\r
  92 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
  93 | IPv4 vhost        | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-pdrdisc     | 2.0       | 2.7       | 3.0       | 11%             |\r
  94 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
  95 | IPv4 LISP         | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-pdrdisc             | 4.6       | 4.8       | 5.5       | 15%             |\r
  96 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
  97 | IPv6              | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-pdrdisc                    | 7.7       | 7.3       | 8.1       | 11%             |\r
  98 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
  99 | IPv6 COP          | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-pdrdisc     | 6.1       | 6.1       | 6.9       | 13%             |\r
 100 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 101 | IPv6 iAcl         | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-pdrdisc        | 6.5       | 6.1       | 6.9       | 13%             |\r
 102 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 103 | IPv6 FIB 2M       | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-pdrdisc                 | 5.3       | 4.2       | 4.6       | 10%             |\r
 104 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 105 \r
 106 Measured improvements are in line with VPP code optimizations listed in\r
 107 `VPP-17.04 release notes\r
 108 <https://docs.fd.io/vpp/17.04/release_notes_1704.html>`_.\r
 109 \r
 110 Additionally, vhost-VM performance improvements are due to both VPP code\r
 111 optimizations as well as due to the FD.io CSIT Linux KVM test environment\r
 112 optimizations for vhost-VM tests - see section "2.1.7. Methodology: KVM VM\r
 113 vhost".\r
 114 \r
 115 \r
 116 Other Performance Changes\r
 117 -------------------------\r
 118 \r
 119 Other changes in measured packet throughput, with either minor relative\r
 120 increase or decrease, have been observed in a number of CSIT |release|\r
 121 tests listed below. Relative changes are calculated against the test\r
 122 results listed in CSIT |release-1| report.\r
 123 \r
 124 NDR Throughput\r
 125 ~~~~~~~~~~~~~~\r
 126 \r
 127 Non-Drop Rate Throughput discovery tests:\r
 128 \r
 129 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 130 | VPP Functionality | Test Name                                                       | VPP-16.09  | VPP-17.01 | VPP-17.04 | 17.01 to 17.04  |\r
 131 |                   |                                                                 | [Mpps]     | [Mpps]    | [Mpps]    | Relative Change |\r
 132 +===================+=================================================================+============+===========+===========+=================+\r
 133 | L2XC              | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-ndrdisc                      | 9.4        | 12.7      | 13.1      | 3%              |\r
 134 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 135 | L2XC              | 10ge2p1xl710: 64B-1t1c-eth-l2xcbase-ndrdisc                     | 9.5        | 12.2      | 12.4      | 2%              |\r
 136 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 137 | L2XC dot1ad       | 10ge2p1x520: 64B-1t1c-dot1ad-l2xcbase-ndrdisc                   | 7.4        | 8.8       | 9.3       | 6%              |\r
 138 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 139 | L2XC dot1q        | 10ge2p1x520: 64B-1t1c-dot1q-l2xcbase-ndrdisc                    | 7.5        | 8.8       | 9.2       | 5%              |\r
 140 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 141 | L2XC VxLAN        | 10ge2p1x520: 64B-1t1c-ethip4vxlan-l2xcbase-ndrdisc              | 5.4        | 6.5       | 6.8       | 5%              |\r
 142 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 143 | L2BD              | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-ndrdisc                | 7.8        | 10.4      | 10.8      | 4%              |\r
 144 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 145 | IPv4              | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ndrdisc                    | 8.7        | 9.7       | 10.6      | 9%              |\r
 146 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 147 | IPv4 COP          | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-ndrdisc     | 7.1        | 8.3       | 9.0       | 8%              |\r
 148 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 149 | IPv4 iAcl         | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-iacldstbase-ndrdisc        | 6.9        | 7.6       | 8.3       | 9%              |\r
 150 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 151 | IPv4 FIB 200k     | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-ndrdisc               | 8.5        | 9.0       | 9.7       | 8%              |\r
 152 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 153 | IPv4 FIB 20k      | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-ndrdisc                | 8.5        | 9.0       | 9.7       | 8%              |\r
 154 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 155 | IPv4 FIB 2M       | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-ndrdisc                 | 8.5        | 7.8       | 8.1       | 4%              |\r
 156 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 157 | IPv4 Policer      | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-ndrdisc    | 6.9        | 7.4       | 8.1       | 9%              |\r
 158 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 159 | IPv6 FIB 200k     | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-ndrdisc               | 6.5        | 5.3       | 5.3       | 0%              |\r
 160 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 161 | IPv6 FIB 20k      | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-ndrdisc                | 6.9        | 6.5       | 6.9       | 6%              |\r
 162 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
 163 \r
 164 PDR Throughput\r
 165 ~~~~~~~~~~~~~~\r
 166 \r
 167 Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
 168 \r
 169 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 170 | VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01 | VPP-17.04 | 17.01 to 17.04  |\r
 171 |                   |                                                                 | [Mpps]    | [Mpps]    | [Mpps]    | Relative Change |\r
 172 +===================+=================================================================+===========+===========+===========+=================+\r
 173 | L2XC              | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-pdrdisc                      | 9.4       | 12.7      | 13.4      | 6%              |\r
 174 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 175 | L2XC dot1ad       | 10ge2p1x520: 64B-1t1c-dot1ad-l2xcbase-pdrdisc                   | 7.4       | 8.8       | 9.3       | 6%              |\r
 176 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 177 | L2XC dot1q        | 10ge2p1x520: 64B-1t1c-dot1q-l2xcbase-pdrdisc                    | 7.5       | 8.8       | 9.2       | 5%              |\r
 178 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 179 | L2XC VxLAN        | 10ge2p1x520: 64B-1t1c-ethip4vxlan-l2xcbase-pdrdisc              | 5.4       | 6.5       | 6.8       | 5%              |\r
 180 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 181 | L2BD              | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-pdrdisc                | 7.8       | 10.6      | 10.8      | 2%              |\r
 182 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 183 | IPv4              | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-pdrdisc                    | 8.7       | 9.7       | 10.6      | 9%              |\r
 184 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 185 | IPv4 COP          | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-pdrdisc     | 7.1       | 8.3       | 9.2       | 11%             |\r
 186 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 187 | IPv4 iAcl         | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-iacldstbase-pdrdisc        | 7.1       | 7.6       | 8.3       | 9%              |\r
 188 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 189 | IPv4 FIB 200k     | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-pdrdisc               | 8.5       | 9.0       | 9.7       | 8%              |\r
 190 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 191 | IPv4 FIB 20k      | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-pdrdisc                | 8.5       | 9.0       | 9.7       | 8%              |\r
 192 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 193 | IPv4 FIB 2M       | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-pdrdisc                 | 8.3       | 8.1       | 8.1       | 0%              |\r
 194 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 195 | IPv4 Policer      | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-pdrdisc    | 7.1       | 7.4       | 8.1       | 9%              |\r
 196 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 197 | IPv6 FIB 200k     | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-pdrdisc               | 6.9       | 5.3       | 5.3       | 0%              |\r
 198 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 199 | IPv6 FIB 20k      | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-pdrdisc                | 6.9       | 6.5       | 6.9       | 6%              |\r
 200 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
 201 \r
 202 Known Issues\r
 203 ------------\r
 204 \r
 205 Here is the list of known issues in CSIT |release| for VPP performance tests:\r
 206 \r
 207 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 208 | # | Issue                                           | Jira ID    | Description                                                     |\r
 209 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 210 | 1 | NDR discovery test failures 1518B frame size    | VPP-663    | VPP reporting errors: dpdk-input Rx ip checksum errors.         |\r
 211 |   | for ip4scale200k, ip4scale2m scale IPv4 routed- |            | Observed frequency: all test runs.                              |\r
 212 |   | forwarding tests. ip4scale20k tests are fine.   |            |                                                                 |\r
 213 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 214 | 2 | Sporadic VAT API timeouts during ip6scale2m     | VPP-712    | Needs fixing VPP VAT API timeouts for large volume of IPv6      |\r
 215 |   | scale IPv6  routed-forwarding tests when volume |            | routes.                                                         |\r
 216 |   | adding IPv6 routes - 2M in this case.           |            |                                                                 |\r
 217 |   | ip6scale200k works.                             |            |                                                                 |\r
 218 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 219 | 3 | Vic1385 and Vic1227 low performance.            | VPP-664    | Low NDR performance.                                            |\r
 220 |   |                                                 |            |                                                                 |\r
 221 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 222 | 4 | Sporadic NDR discovery test failures on x520.   | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |\r
 223 |   |                                                 |            | infrastructure. Issue can't be replicated outside LF.           |\r
 224 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 225 | 5 | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |\r
 226 |   | of discovered NDR throughput values across      |            | throughput. Applies to XL710 and X710 NICs, x520 NICs are fine. |\r
 227 |   | multiple test runs with xl710 and x710 NICs.    |            |                                                                 |\r
 228 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 229 | 6 | Lower than expected NDR and PDR throughput with | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |\r
 230 |   | xl710 and x710 NICs, compared to x520 NICs.     |            | PDR throughput. Applies to XL710 and X710 NICs.                 |\r
 231 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 232 \r