docs/report/vpp_performance_tests/csit_release_notes.rst

   1 CSIT Release Notes\r
   2 ==================\r
   3 \r
   4 Changes in CSIT |release|\r
   5 -------------------------\r
   6 \r
   7 #. Naming change for all VPP performance test suites and test cases.\r
   8 #. VPP performance test environment changes\r
   9 \r
  10     - upgrade to Ubuntu 16.04\r
  11     - VM and vhost-user test environment optimizations\r
  12     - addition of HW cryptodev devices in LFD FD.io physical testbed\r
  13 \r
  14 #. Added tests\r
  15 \r
  16     - NICs\r
  17 \r
  18       - Intel x710 (L2 Bridge Domain switched-forwarding, vhost-user)\r
  19       - Cisco VIC1385 (L2 Bridge Domain switched-forwarding)\r
  20       - Cisco VIC1227 (L2 Bridge Domain switched-forwarding)\r
  21 \r
  22     - vhost-user tests with VM\r
  23 \r
  24       - L2 Bridge Domain switched-forwarding with Intel x710 NIC\r
  25       - L2 Bridge Domain switched-forwarding with VxLAN and Intel x520 NIC\r
  26       - L2 Bridge Domain switched-forwarding with Intel xl710 NIC\r
  27 \r
  28     - Tests with VxLAN\r
  29 \r
  30       - L2 Bridge Domain switched-forwarding with Intel x520 NIC\r
  31       - L2 Bridge Domain switched-forwarding with vhost-user, VM and Intel x520\r
  32         NIC\r
  33 \r
  34 Performance Improvements\r
  35 ------------------------\r
  36 \r
  37 Substantial improvements in measured packet throughput, with relative increase\r
  38 of double-digit percentage points, have been observed in a number of CSIT\r
  39 |release| tests listed below. Relative improvements are calculated against the\r
  40 test results in CSIT rls1609 report.\r
  41 \r
  42 NDR Throughput\r
  43 ~~~~~~~~~~~~~~\r
  44 \r
  45 Non-Drop Rate Throughput discovery tests:\r
  46 \r
  47 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  48 | VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01       | Relative Improvement |\r
  49 +===================+=================================================================+===========+=================+======================+\r
  50 | L2XC              | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-ndrdisc                      | 9.4 Mpps  | 12.7 Mpps       | 35%                  |\r
  51 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  52 | L2XC              | 10ge2p1xl710: 64B-1t1c-eth-l2xcbase-ndrdisc                     | 9.5 Mpps  | 12.2..12.4 Mpps | 28..30%              |\r
  53 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  54 | L2XC dot1ad       | 10ge2p1x520: 64B-1t1c-dot1ad-l2xcbase-ndrdisc                   | 7.4 Mpps  | 8.8..9.0 Mpps   | 19..23%              |\r
  55 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  56 | L2XC dot1q        | 10ge2p1x520: 64B-1t1c-dot1q-l2xcbase-ndrdisc                    | 7.5 Mpps  | 8.8..9.0 Mpps   | 17..20%              |\r
  57 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  58 | L2XC VxLAN        | 10ge2p1x520: 64B-1t1c-ethip4vxlan-l2xcbase-ndrdisc              | 5.4 Mpps  | 6.5 Mpps        | 20%                  |\r
  59 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  60 | L2XC-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-eth-2vhost-1vm-ndrdisc       | 0.5 Mpps  | 2.8..3.2 Mpps   | 460..540%            |\r
  61 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  62 | L2BD              | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-ndrdisc                | 7.8 Mpps  | 10.4..10.6 Mpps | 33..36%              |\r
  63 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  64 | L2BD-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc | 0.4 Mpps  | 2.7..2.8 Mpps   | 575..600%            |\r
  65 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  66 | IPv4              | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ndrdisc                    | 8.7 Mpps  | 9.7 Mpps        | 12%                  |\r
  67 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  68 | IPv4 COP          | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-ndrdisc     | 7.1 Mpps  | 8.3..8.5 Mpps   | 17..20%              |\r
  69 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  70 | IPv4 iAcl         | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-iacldstbase-ndrdisc        | 6.9 Mpps  | 7.6..7.8 Mpps   | 10..13%              |\r
  71 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  72 | IPv4 vhost        | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-ndrdisc     | 0.3 Mpps  | 2.6 Mpps        | 767%                 |\r
  73 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  74 | IPv6              | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-ndrdisc                    | 3.0 Mpps  | 7.3..7.7 Mpps   | 143..157%            |\r
  75 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  76 \r
  77 PDR Throughput\r
  78 ~~~~~~~~~~~~~~\r
  79 \r
  80 Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
  81 \r
  82 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  83 | VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01       | Relative Improvement |\r
  84 +===================+=================================================================+===========+=================+======================+\r
  85 | L2XC              | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-pdrdisc                      | 9.4 Mpps  | 12.7..12.9 Mpps | 35..37%              |\r
  86 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  87 | L2XC dot1ad       | 10ge2p1x520: 64B-1t1c-dot1ad-l2xcbase-pdrdisc                   | 7.4 Mpps  | 8.8..9.1 Mpps   | 19..23%              |\r
  88 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  89 | L2XC dot1q        | 10ge2p1x520: 64B-1t1c-dot1q-l2xcbase-pdrdisc                    | 7.5 Mpps  | 8.8..9.0 Mpps   | 17..20%              |\r
  90 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  91 | L2XC VxLAN        | 10ge2p1x520: 64B-1t1c-ethip4vxlan-l2xcbase-pdrdisc              | 5.4 Mpps  | 6.5 Mpps        | 20%                  |\r
  92 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  93 | L2XC-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-eth-2vhost-1vm-pdrdisc       | 2.6 Mpps  | 3.2..3.3 Mpps   | 23..26%              |\r
  94 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  95 | L2BD              | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-pdrdisc                | 7.8 Mpps  | 10.6 Mpps       | 36%                  |\r
  96 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  97 | L2BD-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-pdrdisc | 2.1 Mpps  | 2.9 Mpps        | 38%                  |\r
  98 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
  99 | IPv4              | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-pdrdisc                    | 8.7 Mpps  | 9.7 Mpps        | 11%                  |\r
 100 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 101 | IPv4 COP          | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-pdrdisc     | 7.1 Mpps  | 8.3..8.5 Mpps   | 17..20%              |\r
 102 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 103 | IPv4 vhost        | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-pdrdisc     | 2.0 Mpps  | 2.7 Mpps        | 35%                  |\r
 104 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 105 \r
 106 Measured improvements are in line with VPP code optimizations listed in\r
 107 `VPP-17.01 release notes\r
 108 <https://docs.fd.io/vpp/17.01/release_notes_1701.html>`_.\r
 109 \r
 110 Additionally, vhost-VM performance improvements are due to both VPP code\r
 111 optimizations as well as due to the FD.io CSIT Linux KVM test environment\r
 112 optimizations for vhost-VM tests - see section "2.1.7. Methodology: KVM VM\r
 113 vhost".\r
 114 \r
 115 \r
 116 Other Performance Changes\r
 117 -------------------------\r
 118 \r
 119 Other changes in measured packet throughput, with either minor relative\r
 120 increase or decrease, have been observed in a number of CSIT |release| tests\r
 121 listed below. Relative changes are calculated against the test results in CSIT\r
 122 rls1609 report.\r
 123 \r
 124 NDR Throughput\r
 125 ~~~~~~~~~~~~~~\r
 126 \r
 127 Non-Drop Rate Throughput discovery tests:\r
 128 \r
 129 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 130 | VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01       | Relative Change      |\r
 131 +===================+=================================================================+===========+=================+======================+\r
 132 | IPv4 FIB 200k     | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-ndrdisc               | 8.5 Mpps  | 9.0 Mpps        | 6%                   |\r
 133 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 134 | IPv4 FIB 20k      | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-ndrdisc                | 8.5 Mpps  | 9.0..9.2 Mpps   | 6..8%                |\r
 135 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 136 | IPv4 FIB 2M       | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-ndrdisc                 | 8.5 Mpps  | 7.8..8.1 Mpps   | -8..-5%              |\r
 137 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 138 | IPv4 Policer      | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-ndrdisc    | 6.9 Mpps  | 7.4..7.6 Mpps   | 7..10%               |\r
 139 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 140 | IPv4 LISP         | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-ndrdisc             | 4.4 Mpps  | 4.8 Mpps        | 9%                   |\r
 141 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 142 | IPv6 COP          | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-ndrdisc     | 6.1 Mpps  | 6.1..6.5 Mpps   | 0..7%                |\r
 143 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 144 | IPv6 FIB 200k     | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-ndrdisc               | 6.5 Mpps  | 5.3..5.7 Mpps   | -18..-12%            |\r
 145 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 146 | IPv6 FIB 20k      | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-ndrdisc                | 6.9 Mpps  | 6.5 Mpps        | -6%                  |\r
 147 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 148 | IPv6 FIB 2M       | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-ndrdisc                 | 5.3 Mpps  | 4.2 Mpps        | -21%                 |\r
 149 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 150 | IPv6 iAcl         | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-ndrdisc        | 6.5 Mpps  | 6.1..6.5 Mpps   | -6..0%               |\r
 151 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 152 \r
 153 PDR Throughput\r
 154 ~~~~~~~~~~~~~~\r
 155 \r
 156 Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
 157 \r
 158 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 159 | VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01       | Relative Change      |\r
 160 +===================+=================================================================+===========+=================+======================+\r
 161 | IPv4 FIB 200k     | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-pdrdisc               | 8.5 Mpps  | 9.0 Mpps        | 6%                   |\r
 162 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 163 | IPv4 FIB 20k      | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-pdrdisc                | 8.5 Mpps  | 9.0..9.2 Mpps   | 6..8%                |\r
 164 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 165 | IPv4 FIB 2M       | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-pdrdisc                 | 8.3 Mpps  | 8.1 Mpps        | -2%                  |\r
 166 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 167 | IPv4 iAcl         | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-iacldstbase-pdrdisc        | 7.1 Mpps  | 7.6..7.8 Mpps   | 7..10%               |\r
 168 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 169 | IPv4 Policer      | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-pdrdisc    | 7.1 Mpps  | 7.4..7.6 Mpps   | 4..7%                |\r
 170 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 171 | IPv4 LISP         | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-pdrdisc             | 4.6 Mpps  | 4.8 Mpps        | 9%                   |\r
 172 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 173 | IPv6              | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-pdrdisc                    | 7.7 Mpps  | 7.3..7.7 Mpps   | -5..0%               |\r
 174 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 175 | IPv6 COP          | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-pdrdisc     | 6.1 Mpps  | 6.1..6.5 Mpps   | 0..7%                |\r
 176 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 177 | IPv6 FIB 200k     | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-pdrdisc               | 6.9 Mpps  | 5.3..5.7 Mpps   | -23..-17%            |\r
 178 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 179 | IPv6 FIB 20k      | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-pdrdisc                | 6.9 Mpps  | 6.5 Mpps        | -6%                  |\r
 180 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 181 | IPv6 FIB 2M       | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-pdrdisc                 | 5.3 Mpps  | 4.2 Mpps        | -21%                 |\r
 182 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 183 | IPv6 iAcl         | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-pdrdisc        | 6.5 Mpps  | 6.1..6.5 Mpps   | -6..0%               |\r
 184 +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+\r
 185 \r
 186 Known Issues\r
 187 ------------\r
 188 \r
 189 Here is the list of known issues in CSIT |release| for VPP performance tests:\r
 190 \r
 191 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
 192 | # | Issue                                           | Description                                                     |\r
 193 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
 194 | 1 | Sporadic IPv4 routed-forwarding NDR discovery   | Suspected issue with DPDK IPv4 checksum calculation.            |\r
 195 |   | test failures for 1518B frame size              | Observed frequency: sporadic, ca. 20% to 30% of test runs.      |\r
 196 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
 197 | 2 | Vic1385 and Vic1227 NICs jumbo frames test      | Suspected issue with VIC DPDK drivers that do not support jumbo |\r
 198 |   | failures (9000B)                                | frames (dropped rx-miss). Observed frequency: 100%.             |\r
 199 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
 200 | 3 | Vic1385 and Vic1227 performance                 | Low performance of NDR results. Big difference between NDR and  |\r
 201 |   |                                                 | PDR.                                  .                         |\r
 202 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
 203 | 4 | Sporadic NDR discovery test failures on x520    | Suspected issue with HW settings (BIOS, FW) in LF               |\r
 204 |   |                                                 | infrastructure. Issue can't be replicated outside LF.           |\r
 205 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
 206 | 5 | Non-repeatible zig-zagging NDR throughput       | Suspected NIC driver issue in multi-thread multi-core operation |\r
 207 |   | results for multi-thread multi-core tests       | affectin NDR throughput. Applies to XL710 and X710 NICs tested. |\r
 208 |   | - 2t2c, 4t4c - for some tested NICs.            |                                                                 |\r
 209 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
 210 \r
 211 Tests to be Added - Cryptodev\r
 212 -----------------------------\r
 213 \r
 214 DPDK Cryptodev functionality support for both SW and HW crypto devices has\r
 215 been introduced in VPP-17.01 release. CSIT functional and performance tests\r
 216 have been also developed and merged. However due to the factors beyond CSIT\r
 217 project control execution of those tests within the LF FD.io test environment\r
 218 still need to complete. Once the results become available, they will be\r
 219 published as an addendum to the current version of CSIT |release| report.\r
 220 \r
 221 Tests to be Added - SNAT44\r
 222 --------------------------\r
 223 \r
 224 VPP SNAT44 functionality has been introduced in VPP-17.01 release. CSIT\r
 225 performance tests are still in development and integration into LFD FD.io test\r
 226 environment. Once the tests are fully integrated and results become available,\r
 227 they will be published as an addendum to the current version of CSIT |release|\r
 228 report.\r