7c17e0f44802dfa56edd4bbf29b8e93eea9ed73a
[csit.git] / docs / report / vpp_performance_tests / csit_release_notes.rst
1 CSIT Release Notes\r
2 ==================\r
3 \r
4 Changes in CSIT |release|\r
5 -------------------------\r
6 \r
7 #. VPP performance test environment changes\r
8 \r
9     - Further optimizations of VM and vhost-user test environment - Qemu virtio\r
10       queue size increased from default value of 256 to 1024.\r
11     - Addition of HW cryptodev devices - Intel QAT 8950 50G - in all three\r
12       LF FD.io physical testbeds.\r
13 \r
14 #. VPP performance test framework changes\r
15 \r
16     - Added VAT command history collection for every test case as part of teardown.\r
17 \r
18 #. Added VPP performance tests\r
19 \r
20     - **CGNAT**\r
21 \r
22       - Carrier Grade Network Address Translation tests with varying number\r
23         of users and ports per user: 1u-15p, 10u-15p, 100u-15p, 1000u-15p,\r
24         2000u-15p, 4000u-15p - with Intel x520 NIC.\r
25 \r
26     - **vhost-user tests with one VM**\r
27 \r
28       - L2 Bridge Domain switched-forwarding with Intel x710 NIC, Intel x520 NIC,\r
29         Intel xl710 NIC.\r
30       - VXLAN and L2 Bridge Domain switched-forwarding with Intel x520 NIC.\r
31 \r
32     - **vhost-user tests with two VMs service chain**\r
33 \r
34       - L2 cross-connect switched-forwarding with Intel x520 NIC, Intel xl710 NIC.\r
35       - L2 Bridge Domain switched-forwarding with Intel x520 NIC, Intel xl710 NIC.\r
36       - IPv4 routed-forwarding with Intel x520 NIC, Intel xl710 NIC.\r
37 \r
38     - **IPSec encryption with**\r
39 \r
40       - AES-GCM, CBC-SHA1 ciphers, in combination with IPv4 routed-forwarding\r
41         with Intel xl710 NIC.\r
42       - CBC-SHA1 ciphers, in combination with LISP-GPE overlay tunneling for\r
43         IPv4-over-IPv4 with Intel xl710 NIC.\r
44 \r
45 Performance Improvements\r
46 ------------------------\r
47 \r
48 Some performance improvements in measured packet throughput have been observed\r
49 in a number of CSIT |release| tests listed below. Relative improvements are\r
50 calculated against the test results in CSIT rls1701 report. VPP-16.09 and\r
51 VPP-17.01 numbers are provided for reference.\r
52 \r
53 NDR Throughput\r
54 ~~~~~~~~~~~~~~\r
55 \r
56 Non-Drop Rate Throughput discovery tests:\r
57 \r
58 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
59 | VPP Functionality | Test Name                                                       | VPP-16.09  | VPP-17.01 | VPP-17.04 | 17.01 to 17.04  |\r
60 |                   |                                                                 |   [Mpps]   |  [Mpps]   |   [Mpps]  | Relative Change |\r
61 +===================+=================================================================+============+===========+===========+=================+\r
62 | L2XC              | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-ndrdisc                      | 9.4        | 12.7      | 13.4      | 6%              |\r
63 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
64 | L2XC              | 10ge2p1xl710: 64B-1t1c-eth-l2xcbase-ndrdisc                     | 9.5        | 12.2      | 12.4      | 2%              |\r
65 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
66 | L2XC dot1ad       | 10ge2p1x520: 64B-1t1c-dot1ad-l2xcbase-ndrdisc                   | 7.4        | 8.8       | 9.3       | 6%              |\r
67 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
68 | L2XC dot1q        | 10ge2p1x520: 64B-1t1c-dot1q-l2xcbase-ndrdisc                    | 7.5        | 8.8       | 9.2       | 5%              |\r
69 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
70 | L2XC VxLAN        | 10ge2p1x520: 64B-1t1c-ethip4vxlan-l2xcbase-ndrdisc              | 5.4        | 6.5       | 6.8       | 5%              |\r
71 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
72 | L2XC-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-eth-2vhost-1vm-ndrdisc       | 0.5        | 2.8       | 3.2       | 14%             |\r
73 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
74 | L2BD              | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-ndrdisc                | 7.8        | 10.4      | 10.8      | 4%              |\r
75 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
76 | L2BD-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc | 0.4        | 2.7       | 3.4       | 26%             |\r
77 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
78 | IPv4              | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ndrdisc                    | 8.7        | 9.7       | 10.6      | 9%              |\r
79 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
80 | IPv4 COP          | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-ndrdisc     | 7.1        | 8.3       | 9.0       | 8%              |\r
81 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
82 | IPv4 iAcl         | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-iacldstbase-ndrdisc        | 6.9        | 7.6       | 8.3       | 9%              |\r
83 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
84 | IPv4 vhost        | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-ndrdisc     | 0.3        | 2.6       | 3.1       | 19%             |\r
85 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
86 | IPv6              | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-ndrdisc                    | 3.0        | 7.3       | 8.1       | 11%             |\r
87 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+-----------------+\r
88 \r
89 PDR Throughput\r
90 ~~~~~~~~~~~~~~\r
91 \r
92 Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
93 \r
94 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
95 | VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01 | VPP-17.04 | 17.01 to 17.04  |\r
96 |                   |                                                                 |   [Mpps]  |  [Mpps]   |   [Mpps]  | Relative Change |\r
97 +===================+=================================================================+===========+===========+===========+=================+\r
98 | L2XC              | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-pdrdisc                      | 9.4       | 12.7      | 13.4      | 6%              |\r
99 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
100 | L2XC dot1ad       | 10ge2p1x520: 64B-1t1c-dot1ad-l2xcbase-pdrdisc                   | 7.4       | 8.8       | 9.3       | 6%              |\r
101 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
102 | L2XC dot1q        | 10ge2p1x520: 64B-1t1c-dot1q-l2xcbase-pdrdisc                    | 7.5       | 8.8       | 9.5       | 8%              |\r
103 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
104 | L2XC VxLAN        | 10ge2p1x520: 64B-1t1c-ethip4vxlan-l2xcbase-pdrdisc              | 5.4       | 6.5       | 6.8       | 5%              |\r
105 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
106 | L2XC-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-eth-2vhost-1vm-pdrdisc       | 2.6       | 3.2       | 3.2       | 0%              |\r
107 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
108 | L2BD              | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-pdrdisc                | 7.8       | 10.6      | 11.1      | 5%              |\r
109 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
110 | L2BD-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-pdrdisc | 2.1       | 2.9       | 3.2       | 10%             |\r
111 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
112 | IPv4              | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-pdrdisc                    | 8.7       | 9.7       | 10.6      | 9%              |\r
113 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
114 | IPv4 COP          | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-pdrdisc     | 7.1       | 8.3       | 9.2       | 11%             |\r
115 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
116 | IPv4 iAcl         | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-iacldstbase-pdrdisc        | 7.1       | 7.6       | 8.3       | 9%              |\r
117 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
118 | IPv4 vhost        | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-pdrdisc     | 2.0       | 2.7       | 3.2       | 19%             |\r
119 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
120 | IPv6              | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-pdrdisc                    | 7.7       | 7.3       | 8.1       | 11%             |\r
121 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
122 \r
123 Measured improvements are in line with VPP code optimizations listed in\r
124 `VPP-17.04 release notes\r
125 <https://docs.fd.io/vpp/17.04/release_notes_1704.html>`_.\r
126 \r
127 Additionally, vhost-VM performance improvements are due to both VPP code\r
128 optimizations as well as due to the FD.io CSIT Linux KVM test environment\r
129 optimizations for vhost-VM tests - see section "2.1.7. Methodology: KVM VM\r
130 vhost".\r
131 \r
132 \r
133 Other Performance Changes\r
134 -------------------------\r
135 \r
136 Other changes in measured packet throughput, with either minor relative\r
137 increase or decrease, have been observed in a number of CSIT |release| tests\r
138 listed below. Relative changes are calculated against the test results in CSIT\r
139 rls1701 report.\r
140 \r
141 NDR Throughput\r
142 ~~~~~~~~~~~~~~\r
143 \r
144 Non-Drop Rate Throughput discovery tests:\r
145 \r
146 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
147 | VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01 | VPP-17.04 | 17.01 to 17.04  |\r
148 |                   |                                                                 |   [Mpps]  |  [Mpps]   |   [Mpps]  | Relative Change |\r
149 +===================+=================================================================+===========+===========+===========+=================+\r
150 | IPv4 FIB 200k     | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-ndrdisc               | 8.5       | 9.0       | 9.7       | 8%              |\r
151 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
152 | IPv4 FIB 20k      | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-ndrdisc                | 8.5       | 9.0       | 9.4       | 4%              |\r
153 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
154 | IPv4 FIB 2M       | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-ndrdisc                 | 8.5       | 7.8       | 8.1       | 4%              |\r
155 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
156 | IPv4 Policer      | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-ndrdisc    | 6.9       | 7.4       | 8.1       | 9%              |\r
157 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
158 | IPv4 LISP         | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-ndrdisc             | 4.4       | 4.8       | 5.5       | 15%             |\r
159 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
160 | IPv6 COP          | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-ndrdisc     | 6.1       | 6.1       | 6.9       | 13%             |\r
161 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
162 | IPv6 FIB 200k     | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-ndrdisc               | 6.5       | 5.3       | 5.3       | 0%              |\r
163 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
164 | IPv6 FIB 20k      | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-ndrdisc                | 6.9       | 6.5       | 6.9       | 6%              |\r
165 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
166 | IPv6 FIB 2M       | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-ndrdisc                 | 5.3       | 4.2       | 4.6       | 10%             |\r
167 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
168 | IPv6 iAcl         | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-ndrdisc        | 6.5       | 6.1       | 6.9       | 13%             |\r
169 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
170 \r
171 PDR Throughput\r
172 ~~~~~~~~~~~~~~\r
173 \r
174 Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
175 \r
176 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
177 | VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01 | VPP-17.04 | 17.01 to 17.04  |\r
178 |                   |                                                                 |   [Mpps]  |  [Mpps]   |   [Mpps]  | Relative Change |\r
179 +===================+=================================================================+===========+===========+===========+=================+\r
180 | IPv4 FIB 200k     | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-pdrdisc               | 8.5       | 9.0       | 9.7       | 8%              |\r
181 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
182 | IPv4 FIB 20k      | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-pdrdisc                | 8.5       | 9.0       | 9.7       | 8%              |\r
183 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
184 | IPv4 FIB 2M       | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-pdrdisc                 | 8.3       | 8.1       | 8.3       | 2%              |\r
185 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
186 | IPv4 Policer      | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-pdrdisc    | 7.1       | 7.4       | 8.1       | 9%              |\r
187 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
188 | IPv4 LISP         | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-pdrdisc             | 4.6       | 4.8       | 5.5       | 15%             |\r
189 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
190 | IPv6 COP          | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-pdrdisc     | 6.1       | 6.1       | 6.9       | 13%             |\r
191 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
192 | IPv6 FIB 200k     | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-pdrdisc               | 6.9       | 5.3       | 5.3       | 0%              |\r
193 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
194 | IPv6 FIB 20k      | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-pdrdisc                | 6.9       | 6.5       | 6.9       | 6%              |\r
195 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
196 | IPv6 FIB 2M       | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-pdrdisc                 | 5.3       | 4.2       | 4.6       | 10%             |\r
197 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
198 | IPv6 iAcl         | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-pdrdisc        | 6.5       | 6.1       | 6.9       | 13%             |\r
199 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-----------------+\r
200 \r
201 Known Issues\r
202 ------------\r
203 \r
204 Here is the list of known issues in CSIT |release| for VPP performance tests:\r
205 \r
206 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
207 | # | Issue                                           | Jira ID    | Description                                                     |\r
208 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
209 | 1 | NDR discovery test failures 1518B frame size    | VPP-663    | VPP reporting errors: dpdk-input Rx ip checksum errors.         |\r
210 |   | for ip4scale200k, ip4scale2m scale IPv4 routed- |            | Observed frequency: all test runs.                              |\r
211 |   | forwarding tests. ip4scale20k tests are fine.   |            |                                                                 |\r
212 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
213 | 2 | VAT API timeouts during ip6scale2m scale IPv6   | VPP-712    | Needs fixing VPP VAT API timeouts for large volume of IPv6      |\r
214 |   | routed-forwarding tests when volume adding IPv6 |            | routes.                                                         |\r
215 |   | routes - 2M in this case. ip6scale2kk works.    |            |                                                                 |\r
216 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
217 | 3 | Vic1385 and Vic1227 low performance             | VPP-664    | Low NDR performance.                                            |\r
218 |   |                                                 |            |                                       .                         |\r
219 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
220 | 4 | Sporadic NDR discovery test failures on x520    | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |\r
221 |   |                                                 |            | infrastructure. Issue can't be replicated outside LF.           |\r
222 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
223 | 5 | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |\r
224 |   | of discovered NDR throughput values across      |            | throughput. Applies to XL710 and X710 NICs, x520 NICs are fine. |\r
225 |   | multiple test runs with xl710 and x710 NICs.    |            |                                       .                         |\r
226 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
227 | 6 | Lower than expected NDR and PDR throughput with | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |\r
228 |   | xl710 and x710 NICs, compared to x520 NICs.     |            | PDR throughput. Applies to XL710 and X710 NICs.                 |\r
229 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
230 \r

©2016 FD.io a Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation. Linux is a registered trademark of Linus Torvalds.
Please see our privacy policy and terms of use.