CSIT rls1704 report - update to handcrafted sections:
[csit.git] / docs / report / vpp_performance_tests / csit_release_notes.rst
1 CSIT Release Notes\r
2 ==================\r
3 \r
4 Changes in CSIT |release| - to be updated\r
5 -------------------------\r
6 \r
7 #. VPP performance test environment changes\r
8 \r
9     - further VM and vhost-user test environment optimizations - Qemu virtio queue size increased from default value of 256 to 1024\r
10     - addition of HW cryptodev devices in all three LF FD.io physical testbeds\r
11 \r
12 #. Added tests (to be updated)\r
13 \r
14     - NICs\r
15 \r
16       - Intel x710 (L2 Bridge Domain switched-forwarding, vhost-user)\r
17       - Cisco VIC1385 (L2 Bridge Domain switched-forwarding)\r
18       - Cisco VIC1227 (L2 Bridge Domain switched-forwarding)\r
19 \r
20     - vhost-user tests with VM\r
21 \r
22       - L2 Bridge Domain switched-forwarding with Intel x710 NIC\r
23       - L2 Bridge Domain switched-forwarding with VxLAN and Intel x520 NIC\r
24       - L2 Bridge Domain switched-forwarding with Intel xl710 NIC\r
25 \r
26     - Tests with VxLAN\r
27 \r
28       - L2 Bridge Domain switched-forwarding with Intel x520 NIC\r
29       - L2 Bridge Domain switched-forwarding with vhost-user, VM and Intel x520\r
30         NIC\r
31 \r
32 Performance Improvements - to be updated\r
33 ------------------------\r
34 \r
35 Substantial improvements in measured packet throughput, with relative increase\r
36 of double-digit percentage points, have been observed in a number of CSIT\r
37 |release| tests listed below. Relative improvements are calculated against the\r
38 test results in CSIT rls1701 report.\r
39 \r
40 NDR Throughput\r
41 ~~~~~~~~~~~~~~\r
42 \r
43 Non-Drop Rate Throughput discovery tests:\r
44 \r
45 +-------------------+-----------------------------------------------------------------+------------+-----------+----------------------------+\r
46 | VPP Functionality | Test Name                                                       | VPP-16.09  | VPP-17.01 | VPP-17.04 | 17.01 to 17.04 |\r
47 |                   |                                                                 |   [Mpps]   |  [Mpps]   |   [Mpps]  |  Improvement   |\r
48 +===================+=================================================================+============+===========+===========+================+\r
49 | L2XC              | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-ndrdisc                      | 9.4        | 12.7      | 13.4      | 6%             |\r
50 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
51 | L2XC              | 10ge2p1xl710: 64B-1t1c-eth-l2xcbase-ndrdisc                     | 9.5        | 12.2      | 12.4      | 2%             |\r
52 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
53 | L2XC dot1ad       | 10ge2p1x520: 64B-1t1c-dot1ad-l2xcbase-ndrdisc                   | 7.4        | 8.8       | 9.3       | 6%             |\r
54 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
55 | L2XC dot1q        | 10ge2p1x520: 64B-1t1c-dot1q-l2xcbase-ndrdisc                    | 7.5        | 8.8       | 9.2       | 5%             |\r
56 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
57 | L2XC VxLAN        | 10ge2p1x520: 64B-1t1c-ethip4vxlan-l2xcbase-ndrdisc              | 5.4        | 6.5       | 6.8       | 5%             |\r
58 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
59 | L2XC-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-eth-2vhost-1vm-ndrdisc       | 0.5        | 2.8       | 3.2       | 14%            |\r
60 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
61 | L2BD              | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-ndrdisc                | 7.8        | 10.4      | 10.8      | 4%             |\r
62 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
63 | L2BD-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc | 0.4        | 2.7       | 3.4       | 26%            |\r
64 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
65 | IPv4              | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ndrdisc                    | 8.7        | 9.7       | 10.6      | 9%             |\r
66 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
67 | IPv4 COP          | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-ndrdisc     | 7.1        | 8.3       | 9.0       | 8%             |\r
68 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
69 | IPv4 iAcl         | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-iacldstbase-ndrdisc        | 6.9        | 7.6       | 8.3       | 9%             |\r
70 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
71 | IPv4 vhost        | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-ndrdisc     | 0.3        | 2.6       | 3.1       | 19%            |\r
72 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
73 | IPv6              | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-ndrdisc                    | 3.0        | 7.3       | 8.5       | 16%            |\r
74 +-------------------+-----------------------------------------------------------------+------------+-----------+-----------+----------------+\r
75 \r
76 PDR Throughput\r
77 ~~~~~~~~~~~~~~\r
78 \r
79 Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
80 \r
81 +-------------------+-----------------------------------------------------------------+-----------+-----------+-------------------------+\r
82 | VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01 | VPP-17.04 | Relative    |\r
83 |                   |                                                                 |   [Mpps]  |  [Mpps]   |   [Mpps]  | Improvement |\r
84 +===================+=================================================================+===========+===========+===========+=============+\r
85 | L2XC              | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-pdrdisc                      | 9.4       | 12.7      | 13.4      |   %         |\r
86 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
87 | L2XC dot1ad       | 10ge2p1x520: 64B-1t1c-dot1ad-l2xcbase-pdrdisc                   | 7.4       | 8.8       | 9.3       |   %         |\r
88 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
89 | L2XC dot1q        | 10ge2p1x520: 64B-1t1c-dot1q-l2xcbase-pdrdisc                    | 7.5       | 8.8       | 9.5       |   %         |\r
90 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
91 | L2XC VxLAN        | 10ge2p1x520: 64B-1t1c-ethip4vxlan-l2xcbase-pdrdisc              | 5.4       | 6.5       | 6.8       |   %         |\r
92 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
93 | L2XC-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-eth-2vhost-1vm-pdrdisc       | 2.6       | 3.2       | 3.2       |   %         |\r
94 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
95 | L2BD              | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-pdrdisc                | 7.8       | 10.6      | 11.1      |   %         |\r
96 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
97 | L2BD-vhost-VM     | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-pdrdisc | 2.1       | 2.9       | 3.2       |   %         |\r
98 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
99 | IPv4              | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-pdrdisc                    | 8.7       | 9.7       | 10.6      |   %         |\r
100 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
101 | IPv4 COP          | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-pdrdisc     | 7.1       | 8.3       | 9.2       |   %         |\r
102 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
103 | IPv4 vhost        | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-pdrdisc     | 2.0       | 2.7       | 3.2       |   %         |\r
104 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
105 \r
106 Measured improvements are in line with VPP code optimizations listed in\r
107 `VPP-17.04 release notes\r
108 <https://docs.fd.io/vpp/17.04/release_notes_1704.html>`_.\r
109 \r
110 Additionally, vhost-VM performance improvements are due to both VPP code\r
111 optimizations as well as due to the FD.io CSIT Linux KVM test environment\r
112 optimizations for vhost-VM tests - see section "2.1.7. Methodology: KVM VM\r
113 vhost".\r
114 \r
115 \r
116 Other Performance Changes - to be updated\r
117 -------------------------\r
118 \r
119 Other changes in measured packet throughput, with either minor relative\r
120 increase or decrease, have been observed in a number of CSIT |release| tests\r
121 listed below. Relative changes are calculated against the test results in CSIT\r
122 rls1701 report.\r
123 \r
124 NDR Throughput\r
125 ~~~~~~~~~~~~~~\r
126 \r
127 Non-Drop Rate Throughput discovery tests:\r
128 \r
129 +-------------------+-----------------------------------------------------------------+-----------+-----------+-------------------------+\r
130 | VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01 | VPP-17.04 | Relative    |\r
131 |                   |                                                                 |   [Mpps]  |  [Mpps]   |   [Mpps]  | Change      |\r
132 +===================+=================================================================+===========+===========+===========+=============+\r
133 | IPv4 FIB 200k     | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-ndrdisc               | 8.5       | 9.0       |           |  %          |\r
134 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
135 | IPv4 FIB 20k      | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-ndrdisc                | 8.5       | 9.0       |           |  %          |\r
136 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
137 | IPv4 FIB 2M       | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-ndrdisc                 | 8.5       | 7.8       |           |  %          |\r
138 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
139 | IPv4 Policer      | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-ndrdisc    | 6.9       | 7.4       |           |  %          |\r
140 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
141 | IPv4 LISP         | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-ndrdisc             | 4.4       | 4.8       |           |  %          |\r
142 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
143 | IPv6 COP          | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-ndrdisc     | 6.1       | 6.1       |           |  %          |\r
144 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
145 | IPv6 FIB 200k     | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-ndrdisc               | 6.5       | 5.3       |           |  %          |\r
146 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
147 | IPv6 FIB 20k      | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-ndrdisc                | 6.9       | 6.5       |           |  %          |\r
148 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
149 | IPv6 FIB 2M       | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-ndrdisc                 | 5.3       | 4.2       |           |  %          |\r
150 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
151 | IPv6 iAcl         | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-ndrdisc        | 6.5       | 6.1       |           |  %          |\r
152 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
153 \r
154 PDR Throughput\r
155 ~~~~~~~~~~~~~~\r
156 \r
157 Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:\r
158 \r
159 +-------------------+-----------------------------------------------------------------+-----------+-----------+-------------------------+\r
160 | VPP Functionality | Test Name                                                       | VPP-16.09 | VPP-17.01 | VPP-17.04 | Relative    |\r
161 |                   |                                                                 |   [Mpps]  |  [Mpps]   |   [Mpps]  | Change      |\r
162 +===================+=================================================================+===========+===========+===========+=============+\r
163 | IPv4 FIB 200k     | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-pdrdisc               | 8.5       | 9.0       |           |  %          |\r
164 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
165 | IPv4 FIB 20k      | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-pdrdisc                | 8.5       | 9.0       |           |  %          |\r
166 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
167 | IPv4 FIB 2M       | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-pdrdisc                 | 8.3       | 8.1       |           |  %          |\r
168 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
169 | IPv4 iAcl         | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-iacldstbase-pdrdisc        | 7.1       | 7.6       |           |  %          |\r
170 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
171 | IPv4 Policer      | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-pdrdisc    | 7.1       | 7.4       |           |  %          |\r
172 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
173 | IPv4 LISP         | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-pdrdisc             | 4.6       | 4.8       |           |  %          |\r
174 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
175 | IPv6              | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-pdrdisc                    | 7.7       | 7.3       |           |  %          |\r
176 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
177 | IPv6 COP          | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-pdrdisc     | 6.1       | 6.1       |           |  %          |\r
178 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
179 | IPv6 FIB 200k     | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-pdrdisc               | 6.9       | 5.3       |           |  %          |\r
180 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
181 | IPv6 FIB 20k      | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-pdrdisc                | 6.9       | 6.5       |           |  %          |\r
182 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
183 | IPv6 FIB 2M       | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-pdrdisc                 | 5.3       | 4.2       |           |  %          |\r
184 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
185 | IPv6 iAcl         | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-pdrdisc        | 6.5       | 6.1       |           |  %          |\r
186 +-------------------+-----------------------------------------------------------------+-----------+-----------+-----------+-------------+\r
187 \r
188 Known Issues - to be updated\r
189 ------------\r
190 \r
191 Here is the list of known issues in CSIT |release| for VPP performance tests: (to be updated)\r
192 \r
193 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
194 | # | Issue                                           | Description                                                     |\r
195 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
196 | 1 | Sporadic IPv4 routed-forwarding NDR discovery   | Suspected issue with DPDK IPv4 checksum calculation.            |\r
197 |   | test failures for 1518B frame size              | Observed frequency: sporadic, ca. 20% to 30% of test runs.      |\r
198 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
199 | 3 | Vic1385 and Vic1227 performance                 | Low performance of NDR results. Big difference between NDR and  |\r
200 |   |                                                 | PDR.                                  .                         |\r
201 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
202 | 4 | Sporadic NDR discovery test failures on x520    | Suspected issue with HW settings (BIOS, FW) in LF               |\r
203 |   |                                                 | infrastructure. Issue can't be replicated outside LF.           |\r
204 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
205 | 5 | Non-repeatible zig-zagging NDR throughput       | Suspected NIC driver issue in multi-thread multi-core operation |\r
206 |   | results for multi-thread multi-core tests       | affectin NDR throughput. Applies to XL710 and X710 NICs tested. |\r
207 |   | - 2t2c, 4t4c - for some tested NICs.            |                                                                 |\r
208 +---+-------------------------------------------------+-----------------------------------------------------------------+\r
209 \r

©2016 FD.io a Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation. Linux is a registered trademark of Linus Torvalds.
Please see our privacy policy and terms of use.