2 .. _performance_test_methodology:
4 Performance Test Methodology
5 ============================
10 Packet and bandwidth throughput are measured in accordance with
11 :rfc:`2544`, using FD.io CSIT Multiple Loss Ratio search (MLRsearch), an
12 optimized binary search algorithm, that measures SUT/DUT throughput at
13 different Packet Loss Ratio (PLR) values.
15 Following MLRsearch values are measured across a range of L2 frame sizes
18 - NON DROP RATE (NDR): packet and bandwidth throughput at PLR=0%.
20 - **Aggregate packet rate**: NDR_LOWER <bi-directional packet rate>
22 - **Aggregate bandwidth rate**: NDR_LOWER <bi-directional bandwidth
25 - PARTIAL DROP RATE (PDR): packet and bandwidth throughput at PLR=0.5%.
27 - **Aggregate packet rate**: PDR_LOWER <bi-directional packet rate>
29 - **Aggregate bandwidth rate**: PDR_LOWER <bi-directional bandwidth
32 NDR and PDR are measured for the following L2 frame sizes (untagged
35 - IPv4 payload: 64B, IMIX_v4_1 (28x64B, 16x570B, 4x1518B), 1518B, 9000B.
36 - IPv6 payload: 78B, 1518B, 9000B.
38 All rates are reported from external Traffic Generator perspective.
40 .. _mlrsearch_algorithm:
45 Multiple Loss Rate search (MLRsearch) is a new search algorithm
46 implemented in FD.io CSIT project. MLRsearch discovers multiple packet
47 throughput rates in a single search, with each rate associated with a
48 distinct Packet Loss Ratio (PLR) criteria.
50 Two throughput measurements used in FD.io CSIT are Non-Drop Rate (NDR,
51 with zero packet loss, PLR=0) and Partial Drop Rate (PDR, with packet
52 loss rate not greater than the configured non-zero PLR). MLRsearch
53 discovers NDR and PDR in a single pass reducing required execution time
54 compared to separate binary searches for NDR and PDR. MLRsearch reduces
55 execution time even further by relying on shorter trial durations
56 of intermediate steps, with only the final measurements
57 conducted at the specified final trial duration.
58 This results in the shorter overall search
59 execution time when compared to a standard NDR/PDR binary search,
60 while guaranteeing the same or similar results.
62 If needed, MLRsearch can be easily adopted to discover more throughput rates
63 with different pre-defined PLRs.
65 .. Note:: All throughput rates are *always* bi-directional
66 aggregates of two equal (symmetric) uni-directional packet rates
67 received and reported by an external traffic generator.
72 The main properties of MLRsearch:
74 - MLRsearch is a duration aware multi-phase multi-rate search algorithm.
76 - Initial phase determines promising starting interval for the search.
77 - Intermediate phases progress towards defined final search criteria.
78 - Final phase executes measurements according to the final search
83 - Uses link rate as a starting transmit rate and discovers the Maximum
84 Receive Rate (MRR) used as an input to the first intermediate phase.
86 - *Intermediate phases*:
88 - Start with initial trial duration (in the first phase) and converge
89 geometrically towards the final trial duration (in the final phase).
90 - Track two values for NDR and two for PDR.
92 - The values are called (NDR or PDR) lower_bound and upper_bound.
93 - Each value comes from a specific trial measurement
94 (most recent for that transmit rate),
95 and as such the value is associated with that measurement's duration and loss.
96 - A bound can be invalid, for example if NDR lower_bound
97 has been measured with nonzero loss.
98 - Invalid bounds are not real boundaries for the searched value,
99 but are needed to track interval widths.
100 - Valid bounds are real boundaries for the searched value.
101 - Each non-initial phase ends with all bounds valid.
103 - Start with a large (lower_bound, upper_bound) interval width and
104 geometrically converge towards the width goal (measurement resolution)
105 of the phase. Each phase halves the previous width goal.
106 - Use internal and external searches:
108 - External search - measures at transmit rates outside the (lower_bound,
109 upper_bound) interval. Activated when a bound is invalid,
110 to search for a new valid bound by doubling the interval width.
111 It is a variant of `exponential search`_.
112 - Internal search - `binary search`_, measures at transmit rates within the
113 (lower_bound, upper_bound) valid interval, halving the interval width.
115 - *Final phase* is executed with the final test trial duration, and the final
116 width goal that determines resolution of the overall search.
117 Intermediate phases together with the final phase are called non-initial phases.
119 The main benefits of MLRsearch vs. binary search include:
121 - In general MLRsearch is likely to execute more search trials overall, but
122 less trials at a set final duration.
123 - In well behaving cases it greatly reduces (>50%) the overall duration
124 compared to a single PDR (or NDR) binary search duration,
125 while finding multiple drop rates.
126 - In all cases MLRsearch yields the same or similar results to binary search.
127 - Note: both binary search and MLRsearch are susceptible to reporting
128 non-repeatable results across multiple runs for very bad behaving
133 - Worst case MLRsearch can take longer than a binary search e.g. in case of
134 drastic changes in behaviour for trials at varying durations.
136 Search Implementation
137 ~~~~~~~~~~~~~~~~~~~~~
139 Following is a brief description of the current MLRsearch
140 implementation in FD.io CSIT.
145 #. *maximum_transmit_rate* - maximum packet transmit rate to be used by
146 external traffic generator, limited by either the actual Ethernet
147 link rate or traffic generator NIC model capabilities. Sample
148 defaults: 2 * 14.88 Mpps for 64B 10GE link rate,
149 2 * 18.75 Mpps for 64B 40GE NIC maximum rate.
150 #. *minimum_transmit_rate* - minimum packet transmit rate to be used for
151 measurements. MLRsearch fails if lower transmit rate needs to be
152 used to meet search criteria. Default: 2 * 10 kpps (could be higher).
153 #. *final_trial_duration* - required trial duration for final rate
154 measurements. Default: 30 sec.
155 #. *initial_trial_duration* - trial duration for initial MLRsearch phase.
157 #. *final_relative_width* - required measurement resolution expressed as
158 (lower_bound, upper_bound) interval width relative to upper_bound.
160 #. *packet_loss_ratio* - maximum acceptable PLR search criteria for
161 PDR measurements. Default: 0.5%.
162 #. *number_of_intermediate_phases* - number of phases between the initial
163 phase and the final phase. Impacts the overall MLRsearch duration.
164 Less phases are required for well behaving cases, more phases
165 may be needed to reduce the overall search duration for worse behaving cases.
166 Default (2). (Value chosen based on limited experimentation to date.
167 More experimentation needed to arrive to clearer guidelines.)
172 1. First trial measures at maximum rate and discovers MRR.
174 a. *in*: trial_duration = initial_trial_duration.
175 b. *in*: offered_transmit_rate = maximum_transmit_rate.
176 c. *do*: single trial.
177 d. *out*: measured loss ratio.
178 e. *out*: mrr = measured receive rate.
180 2. Second trial measures at MRR and discovers MRR2.
182 a. *in*: trial_duration = initial_trial_duration.
183 b. *in*: offered_transmit_rate = MRR.
184 c. *do*: single trial.
185 d. *out*: measured loss ratio.
186 e. *out*: mrr2 = measured receive rate.
188 3. Third trial measures at MRR2.
190 a. *in*: trial_duration = initial_trial_duration.
191 b. *in*: offered_transmit_rate = MRR2.
192 c. *do*: single trial.
193 d. *out*: measured loss ratio.
200 a. *in*: trial_duration for the current phase.
201 Set to initial_trial_duration for the first intermediate phase;
202 to final_trial_duration for the final phase;
203 or to the element of interpolating geometric sequence
204 for other intermediate phases.
205 For example with two intermediate phases, trial_duration
206 of the second intermediate phase is the geometric average
207 of initial_strial_duration and final_trial_duration.
208 b. *in*: relative_width_goal for the current phase.
209 Set to final_relative_width for the final phase;
210 doubled for each preceding phase.
211 For example with two intermediate phases,
212 the first intermediate phase uses quadruple of final_relative_width
213 and the second intermediate phase uses double of final_relative_width.
214 c. *in*: ndr_interval, pdr_interval from the previous main loop iteration
215 or the previous phase.
216 If the previous phase is the initial phase, both intervals have
217 lower_bound = MRR2, uper_bound = MRR.
218 Note that the initial phase is likely to create intervals with invalid bounds.
219 d. *do*: According to the procedure described in point 2,
220 either exit the phase (by jumping to 1.g.),
221 or prepare new transmit rate to measure with.
222 e. *do*: Perform the trial measurement at the new transmit rate
223 and trial_duration, compute its loss ratio.
224 f. *do*: Update the bounds of both intervals, based on the new measurement.
225 The actual update rules are numerous, as NDR external search
226 can affect PDR interval and vice versa, but the result
227 agrees with rules of both internal and external search.
228 For example, any new measurement below an invalid lower_bound
229 becomes the new lower_bound, while the old measurement
230 (previously acting as the invalid lower_bound)
231 becomes a new and valid upper_bound.
232 Go to next iteration (1.c.), taking the updated intervals as new input.
233 g. *out*: current ndr_interval and pdr_interval.
234 In the final phase this is also considered
235 to be the result of the whole search.
236 For other phases, the next phase loop is started
237 with the current results as an input.
239 2. New transmit rate (or exit) calculation (for 1.d.):
241 - If there is an invalid bound then prepare for external search:
243 - *If* the most recent measurement at NDR lower_bound transmit rate
244 had the loss higher than zero, then
245 the new transmit rate is NDR lower_bound
246 decreased by two NDR interval widths.
247 - Else, *if* the most recent measurement at PDR lower_bound
248 transmit rate had the loss higher than PLR, then
249 the new transmit rate is PDR lower_bound
250 decreased by two PDR interval widths.
251 - Else, *if* the most recent measurement at NDR upper_bound
252 transmit rate had no loss, then
253 the new transmit rate is NDR upper_bound
254 increased by two NDR interval widths.
255 - Else, *if* the most recent measurement at PDR upper_bound
256 transmit rate had the loss lower or equal to PLR, then
257 the new transmit rate is PDR upper_bound
258 increased by two PDR interval widths.
259 - If interval width is higher than the current phase goal:
261 - Else, *if* NDR interval does not meet the current phase width goal,
262 prepare for internal search. The new transmit rate is
263 (NDR lower bound + NDR upper bound) / 2.
264 - Else, *if* PDR interval does not meet the current phase width goal,
265 prepare for internal search. The new transmit rate is
266 (PDR lower bound + PDR upper bound) / 2.
267 - Else, *if* some bound has still only been measured at a lower duration,
268 prepare to re-measure at the current duration (and the same transmit rate).
269 The order of priorities is:
275 - *Else*, do not prepare any new rate, to exit the phase.
276 This ensures that at the end of each non-initial phase
277 all intervals are valid, narrow enough, and measured
278 at current phase trial duration.
280 Implementation Deviations
281 ~~~~~~~~~~~~~~~~~~~~~~~~~
283 This document so far has been describing a simplified version of MLRsearch algorithm.
284 The full algorithm as implemented contains additional logic,
285 which makes some of the details (but not general ideas) above incorrect.
286 Here is a short description of the additional logic as a list of principles,
287 explaining their main differences from (or additions to) the simplified description,
288 but without detailing their mutual interaction.
290 1. *Logarithmic transmit rate.*
291 In order to better fit the relative width goal,
292 the interval doubling and halving is done differently.
293 For example, the middle of 2 and 8 is 4, not 5.
294 2. *Optimistic maximum rate.*
295 The increased rate is never higher than the maximum rate.
296 Upper bound at that rate is always considered valid.
297 3. *Pessimistic minimum rate.*
298 The decreased rate is never lower than the minimum rate.
299 If a lower bound at that rate is invalid,
300 a phase stops refining the interval further (until it gets re-measured).
301 4. *Conservative interval updates.*
302 Measurements above current upper bound never update a valid upper bound,
303 even if drop ratio is low.
304 Measurements below current lower bound always update any lower bound
305 if drop ratio is high.
306 5. *Ensure sufficient interval width.*
307 Narrow intervals make external search take more time to find a valid bound.
308 If the new transmit increased or decreased rate would result in width
309 less than the current goal, increase/decrease more.
310 This can happen if the measurement for the other interval
311 makes the current interval too narrow.
312 Similarly, take care the measurements in the initial phase
313 create wide enough interval.
314 6. *Timeout for bad cases.*
315 The worst case for MLRsearch is when each phase converges to intervals
316 way different than the results of the previous phase.
317 Rather than suffer total search time several times larger
318 than pure binary search, the implemented tests fail themselves
319 when the search takes too long (given by argument *timeout*).
321 Maximum Receive Rate MRR
322 ------------------------
324 MRR tests measure the packet forwarding rate under the maximum
325 load offered by traffic generator over a set trial duration,
326 regardless of packet loss. Maximum load for specified Ethernet frame
327 size is set to the bi-directional link rate.
329 Current parameters for MRR tests:
331 - Ethernet frame sizes: 64B (78B for IPv6), IMIX, 1518B, 9000B; all
332 quoted sizes include frame CRC, but exclude per frame transmission
333 overhead of 20B (preamble, inter frame gap).
335 - Maximum load offered: 10GE and 40GE link (sub-)rates depending on NIC
336 tested, with the actual packet rate depending on frame size,
337 transmission overhead and traffic generator NIC forwarding capacity.
339 - For 10GE NICs the maximum packet rate load is 2* 14.88 Mpps for 64B,
340 a 10GE bi-directional link rate.
341 - For 25GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B,
342 a 25GE bi-directional link sub-rate limited by TG 25GE NIC used,
344 - For 40GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B,
345 a 40GE bi-directional link sub-rate limited by TG 40GE NIC used,
346 XL710. Packet rate for other tested frame sizes is limited by PCIe
347 Gen3 x8 bandwidth limitation of ~50Gbps.
349 - Trial duration: 10sec.
351 Similarly to NDR/PDR throughput tests, MRR test should be reporting bi-
352 directional link rate (or NIC rate, if lower) if tested VPP
353 configuration can handle the packet rate higher than bi-directional link
354 rate, e.g. large packet tests and/or multi-core tests.
356 MRR tests are used for continuous performance trending and for
357 comparison between releases. Daily trending job tests subset of frame
358 sizes, focusing on 64B (78B for IPv6) for all tests and IMIX for
359 selected tests (vhost, memif).
364 TRex Traffic Generator (TG) is used for measuring latency of VPP DUTs.
365 Reported latency values are measured using following methodology:
367 - Latency tests are performed at 100% of discovered NDR and PDR rates
368 for each throughput test and packet size (except IMIX).
369 - TG sends dedicated latency streams, one per direction, each at the
370 rate of 9 kpps at the prescribed packet size; these are sent in
371 addition to the main load streams.
372 - TG reports min/avg/max latency values per stream direction, hence two
373 sets of latency values are reported per test case; future release of
374 TRex is expected to report latency percentiles.
375 - Reported latency values are aggregate across two SUTs due to three
376 node topology used for all performance tests; for per SUT latency,
377 reported value should be divided by two.
378 - 1usec is the measurement accuracy advertised by TRex TG for the setup
379 used in FD.io labs used by CSIT project.
380 - TRex setup introduces an always-on error of about 2*2usec per latency
381 flow additonal Tx/Rx interface latency induced by TRex SW writing and
382 reading packet timestamps on CPU cores without HW acceleration on NICs
383 closer to the interface line.
388 All performance tests are executed with single processor core and with
389 multiple cores scenarios.
391 Intel Hyper-Threading (HT)
392 ~~~~~~~~~~~~~~~~~~~~~~~~~~
394 Intel Xeon processors used in FD.io CSIT can operate either in HT
395 Disabled mode (single logical core per each physical core) or in HT
396 Enabled mode (two logical cores per each physical core). HT setting is
397 applied in BIOS and requires server SUT reload for it to take effect,
398 making it impractical for continuous changes of HT mode of operation.
400 |csit-release| performance tests are executed with server SUTs' Intel
401 XEON processors configured with Intel Hyper-Threading Disabled for all
402 Xeon Haswell testbeds (3n-hsw) and with Intel Hyper-Threading Enabled
403 for all Xeon Skylake testbeds.
405 More information about physical testbeds is provided in
406 :ref:`tested_physical_topologies`.
411 |csit-release| multi-core tests are executed in the following VPP worker
412 thread and physical core configurations:
414 #. Intel Xeon Haswell testbeds (3n-hsw) with Intel HT disabled
415 (1 logical CPU core per each physical core):
417 #. 1t1c - 1 VPP worker thread on 1 physical core.
418 #. 2t2c - 2 VPP worker threads on 2 physical cores.
419 #. 4t4c - 4 VPP worker threads on 4 physical cores.
421 #. Intel Xeon Skylake testbeds (2n-skx, 3n-skx) with Intel HT enabled
422 (2 logical CPU cores per each physical core):
424 #. 2t1c - 2 VPP worker threads on 1 physical core.
425 #. 4t2c - 4 VPP worker threads on 2 physical cores.
426 #. 8t4c - 8 VPP worker threads on 4 physical cores.
428 VPP worker threads are the data plane threads running on isolated
429 logical cores. With Intel HT enabled VPP workers are placed as sibling
430 threads on each used physical core. VPP control threads (main, stats)
431 are running on a separate non-isolated core together with other Linux
434 In all CSIT tests care is taken to ensure that each VPP worker handles
435 the same amount of received packet load and does the same amount of
436 packet processing work. This is achieved by evenly distributing per
437 interface type (e.g. physical, virtual) receive queues over VPP workers
438 using default VPP round- robin mapping and by loading these queues with
439 the same amount of packet flows.
441 If number of VPP workers is higher than number of physical or virtual
442 interfaces, multiple receive queues are configured on each interface.
443 NIC Receive Side Scaling (RSS) for physical interfaces and multi-queue
444 for virtual interfaces are used for this purpose.
446 Section :ref:`throughput_speedup_multi_core` includes a set of graphs
447 illustrating packet throughout speedup when running VPP worker threads
448 on multiple cores. Note that in quite a few test cases running VPP
449 workers on 2 or 4 physical cores hits the I/O bandwidth or packets-per-
450 second limit of tested NIC.
455 CSIT code manipulates a number of VPP settings in startup.conf for optimized
456 performance. List of common settings applied to all tests and test
457 dependent settings follows.
459 See `VPP startup.conf <https://git.fd.io/vpp/tree/src/vpp/conf/startup.conf?h=stable/1807>`_
460 for a complete set and description of listed settings.
465 List of vpp startup.conf settings applied to all tests:
467 #. heap-size <value> - set separately for ip4, ip6, stats, main
468 depending on scale tested.
469 #. no-tx-checksum-offload - disables UDP / TCP TX checksum offload in DPDK.
470 Typically needed for use faster vector PMDs (together with
472 #. socket-mem <value>,<value> - memory per numa. (Not required anymore
473 due to VPP code changes, should be removed in CSIT-18.10.)
478 List of vpp startup.conf settings applied dynamically per test:
480 #. corelist-workers <list_of_cores> - list of logical cores to run VPP
481 worker data plane threads. Depends on HyperThreading and core per
483 #. num-rx-queues <value> - depends on a number of VPP threads and NIC
485 #. num-rx-desc/num-tx-desc - number of rx/tx descriptors for specific
486 NICs, incl. xl710, x710, xxv710.
487 #. num-mbufs <value> - increases number of buffers allocated, needed
488 only in scenarios with large number of interfaces and worker threads.
489 Value is per CPU socket. Default is 16384.
490 #. no-multi-seg - disables multi-segment buffers in DPDK, improves
491 packet throughput, but disables Jumbo MTU support. Disabled for all
492 tests apart from the ones that require Jumbo 9000B frame support.
493 #. UIO driver - depends on topology file definition.
494 #. QAT VFs - depends on NRThreads, each thread = 1QAT VFs.
499 FD.io CSIT performance lab is testing VPP vhost with KVM VMs using
500 following environment settings:
502 - Tests with varying Qemu virtio queue (a.k.a. vring) sizes: [vr256]
503 default 256 descriptors, [vr1024] 1024 descriptors to optimize for
505 - Tests with varying Linux :abbr:`CFS (Completely Fair Scheduler)`
506 settings: [cfs] default settings, [cfsrr1] CFS RoundRobin(1) policy
507 applied to all data plane threads handling test packet path including
508 all VPP worker threads and all Qemu testpmd poll-mode threads.
509 - Resulting test cases are all combinations with [vr256,vr1024] and
510 [cfs,cfsrr1] settings.
511 - Adjusted Linux kernel :abbr:`CFS (Completely Fair Scheduler)`
512 scheduler policy for data plane threads used in CSIT is documented in
513 `CSIT Performance Environment Tuning wiki <https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604>`_.
514 - The purpose is to verify performance impact (MRR and NDR/PDR
515 throughput) and same test measurements repeatability, by making VPP
516 and VM data plane threads less susceptible to other Linux OS system
517 tasks hijacking CPU cores running those data plane threads.
519 LXC/DRC Container Memif
520 -----------------------
522 |csit-release| includes tests taking advantage of VPP memif virtual
523 interface (shared memory interface) to interconnect VPP running in
524 Containers. VPP vswitch instance runs in bare-metal user-mode handling
525 NIC interfaces and connecting over memif (Slave side) to VPPs running in
526 :abbr:`Linux Container (LXC)` or in Docker Container (DRC) configured
527 with memif (Master side). LXCs and DRCs run in a priviliged mode with
528 VPP data plane worker threads pinned to dedicated physical CPU cores per
529 usual CSIT practice. All VPP instances run the same version of software.
530 This test topology is equivalent to existing tests with vhost-user and
531 VMs as described earlier in :ref:`tested_logical_topologies`.
533 In addition to above vswitch tests, a single memif interface test is
534 executed. It runs in a simple topology of two VPP container instances
535 connected over memif interface in order to verify standalone memif
536 interface performance.
538 More information about CSIT LXC and DRC setup and control is available
539 in :ref:`container_orchestration_in_csit`.
544 |csit-release| includes tests of VPP topologies running in K8s
545 orchestrated Pods/Containers and connected over memif virtual
546 interfaces. In order to provide simple topology coding flexibility and
547 extensibility container orchestration is done with `Kubernetes
548 <https://github.com/kubernetes>`_ using `Docker
549 <https://github.com/docker>`_ images for all container applications
550 including VPP. `Ligato <https://github.com/ligato>`_ is used for the
551 Pod/Container networking orchestration that is integrated with K8s,
552 including memif support.
554 In these tests VPP vswitch runs in a K8s Pod with Docker Container (DRC)
555 handling NIC interfaces and connecting over memif to more instances of
556 VPP running in Pods/DRCs. All DRCs run in a priviliged mode with VPP
557 data plane worker threads pinned to dedicated physical CPU cores per
558 usual CSIT practice. All VPP instances run the same version of software.
559 This test topology is equivalent to existing tests with vhost-user and
560 VMs as described earlier in :ref:`tested_physical_topologies`.
562 Further documentation is available in
563 :ref:`container_orchestration_in_csit`.
568 VPP IPSec performance tests are using DPDK cryptodev device driver in
569 combination with HW cryptodev devices - Intel QAT 8950 50G - present in
570 LF FD.io physical testbeds. DPDK cryptodev can be used for all IPSec
571 data plane functions supported by VPP.
573 Currently |csit-release| implements following IPSec test cases:
575 - AES-GCM, CBC-SHA1 ciphers, in combination with IPv4 routed-forwarding
576 with Intel xl710 NIC.
577 - CBC-SHA1 ciphers, in combination with LISP-GPE overlay tunneling for
578 IPv4-over-IPv4 with Intel xl710 NIC.
580 TRex Traffic Generator
581 ----------------------
586 `TRex traffic generator <https://wiki.fd.io/view/TRex>`_ is used for all
587 CSIT performance tests. TRex stateless mode is used to measure NDR and
588 PDR throughputs using binary search (NDR and PDR discovery tests) and
589 for quick checks of DUT performance against the reference NDRs (NDR
590 check tests) for specific configuration.
592 TRex is installed and run on the TG compute node. The typical procedure
595 - If the TRex is not already installed on TG, it is installed in the
596 suite setup phase - see `TRex intallation`_.
597 - TRex configuration is set in its configuration file
602 - TRex is started in the background mode
605 $ sh -c 'cd <t-rex-install-dir>/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --iom 0 > /tmp/trex.log 2>&1 &' > /dev/null
607 - There are traffic streams dynamically prepared for each test, based on traffic
608 profiles. The traffic is sent and the statistics obtained using
609 :command:`trex_stl_lib.api.STLClient`.
611 Measuring Packet Loss
612 ~~~~~~~~~~~~~~~~~~~~~
614 Following sequence is followed to measure packet loss:
616 - Create an instance of STLClient.
617 - Connect to the client.
620 - Send the traffic for defined time.
621 - Get the statistics.
623 If there is a warm-up phase required, the traffic is sent also before
624 test and the statistics are ignored.
629 If measurement of latency is requested, two more packet streams are
630 created (one for each direction) with TRex flow_stats parameter set to
631 STLFlowLatencyStats. In that case, returned statistics will also include
632 min/avg/max latency values.
634 HTTP/TCP with WRK tool
635 ----------------------
637 `WRK HTTP benchmarking tool <https://github.com/wg/wrk>`_ is used for
638 experimental TCP/IP and HTTP tests of VPP TCP/IP stack and built-in
639 static HTTP server. WRK has been chosen as it is capable of generating
640 significant TCP/IP and HTTP loads by scaling number of threads across
641 multi-core processors.
643 This in turn enables quite high scale benchmarking of the main TCP/IP
644 and HTTP service including HTTP TCP/IP Connections-Per-Second (CPS),
645 HTTP Requests-Per-Second and HTTP Bandwidth Throughput.
647 The initial tests are designed as follows:
649 - HTTP and TCP/IP Connections-Per-Second (CPS)
651 - WRK configured to use 8 threads across 8 cores, 1 thread per core.
652 - Maximum of 50 concurrent connections across all WRK threads.
653 - Timeout for server responses set to 5 seconds.
654 - Test duration is 30 seconds.
655 - Expected HTTP test sequence:
657 - Single HTTP GET Request sent per open connection.
658 - Connection close after valid HTTP reply.
659 - Resulting flow sequence - 8 packets: >Syn, <Syn-Ack, >Ack, >Req,
660 <Rep, >Fin, <Fin, >Ack.
662 - HTTP Requests-Per-Second
664 - WRK configured to use 8 threads across 8 cores, 1 thread per core.
665 - Maximum of 50 concurrent connections across all WRK threads.
666 - Timeout for server responses set to 5 seconds.
667 - Test duration is 30 seconds.
668 - Expected HTTP test sequence:
670 - Multiple HTTP GET Requests sent in sequence per open connection.
671 - Connection close after set test duration time.
672 - Resulting flow sequence: >Syn, <Syn-Ack, >Ack, >Req[1], <Rep[1],
673 .., >Req[n], <Rep[n], >Fin, <Fin, >Ack.
675 .. _binary search: https://en.wikipedia.org/wiki/Binary_search
676 .. _exponential search: https://en.wikipedia.org/wiki/Exponential_search
677 .. _estimation of standard deviation: https://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation
678 .. _simplified error propagation formula: https://en.wikipedia.org/wiki/Propagation_of_uncertainty#Simplification