4 Network data plane packet and bandwidth throughput are measured using
5 multiple methods in order to obtain representative and repeatable
6 results across the large set of performance test cases implemented and
7 executed within CSIT. Following throughput test methods are used:
9 #. MLRsearch: Multiple Loss Ratio search
11 - **Description**: MLRsearch discovers multiple packet throughput
12 rates in a single search, reducing the overall test execution
13 time compared to a binary search. Each rate associated with a
14 distinct Packet Loss Ratio (PLR) criteria. In FD.io CSIT two
15 throughput rates are discovered: Non-Drop Rate (NDR, with zero
16 packet loss, PLR=0) and Partial Drop Rate (PDR, with PLR<0.5%).
17 MLRsearch is compliant with :rfc:`2544`.
18 - **Usage**: MLRsearch tests are run to discover NDR and PDR rates
19 for each VPP and DPDK release covered by CSIT report. Results for
20 small frame sizes (64b/78B, IMIX) are presented in packet
21 throughput graphs (Box-and-Whisker Plots) with NDR and PDR rates
22 plotted against the test cases covering popular VPP packet paths.
23 Each test is executed at least 10 times to verify measurements
24 repeatability and results are compared between releases and test
25 environments. NDR and PDR packet and bandwidth throughput results
26 for all frame sizes and for all tests are presented in detailed
28 - **References**: See :ref:`mlrsearch_algorithm` for more detailed
29 description of MLRsearch tests. MLRsearch is being standardized
30 in IETF with `draft-vpolak-mkonstan-mlrsearch
31 <https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch>`_.
33 #. MRR Measurements: Maximum Receive Rate
35 - **Description**: MRR tests are complementary to MLRsearch tests,
36 as they provide a maximum “raw” throughput benchmark for
37 development and testing community. MRR tests measure the packet
38 forwarding rate under the maximum load offered by traffic
39 generator (dependent on link type and NIC model) over a set trial
40 duration, regardless of packet loss. Maximum load for specified
41 Ethernet frame size is set to the bi-directional link rate.
42 - **Usage**: MRR tests are much faster than MLRsearch as they rely
43 on a single trial or a small set of trials with very short
44 duration. It is this property that makes them suitable for
45 continuous execution in daily performance trending jobs enabling
46 detection of performance anomalies (regressions, progressions)
47 resulting from data plane code changes. MRR tests are also used
48 for VPP per patch performance jobs verifying patch performance
49 vs. parent. CSIT reports include MRR throughput comparisons
50 between releases and test environments. Small frame sizes only
52 - **References**: See :ref:`mrr_throughput` for more detailed
53 description of MRR tests configuration used for daily performance
54 trending jobs. VPP per patch test methodology is available on
55 `FD.io CSIT trending pages
56 <https://docs.fd.io/csit/master/trending/methodology/perpatch_performance_tests.html>`_.
58 #. PLRsearch: Probabilistic Loss Ratio search
60 - **Description**: PLRsearch discovers a packet throughput rate
61 associated with configured Packet Loss Ratio (PLR) criteria for
62 tests run over an extended period of time a.k.a. soak testing.
63 PLRsearch assumes that system under test is probabilistic in
64 nature, and not deterministic.
65 - **Usage**: PLRsearch are run to discover a sustained throughput
66 for PLR=10^-7 (close to NDR) for VPP release covered by CSIT
67 report. Results for small frame sizes (64b/78B) are presented in
68 packet throughput graphs (Box Plots) for a small subset of
69 baseline tests. Each soak test lasts 2hrs and is executed at
70 least twice. Results are compared against NDR and PDR rates
71 discovered with MLRsearch.
72 - **References**: See :ref:`plrsearch_algorithm` for more detailed
73 description of PLRsearch tests. PLRsearch is being standardized
74 in IETF with `draft-vpolak-bmwg-plrsearch
75 <https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch>`_.
77 All of the listed data plane throughput test methodologies share
80 - Tested L2 frame sizes (untagged Ethernet):
82 - IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B.
83 - IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B.
85 - All measured rates are aggregate bi-directional rates reported from
86 external Traffic Generator perspective.