1 Data Plane Throughput Tests
2 ---------------------------
4 Network data plane throughput is measured using multiple test methods in
5 order to obtain representative and repeatable results across the large
6 set of performance test cases implemented and executed within CSIT.
8 Following throughput test methods are used:
10 - MLRsearch - Multiple Loss Ratio search
11 - MRR - Maximum Receive Rate
12 - PLRsearch - Probabilistic Loss Ratio search
14 Description of each test method is followed by generic test properties
15 shared by all methods.
23 Multiple Loss Ratio search (MLRsearch) tests discover multiple packet
24 throughput rates in a single search, reducing the overall test execution
25 time compared to a binary search. Each rate is associated with a
26 distinct Packet Loss Ratio (PLR) criteria. In FD.io CSIT two throughput
27 rates are discovered: Non-Drop Rate (NDR, with zero packet loss, PLR=0)
28 and Partial Drop Rate (PDR, with PLR<0.5%). MLRsearch is compliant with
34 MLRsearch tests are run to discover NDR and PDR rates for each VPP and
35 DPDK release covered by CSIT report. Results for small frame sizes
36 (64b/78B, IMIX) are presented in packet throughput graphs
37 (Box-and-Whisker Plots) with NDR and PDR rates plotted against the test
38 cases covering popular VPP packet paths.
40 Each test is executed at least 10 times to verify measurements
41 repeatability and results are compared between releases and test
42 environments. NDR and PDR packet and bandwidth throughput results for
43 all frame sizes and for all tests are presented in detailed results
49 See :ref:`mlrsearch_algorithm` section for more detail. MLRsearch is
50 being standardized in IETF in `draft-vpolak-mkonstan-mlrsearch
51 <https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch>`_.
59 Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests,
60 as they provide a maximum “raw” throughput benchmark for development and
63 MRR tests measure the packet forwarding rate under the maximum load
64 offered by traffic generator (dependent on link type and NIC model) over
65 a set trial duration, regardless of packet loss. Maximum load for
66 specified Ethernet frame size is set to the bi-directional link rate.
71 MRR tests are much faster than MLRsearch as they rely on a single trial
72 or a small set of trials with very short duration. It is this property
73 that makes them suitable for continuous execution in daily performance
74 trending jobs enabling detection of performance anomalies (regressions,
75 progressions) resulting from data plane code changes.
77 MRR tests are also used for VPP per patch performance jobs verifying
78 patch performance vs. parent. CSIT reports include MRR throughput
79 comparisons between releases and test environments. Small frame sizes
85 See :ref:`mrr_throughput` section for more detail about MRR tests
88 FD.io CSIT performance dashboard includes complete description of
89 `daily performance trending tests
90 <https://docs.fd.io/csit/master/trending/methodology/performance_tests.html>`_
91 and `VPP per patch tests
92 <https://docs.fd.io/csit/master/trending/methodology/perpatch_performance_tests.html>`_.
100 Probabilistic Loss Ratio search (PLRsearch) tests discovers a packet
101 throughput rate associated with configured Packet Loss Ratio (PLR)
102 criteria for tests run over an extended period of time a.k.a. soak
103 testing. PLRsearch assumes that system under test is probabilistic in
104 nature, and not deterministic.
109 PLRsearch are run to discover a sustained throughput for PLR=10^-7
110 (close to NDR) for VPP release covered by CSIT report. Results for small
111 frame sizes (64b/78B) are presented in packet throughput graphs (Box
112 Plots) for a small subset of baseline tests.
114 Each soak test lasts 2hrs and is executed at least twice. Results are
115 compared against NDR and PDR rates discovered with MLRsearch.
120 See :ref:`plrsearch` methodology section for more detail. PLRsearch is
121 being standardized in IETF in `draft-vpolak-bmwg-plrsearch
122 <https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch>`_.
124 Generic Test Properties
125 ^^^^^^^^^^^^^^^^^^^^^^^
127 All data plane throughput test methodologies share following generic
130 - Tested L2 frame sizes (untagged Ethernet):
132 - IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B.
133 - IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B.
134 - All quoted sizes include frame CRC, but exclude per frame
135 transmission overhead of 20B (preamble, inter frame gap).
137 - Offered packet load is always bi-directional and symmetric.
138 - All measured and reported packet and bandwidth rates are aggregate
139 bi-directional rates reported from external Traffic Generator