2 title: Multiple Loss Ratio Search for Packet Throughput (MLRsearch)
4 docname: draft-ietf-bmwg-mlrsearch-01
9 wg: Benchmarking Working Group
14 pi: # can use array (if all yes) or hash here
19 sortrefs: # defaults to yes
24 ins: M. Konstantynowicz
25 name: Maciek Konstantynowicz
28 email: mkonstan@cisco.com
34 email: vrpolak@cisco.com
41 target: https://docs.fd.io/csit/rls2101/report/introduction/methodology_data_plane_throughput/methodology_mlrsearch_tests.html
42 title: "FD.io CSIT Test Methodology - MLRsearch"
45 target: https://pypi.org/project/MLRsearch/0.4.0/
46 title: "MLRsearch 0.4.0, Python Package Index"
51 This document proposes changes to [RFC2544], specifically to packet
52 throughput search methodology, by defining a new search algorithm
53 referred to as Multiple Loss Ratio search (MLRsearch for short). Instead
54 of relying on binary search with pre-set starting offered load, it
55 proposes a novel approach discovering the starting point in the initial
56 phase, and then searching for packet throughput based on defined packet
57 loss ratio (PLR) input criteria and defined final trial duration time.
58 One of the key design principles behind MLRsearch is minimizing the
59 total test duration and searching for multiple packet throughput rates
60 (each with a corresponding PLR) concurrently, instead of doing it
63 The main motivation behind MLRsearch is the new set of challenges and
64 requirements posed by NFV (Network Function Virtualization),
65 specifically software based implementations of NFV data planes. Using
66 [RFC2544] in the experience of the authors yields often not repetitive
67 and not replicable end results due to a large number of factors that are
68 out of scope for this draft. MLRsearch aims to address this challenge
69 in a simple way of getting the same result sooner, so more repetitions
70 can be done to describe the replicability.
76 * Frame size: size of an Ethernet Layer-2 frame on the wire, including
77 any VLAN tags (dot1q, dot1ad) and Ethernet FCS, but excluding Ethernet
78 preamble and inter-frame gap. Measured in bytes (octets).
79 * Packet size: same as frame size, both terms used interchangeably.
80 * Device Under Test (DUT): In software networking, "device" denotes a
81 specific piece of software tasked with packet processing. Such device
82 is surrounded with other software components (such as operating system
83 kernel). It is not possible to run devices without also running the
84 other components, and hardware resources are shared between both. For
85 purposes of testing, the whole set of hardware and software components
86 is called "system under test" (SUT). As SUT is the part of the whole
87 test setup performance of which can be measured by [RFC2544] methods,
88 this document uses SUT instead of [RFC2544] DUT. Device under test
89 (DUT) can be re-introduced when analysing test results using whitebox
90 techniques, but this document sticks to blackbox testing.
91 * System Under Test (SUT): System under test (SUT) is a part of the
92 whole test setup whose performance is to be benchmarked. The complete
93 test setup contains other parts, whose performance is either already
94 established, or not affecting the benchmarking result.
95 * Bi-directional throughput tests: involve packets/frames flowing in
96 both transmit and receive directions over every tested interface of
97 SUT/DUT. Packet flow metrics are measured per direction, and can be
98 reported as aggregate for both directions and/or separately
99 for each measured direction. In most cases bi-directional tests
100 use the same (symmetric) load in both directions.
101 * Uni-directional throughput tests: involve packets/frames flowing in
102 only one direction, i.e. either transmit or receive direction, over
103 every tested interface of SUT/DUT. Packet flow metrics are measured
104 and are reported for measured direction.
105 * Packet Loss Ratio (PLR): ratio of packets received relative to packets
106 transmitted over the test trial duration, calculated using formula:
107 PLR = ( pkts_transmitted - pkts_received ) / pkts_transmitted.
108 For bi-directional throughput tests aggregate PLR is calculated based
109 on the aggregate number of packets transmitted and received.
110 * Effective loss ratio: A corrected value of measured packet loss ratio
111 chosen to avoid difficulties if SUT exhibits decreasing loss
112 with increasing load. Maximum of packet loss ratios measured at the same
113 duration on all loads smaller than (and including) the current one.
114 * Target loss ratio: A packet loss ratio value acting as an imput for search.
115 The search is finding tight enough lower and upper bound in intended load,
116 so that the lower bound has smaller or equal loss ratio, and upper bound
117 has strictly larger loss ratio. For the tighterst upper bound,
118 the effective loss ratio is the same as packet loss ratio.
119 For the tightest lower bound, the effective loss ratio can be higher
120 than the packet loss ratio, but still not larger than the target loss ratio.
121 * Packet Throughput Rate: maximum packet offered load DUT/SUT forwards
122 within the specified Packet Loss Ratio (PLR). In many cases the rate
123 depends on the frame size processed by DUT/SUT. Hence packet
124 throughput rate MUST be quoted with specific frame size as received by
125 DUT/SUT during the measurement. For bi-directional tests, packet
126 throughput rate should be reported as aggregate for both directions.
127 Measured in packets-per-second (pps) or frames-per-second (fps),
129 * Bandwidth Throughput Rate: a secondary metric calculated from packet
130 throughput rate using formula: bw_rate = pkt_rate * (frame_size +
131 L1_overhead) * 8, where L1_overhead for Ethernet includes preamble (8
132 octets) and inter-frame gap (12 octets). For bi-directional tests,
133 bandwidth throughput rate should be reported as aggregate for both
134 directions. Expressed in bits-per-second (bps).
135 * Non Drop Rate (NDR): maximum packet/bandwith throughput rate sustained
136 by DUT/SUT at PLR equal zero (zero packet loss) specific to tested
137 frame size(s). MUST be quoted with specific packet size as received by
138 DUT/SUT during the measurement. Packet NDR measured in
139 packets-per-second (or fps), bandwidth NDR expressed in
140 bits-per-second (bps).
141 * Partial Drop Rate (PDR): maximum packet/bandwith throughput rate
142 sustained by DUT/SUT at PLR greater than zero (non-zero packet loss)
143 specific to tested frame size(s). MUST be quoted with specific packet
144 size as received by DUT/SUT during the measurement. Packet PDR
145 measured in packets-per-second (or fps), bandwidth PDR expressed in
146 bits-per-second (bps).
147 * Maximum Receive Rate (MRR): packet/bandwidth rate regardless of PLR
148 sustained by DUT/SUT under specified Maximum Transmit Rate (MTR)
149 packet load offered by traffic generator. MUST be quoted with both
150 specific packet size and MTR as received by DUT/SUT during the
151 measurement. Packet MRR measured in packets-per-second (or fps),
152 bandwidth MRR expressed in bits-per-second (bps).
153 * Trial: a single measurement step. See [RFC2544] section 23.
154 * Trial duration: amount of time over which packets are transmitted
155 in a single measurement step.
157 # MLRsearch Background
159 Multiple Loss Ratio search (MLRsearch) is a packet throughput search
160 algorithm suitable for deterministic systems (as opposed to
161 probabilistic systems). MLRsearch discovers multiple packet throughput
162 rates in a single search, each rate is associated with a distinct
163 Packet Loss Ratio (PLR) criterion.
165 For cases when multiple rates need to be found, this property makes
166 MLRsearch more efficient in terms of time execution, compared to
167 traditional throughput search algorithms that discover a single packet
168 rate per defined search criteria (e.g. a binary search specified by
169 [RFC2544]). MLRsearch reduces execution time even further by relying on
170 shorter trial durations of intermediate steps, with only the final
171 measurements conducted at the specified final trial duration. This
172 results in the shorter overall search execution time when compared to a
173 traditional binary search, while guaranteeing the same results for
174 deterministic systems.
176 In practice two rates with distinct PLRs are commonly used for packet
177 throughput measurements of NFV systems: Non Drop Rate (NDR) with PLR=0
178 and Partial Drop Rate (PDR) with PLR>0. The rest of this document
179 describes MLRsearch with NDR and PDR pair as an example.
181 Similarly to other throughput search approaches like binary search,
182 MLRsearch is effective for SUTs/DUTs with PLR curve that is
183 non-decreasing with growing offered load. It may not be as
184 effective for SUTs/DUTs with abnormal PLR curves, although
185 it will always converge to some value.
187 MLRsearch relies on traffic generator to qualify the received packet
188 stream as error-free, and invalidate the results if any disqualifying
189 errors are present e.g. out-of-sequence frames.
191 MLRsearch can be applied to both uni-directional and bi-directional
194 For bi-directional tests, MLRsearch rates and ratios are aggregates of
195 both directions, based on the following assumptions:
197 * Traffic transmitted by traffic generator and received by SUT/DUT
198 has the same packet rate in each direction,
199 in other words the offered load is symmetric.
200 * SUT/DUT packet processing capacity is the same in both directions,
201 resulting in the same packet loss under load.
203 MLRsearch can be applied even without those assumptions,
204 but in that case the aggregate loss ratio is less useful as a metric.
206 MLRsearch can be used for network transactions consisting of more than
207 just one packet, or anything else that has intended load as input
208 and loss ratio as output (duration as input is optional).
209 This text uses mostly packet-centric language.
213 The main properties of MLRsearch:
215 * MLRsearch is a duration aware multi-phase multi-rate search algorithm:
216 * Initial Phase determines promising starting interval for the search.
217 * Intermediate Phases progress towards defined final search criteria.
218 * Final Phase executes measurements according to the final search
220 * Final search criteria are defined by following inputs:
221 * Target PLRs (e.g. 0.0 and 0.005 when searching for NDR and PDR).
222 * Final trial duration.
223 * Measurement resolution.
225 * Measure MRR over initial trial duration.
226 * Measured MRR is used as an input to the first intermediate phase.
227 * Multiple Intermediate Phases:
229 * Start with initial trial duration in the first intermediate phase.
230 * Converge geometrically towards the final trial duration.
231 * Track all previous trial measurement results:
232 * Duration, offered load and loss ratio are tracked.
233 * Effective loss ratios are tracked.
234 * While in practice, real loss ratios can decrease with increasing load,
235 effective loss ratios never decrease. This is achieved by sorting
236 results by load, and using the effective loss ratio of the previous load
237 if the current loss ratio is smaller than that.
238 * The algorithm queries the results to find best lower and upper bounds.
239 * Effective loss ratios are always used.
240 * The phase ends if all target loss ratios have tight enough bounds.
242 * Iterate over target loss ratios in increasing order.
243 * If both upper and lower bound are in measurement results for this duration,
244 apply bisect until the bounds are tight enough,
245 and continue with next loss ratio.
246 * If a bound is missing for this duration, but there exists a bound
247 from the previous duration (compatible with the other bound
248 at this duration), re-measure at the current duration.
249 * If a bound in one direction (upper or lower) is missing for this duration,
250 and the previous duration does not have a compatible bound,
251 compute the current "interval size" from the second tightest bound
252 in the other direction (lower or upper respectively)
253 for the current duration, and choose next offered load for external search.
254 * The logic guarantees that a measurement is never repeated with both
255 duration and offered load being the same.
256 * The logic guarantees that measurements for higher target loss ratio
257 iterations (still within the same phase duration) do not affect validity
258 and tightness of bounds for previous target loss ratio iterations
259 (at the same duration).
260 * Use of internal and external searches:
262 * It is a variant of "exponential search".
263 * The "interval size" is multiplied by a configurable constant
264 (powers of two work well with the subsequent internal search).
266 * A variant of binary search that measures at offered load between
267 the previously found bounds.
268 * The interval does not need to be split into exact halves,
269 if other split can get to the target width goal faster.
270 * The idea is to avoid returning interval narrower than the current
271 width goal. See sample implementation details, below.
273 * Executed with the final test trial duration, and the final width
274 goal that determines resolution of the overall search.
275 * Intermediate Phases together with the Final Phase are called
277 * The returned bounds stay within prescribed min_rate and max_rate.
278 * When returning min_rate or max_rate, the returned bounds may be invalid.
279 * E.g. upper bound at max_rate may come from a measurement
280 with loss ratio still not higher than the target loss ratio.
282 The main benefits of MLRsearch vs. binary search include:
284 * In general MLRsearch is likely to execute more trials overall, but
285 likely less trials at a set final trial duration.
286 * In well behaving cases, e.g. when results do not depend on trial
287 duration, it greatly reduces (>50%) the overall duration compared to a
288 single PDR (or NDR) binary search over duration, while finding
290 * In all cases MLRsearch yields the same or similar results to binary
292 * Note: both binary search and MLRsearch are susceptible to reporting
293 non-repeatable results across multiple runs for very bad behaving
298 * Worst case MLRsearch can take longer than a binary search, e.g. in case of
299 drastic changes in behaviour for trials at varying durations.
300 * Re-measurement at higher duration can trigger a long external search.
301 That never happens in binary search, which uses the final duration
304 # Sample Implementation
306 Following is a brief description of a sample MLRsearch implementation,
307 which is a simplified version of the existing implementation.
311 1. **max_rate** - Maximum Transmit Rate (MTR) of packets to
312 be used by external traffic generator implementing MLRsearch,
313 limited by the actual Ethernet link(s) rate, NIC model or traffic
314 generator capabilities.
315 2. **min_rate** - minimum packet transmit rate to be used for
316 measurements. MLRsearch fails if lower transmit rate needs to be
317 used to meet search criteria.
318 3. **final_trial_duration** - required trial duration for final rate
320 4. **initial_trial_duration** - trial duration for initial MLRsearch phase.
321 5. **final_relative_width** - required measurement resolution expressed as
322 (lower_bound, upper_bound) interval width relative to upper_bound.
323 6. **packet_loss_ratios** - list of maximum acceptable PLR search criteria.
324 7. **number_of_intermediate_phases** - number of phases between the initial
325 phase and the final phase. Impacts the overall MLRsearch duration.
326 Less phases are required for well behaving cases, more phases
327 may be needed to reduce the overall search duration for worse behaving cases.
331 1. First trial measures at configured maximum transmit rate (MTR) and
332 discovers maximum receive rate (MRR).
333 * IN: trial_duration = initial_trial_duration.
334 * IN: offered_transmit_rate = maximum_transmit_rate.
336 * OUT: measured loss ratio.
337 * OUT: MRR = measured receive rate.
338 Received rate is computed as intended load multiplied by pass ratio
339 (which is one minus loss ratio). This is useful when loss ratio is computed
340 from a different metric than intended load. For example, intended load
341 can be in transactions (multiple packets each), but loss ratio is computed
342 on level of packets, not transactions.
344 * Example: If MTR is 10 transactions per second, and each transaction has
345 10 packets, and receive rate is 90 packets per second, then loss rate
346 is 10%, and MRR is computed to be 9 transactions per second.
348 If MRR is too close to MTR, MRR is set below MTR so that interval width
349 is equal to the width goal of the first intermediate phase.
350 If MRR is less than min_rate, min_rate is used.
351 2. Second trial measures at MRR and discovers MRR2.
352 * IN: trial_duration = initial_trial_duration.
353 * IN: offered_transmit_rate = MRR.
355 * OUT: measured loss ratio.
356 * OUT: MRR2 = measured receive rate.
357 If MRR2 is less than min_rate, min_rate is used.
358 If loss ratio is less or equal to the smallest target loss ratio,
359 MRR2 is set to a value above MRR, so that interval width is equal
360 to the width goal of the first intermediate phase.
361 MRR2 could end up being equal to MTR (for example if both measurements so far
362 had zero loss), which was already measured, step 3 is skipped in that case.
363 3. Third trial measures at MRR2.
364 * IN: trial_duration = initial_trial_duration.
365 * IN: offered_transmit_rate = MRR2.
367 * OUT: measured loss ratio.
368 * OUT: MRR3 = measured receive rate.
369 If MRR3 is less than min_rate, min_rate is used.
370 If step 3 is not skipped, the first trial measurement is forgotten.
371 This is done because in practice (if MRR2 is above MRR), external search
372 from MRR and MRR2 is likely to lead to a faster intermediate phase
373 than a bisect between MRR2 and MTR.
375 ## Non-Initial Phases
378 1. IN: trial_duration for the current phase. Set to
379 initial_trial_duration for the first intermediate phase; to
380 final_trial_duration for the final phase; or to the element of
381 interpolating geometric sequence for other intermediate phases.
382 For example with two intermediate phases, trial_duration of the
383 second intermediate phase is the geometric average of
384 initial_trial_duration and final_trial_duration.
385 2. IN: relative_width_goal for the current phase. Set to
386 final_relative_width for the final phase; doubled for each
387 preceding phase. For example with two intermediate phases, the
388 first intermediate phase uses quadruple of final_relative_width
389 and the second intermediate phase uses double of
390 final_relative_width.
391 3. IN: Measurement results from the previous phase (previous duration).
392 4. Internal target ratio loop:
393 1. IN: Target loss ratio for this iteration of ratio loop.
394 2. IN: Measurement results from all previous ratio loop iterations
395 of current phase (current duration).
396 3. DO: According to the procedure described in point 2:
397 1. either exit the phase (by jumping to 1.5),
398 2. or exit loop iteration (by continuing with next target loss ratio,
400 3. or calculate new transmit rate to measure with.
401 4. DO: Perform the trial measurement at the new transmit rate and
402 current trial duration, compute its loss ratio.
403 5. DO: Add the result and go to next iteration (1.4.1),
404 including the added trial result in 1.4.2.
405 5. OUT: Measurement results from this phase.
406 6. OUT: In the final phase, bounds for each target loss ratio
407 are extracted and returned.
408 1. If a valid bound does not exist, use min_rate or max_rate.
409 2. New transmit rate (or exit) calculation (for point 1.4.3):
410 1. If the previous duration has the best upper and lower bound,
411 select the middle point as the new transmit rate.
412 1. See 2.5.3. below for the exact splitting logic.
413 2. This can be a no-op if interval is narrow enough already,
414 in that case continue with 2.2.
415 3. Discussion, assuming the middle point is selected and measured:
416 1. Regardless of loss rate measured, the result becomes
417 either best upper or best lower bound at current duration.
418 2. So this condition is satisfied at most once per iteration.
419 3. This also explains why previous phase has double width goal:
420 1. We avoid one more bisection at previous phase.
421 2. At most one bound (per iteration) is re-measured
422 with current duration.
423 3. Each re-measurement can trigger an external search.
424 4. Such surprising external searches are the main hurdle
425 in achieving low overal search durations.
426 5. Even without 1.1, there is at most one external search
427 per phase and target loss ratio.
428 6. But without 1.1 there can be two re-measurements,
429 each coming with a risk of triggering external search.
430 2. If the previous duration has one bound best, select its transmit rate.
431 In deterministic case this is the last measurement needed this iteration.
432 3. If only upper bound exists in current duration results:
433 1. This can only happen for the smallest target loss ratio.
434 2. If the upper bound was measured at min_rate,
435 exit the whole phase early (not investigating other target loss ratios).
436 3. Select new transmit rate using external search:
437 1. For computing previous interval size, use:
438 1. second tightest bound at current duration,
439 2. or tightest bound of previous duration,
440 if compatible and giving a more narrow interval,
441 3. or target interval width if none of the above is available.
442 4. In any case increase to target interval width if smaller.
443 2. Quadruple the interval width.
444 3. Use min_rate if the new transmit rate is lower.
445 4. If only lower bound exists in current duration results:
446 1. If the lower bound was measured at max_rate,
447 exit this iteration (continue with next lowest target loss ratio).
448 2. Select new transmit rate using external search:
449 1. For computing previous interval size, use:
450 1. second tightest bound at current duration,
451 2. or tightest bound of previous duration,
452 if compatible and giving a more narrow interval,
453 3. or target interval width if none of the above is available.
454 4. In any case increase to target interval width if smaller.
455 2. Quadruple the interval width.
456 3. Use max_rate if the new transmit rate is higher.
457 5. The only remaining option is both bounds in current duration results.
458 1. This can happen in two ways, depending on how the lower bound
460 1. It could have been selected for the current loss ratio,
461 e.g. in re-measurement (2.2) or in initial bisect (2.1).
462 2. It could have been found as an upper bound for the previous smaller
463 target loss ratio, in which case it might be too low.
464 3. The algorithm does not track which one is the case,
465 as the decision logic works well regardless.
466 2. Compute "extending down" candidate transmit rate exactly as in 2.3.
467 3. Compute "bisecting" candidate transmit rate:
468 1. Compute the current interval width from the two bounds.
469 2. Express the width as a (float) multiple of the target width goal
471 3. If the multiple is not higher than one, it means the width goal
472 is met. Exit this iteration and continue with next higher
474 4. If the multiple is two or less, use half of that
475 for new width if the lower subinterval.
476 5. Round the multiple up to nearest even integer.
477 6. Use half of that for new width if the lower subinterval.
478 7. Example: If lower bound is 2.0 and upper bound is 5.0, and width
479 goal is 1.0, the new candidate transmit rate will be 4.0.
480 This can save a measurement when 4.0 has small loss.
481 Selecting the average (3.5) would never save a measurement,
482 giving more narrow bounds instead.
483 4. If either candidate computation want to exit the iteration,
484 do as bisecting candidate computation says.
485 5. The remaining case is both candidates wanting to measure at some rate.
486 Use the higher rate. This prefers external search down narrow enough
487 interval, competing with perfectly sized lower bisect subinterval.
489 # FD.io CSIT Implementation
491 The only known working implementation of MLRsearch is in
492 the open-source code running in Linux Foundation
493 FD.io CSIT project [FDio-CSIT-MLRsearch] as part of
494 a Continuous Integration / Continuous Development (CI/CD) framework.
496 MLRsearch is also available as a Python package in [PyPI-MLRsearch].
498 ## Additional details
500 This document so far has been describing a simplified version of
501 MLRsearch algorithm. The full algorithm as implemented in CSIT contains
502 additional logic, which makes some of the details (but not general
503 ideas) above incorrect. Here is a short description of the additional
504 logic as a list of principles, explaining their main differences from
505 (or additions to) the simplified description, but without detailing
506 their mutual interaction.
508 1. Logarithmic transmit rate.
509 * In order to better fit the relative width goal, the interval
510 doubling and halving is done differently.
511 * For example, the middle of 2 and 8 is 4, not 5.
512 2. Timeout for bad cases.
513 * The worst case for MLRsearch is when each phase converges to
514 intervals way different than the results of the previous phase.
515 * Rather than suffer total search time several times larger than pure
516 binary search, the implemented tests fail themselves when the
517 search takes too long (given by argument *timeout*).
519 * The number of packets to send during the trial should be equal to
520 the intended load multiplied by the duration.
521 * Also multiplied by a coefficient, if loss ratio is calculated
522 from a different metric.
523 * Example: If a successful transaction uses 10 packets,
524 load is given in transactions per second, byt loss ratio is calculated
525 from packets, the coefficient to get intended count of packets is 10.
526 * But in practice that does not work.
527 * It could result in a fractional number of packets,
528 * so it has to be rounded in a way traffic generator chooses,
529 * which may depend on the number of traffic flows
530 and traffic generator worker threads.
531 4. Attempted count. As the real number of intended packets is not known exactly,
532 the computation uses the number of packets traffic generator reports as sent.
533 Unless overriden by the next point.
534 5. Duration stretching.
535 * In some cases, traffic generator may get overloaded,
536 causing it to take significantly longer (than duration) to send all packets.
537 * The implementation uses an explicit stop,
538 * causing lower attempted count in those cases.
539 * The implementation tolerates some small difference between
540 attempted count and intended count.
541 * 10 microseconds worth of traffic is sufficient for our tests.
542 * If the difference is higher, the unsent packets are counted as lost.
543 * This forces the search to avoid the regions of high duration stretching.
544 * The final bounds describe the performance of not just SUT,
545 but of the whole system, including the traffic generator.
547 * In some test (e.g. using TCP flows) Traffic generator reacts to packet loss
548 by retransmission. Usually, such packet loss is already affecting loss ratio.
549 If a test also wants to treat retransmissions due to heavily delayed packets
550 also as a failure, this is once again visible as a mismatch between
551 the intended count and the attempted count.
552 * The CSIT implementation simply looks at absolute value of the difference,
553 so it offes the same small tolerance before it start marking a "loss".
554 7. For result processing, we use lower bounds and ignore upper bounds.
556 ### FD.io CSIT Input Parameters
558 1. **max_rate** - Typical values: 2 * 14.88 Mpps for 64B
559 10GE link rate, 2 * 18.75 Mpps for 64B 40GE NIC (specific model).
560 2. **min_rate** - Value: 2 * 9001 pps (we reserve 9000 pps
561 for latency measurements).
562 3. **final_trial_duration** - Value: 30.0 seconds.
563 4. **initial_trial_duration** - Value: 1.0 second.
564 5. **final_relative_width** - Value: 0.005 (0.5%).
565 6. **packet_loss_ratios** - Value: 0.0, 0.005 (0.0% for NDR, 0.5% for PDR).
566 7. **number_of_intermediate_phases** - Value: 2.
567 The value has been chosen based on limited experimentation to date.
568 More experimentation needed to arrive to clearer guidelines.
569 8. **timeout** - Limit for the overall search duration (for one search).
570 If MLRsearch oversteps this limit, it immediatelly declares the test failed,
571 to avoid wasting even more time on a misbehaving SUT.
572 Value: 600.0 (seconds).
573 9. **expansion_coefficient** - Width multiplier for external search.
574 Value: 4.0 (interval width is quadroupled).
575 Value of 2.0 is best for well-behaved SUTs, but value of 4.0 has been found
576 to decrease overall search time for worse-behaved SUT configurations,
577 contributing more to the overall set of different SUT configurations tested.
580 ## Example MLRsearch Run
583 The following list describes a search from a real test run in CSIT
584 (using the default input values as above).
586 * Initial phase, trial duration 1.0 second.
588 Measurement 1, intended load 18750000.0 pps (MTR),
589 measured loss ratio 0.7089514628479618 (valid upper bound for both NDR and PDR).
591 Measurement 2, intended load 5457160.071600716 pps (MRR),
592 measured loss ratio 0.018650817320118702 (new tightest upper bounds).
594 Measurement 3, intended load 5348832.933500009 pps (slightly less than MRR2
595 in preparation for first intermediate phase target interval width),
596 measured loss ratio 0.00964383362905351 (new tightest upper bounds).
598 * First intermediate phase starts, trial duration still 1.0 seconds.
600 Measurement 4, intended load 4936605.579021453 pps (no lower bound,
601 performing external search downwards, for NDR),
602 measured loss ratio 0.0 (valid lower bound for both NDR and PDR).
604 Measurement 5, intended load 5138587.208637197 pps (bisecting for NDR),
605 measured loss ratio 0.0 (new tightest lower bounds).
607 Measurement 6, intended load 5242656.244044665 pps (bisecting),
608 measured loss ratio 0.013523745379347257 (new tightest upper bounds).
610 * Both intervals are narrow enough.
611 * Second intermediate phase starts, trial duration 5.477225575051661 seconds.
613 Measurement 7, intended load 5190360.904111567 pps (initial bisect for NDR),
614 measured loss ratio 0.0023533920869969953 (NDR upper bound, PDR lower bound).
616 Measurement 8, intended load 5138587.208637197 pps (re-measuring NDR lower bound),
617 measured loss ratio 1.2080222912800403e-06 (new tightest NDR upper bound).
619 * The two intervals have separate bounds from now on.
621 Measurement 9, intended load 4936605.381062318 pps (external NDR search down),
622 measured loss ratio 0.0 (new valid NDR lower bound).
624 Measurement 10, intended load 5036583.888432355 pps (NDR bisect),
625 measured loss ratio 0.0 (new tightest NDR lower bound).
627 Measurement 11, intended load 5087329.903232804 pps (NDR bisect),
628 measured loss ratio 0.0 (new tightest NDR lower bound).
630 * NDR interval is narrow enough, PDR interval not ready yet.
632 Measurement 12, intended load 5242656.244044665 pps (re-measuring PDR upper bound),
633 measured loss ratio 0.0101174866190136 (still valid PDR upper bound).
635 * Also PDR interval is narrow enough, with valid bounds for this duration.
636 * Final phase starts, trial duration 30.0 seconds.
638 Measurement 13, intended load 5112894.3238511775 pps (initial bisect for NDR),
639 measured loss ratio 0.0 (new tightest NDR lower bound).
641 Measurement 14, intended load 5138587.208637197 (re-measuring NDR upper bound),
642 measured loss ratio 2.030389804256833e-06 (still valid PDR upper bound).
644 * NDR interval is narrow enough, PDR interval not yet.
646 Measurement 15, intended load 5216443.04126728 pps (initial bisect for PDR),
647 measured loss ratio 0.005620871287975237 (new tightest PDR upper bound).
649 Measurement 16, intended load 5190360.904111567 (re-measuring PDR lower bound),
650 measured loss ratio 0.0027629971184465604 (still valid PDR lower bound).
652 * PDR interval is also narrow enough.
654 * NDR_LOWER = 5112894.3238511775 pps; NDR_UPPER = 5138587.208637197 pps;
655 * PDR_LOWER = 5190360.904111567 pps; PDR_UPPER = 5216443.04126728 pps.
657 # IANA Considerations
661 # Security Considerations
663 Benchmarking activities as described in this memo are limited to
664 technology characterization of a DUT/SUT using controlled stimuli in a
665 laboratory environment, with dedicated address space and the constraints
666 specified in the sections above.
668 The benchmarking network topology will be an independent test setup and
669 MUST NOT be connected to devices that may forward the test traffic into
670 a production network or misroute traffic to the test management network.
672 Further, benchmarking is performed on a "black-box" basis, relying
673 solely on measurements observable external to the DUT/SUT.
675 Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
676 benchmarking purposes.Any implications for network security arising
677 from the DUT/SUT SHOULD be identical in the lab and in production
682 Many thanks to Alec Hothan of OPNFV NFVbench project for thorough
683 review and numerous useful comments and suggestions.