From: Vratko Polak Date: Wed, 3 Sep 2025 11:39:45 +0000 (+0200) Subject: feat(ietf): Edit MLRsearch draft 12 for nits X-Git-Url: https://gerrit.fd.io/r/gitweb?a=commitdiff_plain;h=6a74002a03e8ffb470950ff600ed8cd6e2a5c01c;p=csit.git feat(ietf): Edit MLRsearch draft 12 for nits + Updated Acknowledgement section. + Formatting. + Removed comments as they might interfere with formatting. + Fixed some typos. - Grammar errors around indefinite and definite articles still left. - Also: Capitalization, dots after abbeviations, other minor issues. Change-Id: Ic3477f619aabf1de018f8e64ce47d967fa900ae7 Signed-off-by: Maciek Konstantynowicz Signed-off-by: Vratko Polak --- diff --git a/docs/ietf/draft-ietf-bmwg-mlrsearch-12.md b/docs/ietf/draft-ietf-bmwg-mlrsearch-12.md index 00a7d88620..5c7b49d0d8 100644 --- a/docs/ietf/draft-ietf-bmwg-mlrsearch-12.md +++ b/docs/ietf/draft-ietf-bmwg-mlrsearch-12.md @@ -80,7 +80,7 @@ and comparability. MLRsearch is motivated by the pressing need to address the challenges of evaluating and testing the various data plane solutions, especially in -software- based networking systems based on Commercial Off-the-Shelf +software-based networking systems based on Commercial Off-the-Shelf (COTS) CPU hardware vs purpose-built ASIC / NPU / FPGA hardware. --- middle @@ -149,7 +149,7 @@ early MLRsearch implementations employed the following enhancements: 5. Apply several time-saving load selection heuristics that deliberately prevent the bounds from narrowing unnecessarily. -Enhacements 1, 2 and partly 4 are formalized as MLRsearch Specification +Enhancements 1, 2 and partly 4 are formalized as MLRsearch Specification within this document, other implementation details are out the scope. The remaining enhancements are treated as implementation details, @@ -214,9 +214,8 @@ mainly a binary search for [RFC2544] unconditionally compliant throughput. ## Long Search Duration -The proliferation of software DUTs, with frequent software updates and a - -number of different frame processing modes and configurations, +The proliferation of software DUTs, with frequent software updates +and a number of different frame processing modes and configurations, has increased both the number of performance tests required to verify the DUT update and the frequency of running those tests. This makes the overall test execution time even more important than before. @@ -274,7 +273,7 @@ that share the same CPUs, memory and I/O resources. Given that a SUT is a shared multi-tenant environment, the DUT might inadvertently experience interference from the operating system -or other software operating on the same server. +or from other software operating on the same server. Some of this interference can be mitigated. For instance, in multi-core CPU systems, pinning DUT program threads to @@ -371,7 +370,7 @@ no tolerance of a single frame loss) affect the throughput result as follows: The SUT behavior close to the noiseful end of its performance spectrum consists of rare occasions of significantly low performance, but the long trial duration makes those occasions not so rare on the trial level. -Therefore, the binary search results tend to wander away from the noiseless end +Therefore, the binary search results tend to spread away from the noiseless end of SUT performance spectrum, more frequently and more widely than shorter trials would, thus causing unacceptable throughput repeatability. @@ -416,7 +415,7 @@ Motivations are many: - Networking protocols tolerate frame loss better, compared to the time when [RFC1242] and [RFC2544] were specified. -- Increased link speeds require trials sending way more frames within the same duration, +- Increased link speeds require trials sending more frames within the same duration, increasing the chance of a small SUT performance fluctuation being enough to cause frame loss. @@ -526,7 +525,7 @@ complies with MLRsearch Specification. Some terms used in the specification are capitalized. It is just a stylistic choice for this document, reminding the reader this term is introduced, defined or explained -elsewhere in the document. Lowercase variants are equally valid. +elsewhere in the document. Lower case variants are equally valid. This document does not separate terminology from methodology. Terms are fully specified and discussed in their own subsections, under sections @@ -633,7 +632,7 @@ by calling Controller once for each benchmark. The Manager calls a Controller once, and the Controller then invokes the Measurer repeatedly -until Controler decides it has enough information to return outputs. +until Controller decides it has enough information to return outputs. The part during which the Controller invokes the Measurer is termed the Search. Any work the Manager performs either before invoking the @@ -963,7 +962,7 @@ this document uses a shorthand **Load**. interfaces, treating it as the same quantity expressed using different units. Each reported Trial Load value MUST state unambiguously whether it refers to (i) a single interface, (ii) a specified subset of -interfaces (e.g., such as all logical interfaces mapped to one physical +interfaces (such as all logical interfaces mapped to one physical port), or (iii) the total across every interface. For any aggregate load value, the report MUST also give the fixed conversion factor that links the per-interface and multi-interface load values. @@ -1031,7 +1030,7 @@ This is why the traffic profile is not part of the Trial Input.   : Specification of traffic properties included in the Traffic Profile is the responsibility of the Manager, but the specific configuration mechanisms -are outside of the scope of this docunment. +are outside of the scope of this document.   : Informally, implementations of the Manager and the Measurer @@ -1042,8 +1041,7 @@ Typically, Manager and Measurer implementations are tightly integrated.   : Integration efforts between independent Manager and Measurer implementations are outside of the scope of this document. -An example standardization effort is [Vassilev], -a draft at the time of writing. +An example standardization effort is [Vassilev].   : Examples of traffic properties include: @@ -1903,7 +1901,7 @@ Each instance is either a Regular Goal Result or an Irregular Goal Result. Discussion:   -: The Manager MUST be able to distinguish whether the instance is regular or not. +: The Manager MUST be able of distinguishing whether the instance is regular or not. ### Search Result @@ -2047,7 +2045,7 @@ as long as Goal Result instances are regular. Definition:   -: The Manager is a functional element that is reponsible for +: The Manager is a functional element that is responsible for provisioning other components, calling a Controller component once, and for creating the test report following the reporting format as defined in Section 26 of [RFC2544]. @@ -2239,9 +2237,7 @@ and its variance. ## Loss Ratios and Loss Inversion -The biggest - -difference between MLRsearch and [RFC2544] binary search +The biggest difference between MLRsearch and [RFC2544] binary search is in the goals of the search. [RFC2544] has a single goal, based on classifying a single full-length trial as either zero-loss or non-zero-loss. @@ -2259,9 +2255,7 @@ when the search is started with only one Search Goal instance. ### Multiple Goals and Loss Inversion -MLRsearch Specification - -supports multiple Search Goals, making the search procedure +MLRsearch Specification supports multiple Search Goals, making the search procedure more complicated compared to binary search with single goal, but most of the complications do not affect the final results much. Except for one phenomenon: Loss Inversion. @@ -2477,7 +2471,7 @@ and uses more intuitive names for the intermediate values. ## Load Classification Logic -Note: For explanation clarity variables are taged as (I)nput, +Note: For clarity of explanation, variables are tagged as (I)nput, (T)emporary, (O)utput. - Collect Trial Results: @@ -2697,14 +2691,21 @@ guidelines. Thank You Al for the close collaboration over the years, Your Mentor Your continuous unwavering encouragement full of empathy and energizing positive attitude. Al, You are dearly missed. -Thanks to Gabor Lencse, Giuseppe Fioccola and BMWG contributors for good -discussions and thorough reviews, guiding and helping us to improve the -clarity and formality of this document. +Thanks to Gabor Lencse, Giuseppe Fioccola, Carsten Rossenhövel and BMWG +contributors for good discussions and thorough reviews, guiding and +helping us to improve the clarity and formality of this document. Many thanks to Alec Hothan of the OPNFV NFVbench project for a thorough review and numerous useful comments and suggestions in the earlier versions of this document. +We are equally indebted to Mohamed Boucadair for a very thorough and +detailed AD review and providing many good comments and suggestions, +helping us make this document complete. + +Our appreciation is also extended to Shawn Emery, Yoshifumi Nishida, +David Dong, Nabeel Cocker and Lars Eggert for their reviews and valueable comments. + --- back # Load Classification Code @@ -3217,7 +3218,7 @@ One has Trial Loss Ratio of 0%, the other of 0.1%. - New remaining sum is 60s - 60s = 0s. - For second result (duration 60s, loss 0.1%): - Remaining sum is not larger than zero, exiting the loop. -- Current forwarding ratio was most recently set to 0%. +- Current loss ratio was most recently set to 0%. - Current forwarding ratio is one minus the current loss ratio, so 100%. - Conditional Throughput is the current forwarding ratio multiplied by the Load value. @@ -3251,7 +3252,7 @@ The result does not depend on the order of 0% loss trials. - After 61 trials, duration of 60x1s + 1x60s has been subtracted from 120s, leaving 0s. - For 62-th result (duration 60s, loss 0.1%): - Remaining sum is not larger than zero, exiting the loop. -- Current forwarding ratio was most recently set to 0%. +- Current loss ratio was most recently set to 0%. - Current forwarding ratio is one minus the current loss ratio, so 100%. - Conditional Throughput is the current forwarding ratio multiplied by the Load value. @@ -3282,7 +3283,7 @@ One has Trial Loss Ratio of 0%, the other of 0.1%. - Decrease the remaining sum by this trial's Trial Effective Duration. - New remaining sum is 36s - 60s = -24s. - No more trials (and remaining sum is not larger than zero), exiting loop. -- Current forwarding ratio was most recently set to 0.1%. +- Current loss ratio was most recently set to 0.1%. - Current forwarding ratio is one minus the current loss ratio, so 99.9%. - Conditional Throughput is the current forwarding ratio multiplied by the Load value. diff --git a/docs/ietf/draft-ietf-bmwg-mlrsearch-12.txt b/docs/ietf/draft-ietf-bmwg-mlrsearch-12.txt index 410f175649..8dbca9d99a 100644 --- a/docs/ietf/draft-ietf-bmwg-mlrsearch-12.txt +++ b/docs/ietf/draft-ietf-bmwg-mlrsearch-12.txt @@ -22,7 +22,7 @@ Abstract MLRsearch is motivated by the pressing need to address the challenges of evaluating and testing the various data plane solutions, - especially in software- based networking systems based on Commercial + especially in software-based networking systems based on Commercial Off-the-Shelf (COTS) CPU hardware vs purpose-built ASIC / NPU / FPGA hardware. @@ -146,7 +146,7 @@ Internet-Draft MLRsearch September 2025 5.3.1. Single Goal and Hard Bounds . . . . . . . . . . . . . 47 5.3.2. Multiple Goals and Loss Inversion . . . . . . . . . . 47 5.3.3. Conservativeness and Relevant Bounds . . . . . . . . 48 - 5.3.4. Consequences . . . . . . . . . . . . . . . . . . . . 49 + 5.3.4. Consequences . . . . . . . . . . . . . . . . . . . . 48 5.4. Exceed Ratio and Multiple Trials . . . . . . . . . . . . 49 5.5. Short Trials and Duration Selection . . . . . . . . . . . 50 5.6. Generalized Throughput . . . . . . . . . . . . . . . . . 50 @@ -174,7 +174,7 @@ Internet-Draft MLRsearch September 2025 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 58 10.1. Normative References . . . . . . . . . . . . . . . . . . 58 10.2. Informative References . . . . . . . . . . . . . . . . . 58 - Appendix A. Load Classification Code . . . . . . . . . . . . . . 59 + Appendix A. Load Classification Code . . . . . . . . . . . . . . 60 Appendix B. Conditional Throughput Code . . . . . . . . . . . . 61 Appendix C. Example Search . . . . . . . . . . . . . . . . . . . 63 C.1. Example Goals . . . . . . . . . . . . . . . . . . . . . . 64 @@ -270,7 +270,7 @@ Internet-Draft MLRsearch September 2025 5. Apply several time-saving load selection heuristics that deliberately prevent the bounds from narrowing unnecessarily. - Enhacements 1, 2 and partly 4 are formalized as MLRsearch + Enhancements 1, 2 and partly 4 are formalized as MLRsearch Specification within this document, other implementation details are out the scope. @@ -357,12 +357,10 @@ Internet-Draft MLRsearch September 2025 2.1. Long Search Duration The proliferation of software DUTs, with frequent software updates - and a - - number of different frame processing modes and configurations, has - increased both the number of performance tests required to verify the - DUT update and the frequency of running those tests. This makes the - overall test execution time even more important than before. + and a number of different frame processing modes and configurations, + has increased both the number of performance tests required to verify + the DUT update and the frequency of running those tests. This makes + the overall test execution time even more important than before. The throughput definition per [RFC2544] restricts the potential for time-efficiency improvements. The bisection method, when used in a @@ -386,6 +384,8 @@ Internet-Draft MLRsearch September 2025 * Set Max = line-rate and Min = a proven loss-free load. + * Run a single 60-s trial at the midpoint. + @@ -394,8 +394,6 @@ Konstantynowicz & Polak Expires 6 March 2026 [Page 7] Internet-Draft MLRsearch September 2025 - * Run a single 60-s trial at the midpoint. - * Zero-loss -> midpoint becomes new Min; any loss-> new Max. * Repeat until the Max-Min gap meets the desired precision, then @@ -430,7 +428,7 @@ Internet-Draft MLRsearch September 2025 Given that a SUT is a shared multi-tenant environment, the DUT might inadvertently experience interference from the operating system or - other software operating on the same server. + from other software operating on the same server. Some of this interference can be mitigated. For instance, in multi- core CPU systems, pinning DUT program threads to specific CPU cores @@ -445,6 +443,8 @@ Internet-Draft MLRsearch September 2025 + + Konstantynowicz & Polak Expires 6 March 2026 [Page 8] Internet-Draft MLRsearch September 2025 @@ -545,7 +545,7 @@ Internet-Draft MLRsearch September 2025 The SUT behavior close to the noiseful end of its performance spectrum consists of rare occasions of significantly low performance, but the long trial duration makes those occasions not so rare on the - trial level. Therefore, the binary search results tend to wander + trial level. Therefore, the binary search results tend to spread away from the noiseless end of SUT performance spectrum, more frequently and more widely than shorter trials would, thus causing unacceptable throughput repeatability. @@ -596,8 +596,8 @@ Internet-Draft MLRsearch September 2025 * Networking protocols tolerate frame loss better, compared to the time when [RFC1242] and [RFC2544] were specified. - * Increased link speeds require trials sending way more frames - within the same duration, increasing the chance of a small SUT + * Increased link speeds require trials sending more frames within + the same duration, increasing the chance of a small SUT performance fluctuation being enough to cause frame loss. * Because noise-related drops usually arrive in small bursts, their @@ -732,8 +732,8 @@ Internet-Draft MLRsearch September 2025 Some terms used in the specification are capitalized. It is just a stylistic choice for this document, reminding the reader this term is - introduced, defined or explained elsewhere in the document. - Lowercase variants are equally valid. + introduced, defined or explained elsewhere in the document. Lower + case variants are equally valid. This document does not separate terminology from methodology. Terms are fully specified and discussed in their own subsections, under @@ -863,7 +863,7 @@ Internet-Draft MLRsearch September 2025 once for each benchmark. The Manager calls a Controller once, and the Controller then invokes - the Measurer repeatedly until Controler decides it has enough + the Measurer repeatedly until Controller decides it has enough information to return outputs. The part during which the Controller invokes the Measurer is termed @@ -1234,11 +1234,11 @@ Konstantynowicz & Polak Expires 6 March 2026 [Page 22] Internet-Draft MLRsearch September 2025 - specified subset of interfaces (e.g., such as all logical - interfaces mapped to one physical port), or (iii) the total across - every interface. For any aggregate load value, the report MUST - also give the fixed conversion factor that links the per-interface - and multi-interface load values. + specified subset of interfaces (such as all logical interfaces + mapped to one physical port), or (iii) the total across every + interface. For any aggregate load value, the report MUST also + give the fixed conversion factor that links the per-interface and + multi-interface load values. The per-interface value remains the primary unit, consistent with prevailing practice in [RFC1242], [RFC2544], and [RFC2285]. @@ -1303,7 +1303,7 @@ Internet-Draft MLRsearch September 2025 Specification of traffic properties included in the Traffic Profile is the responsibility of the Manager, but the specific configuration mechanisms are outside of the scope of this - docunment. + document. Informally, implementations of the Manager and the Measurer must be aware of their common set of capabilities, so that Traffic @@ -1313,8 +1313,7 @@ Internet-Draft MLRsearch September 2025 Integration efforts between independent Manager and Measurer implementations are outside of the scope of this document. An - example standardization effort is [Vassilev], a draft at the time - of writing. + example standardization effort is [Vassilev]. Examples of traffic properties include: - Data link frame size - Fixed sizes as listed in Section 3.5 of [RFC1242] and in Section 9 @@ -1341,6 +1340,7 @@ Internet-Draft MLRsearch September 2025 + Konstantynowicz & Polak Expires 6 March 2026 [Page 24] Internet-Draft MLRsearch September 2025 @@ -2222,7 +2222,7 @@ Internet-Draft MLRsearch September 2025 Discussion: - The Manager MUST be able to distinguish whether the instance is + The Manager MUST be able of distinguishing whether the instance is regular or not. 4.8.5. Search Result @@ -2379,7 +2379,7 @@ Internet-Draft MLRsearch September 2025 Definition: - The Manager is a functional element that is reponsible for + The Manager is a functional element that is responsible for provisioning other components, calling a Controller component once, and for creating the test report following the reporting format as defined in Section 26 of [RFC2544]. @@ -2604,10 +2604,8 @@ Internet-Draft MLRsearch September 2025 5.3. Loss Ratios and Loss Inversion - The biggest - - difference between MLRsearch and [RFC2544] binary search is in the - goals of the search. [RFC2544] has a single goal, based on + The biggest difference between MLRsearch and [RFC2544] binary search + is in the goals of the search. [RFC2544] has a single goal, based on classifying a single full-length trial as either zero-loss or non- zero-loss. MLRsearch supports searching for multiple Search Goals at once, usually differing in their Goal Loss Ratio values. @@ -2623,8 +2621,10 @@ Internet-Draft MLRsearch September 2025 5.3.2. Multiple Goals and Loss Inversion - MLRsearch Specification - + MLRsearch Specification supports multiple Search Goals, making the + search procedure more complicated compared to binary search with + single goal, but most of the complications do not affect the final + results much. Except for one phenomenon: Loss Inversion. @@ -2634,11 +2634,6 @@ Konstantynowicz & Polak Expires 6 March 2026 [Page 47] Internet-Draft MLRsearch September 2025 - supports multiple Search Goals, making the search procedure more - complicated compared to binary search with single goal, but most of - the complications do not affect the final results much. Except for - one phenomenon: Loss Inversion. - Depending on Search Goal attributes, Load Classification results may be resistant to small amounts of Section Inconsistent Trial Results (Section 2.5). However, for larger amounts, a Load that is @@ -2681,6 +2676,11 @@ Internet-Draft MLRsearch September 2025 search. When they meet the stopping conditions, the Relevant Bounds are used in the output. +5.3.4. Consequences + + The consequence of the way the Relevant Bounds are defined is that + every Trial Result can have an impact on any current Relevant Bound + larger than that Trial Load, namely by becoming a new Upper Bound. @@ -2690,12 +2690,6 @@ Konstantynowicz & Polak Expires 6 March 2026 [Page 48] Internet-Draft MLRsearch September 2025 -5.3.4. Consequences - - The consequence of the way the Relevant Bounds are defined is that - every Trial Result can have an impact on any current Relevant Bound - larger than that Trial Load, namely by becoming a new Upper Bound. - This also applies when that Load is measured before another Load gets enough measurements to become a current Relevant Bound. @@ -2737,6 +2731,12 @@ Internet-Draft MLRsearch September 2025 also needs something that controls the number of trials needed. Therefore, each goal also has an attribute called Goal Duration Sum. + The meaning of a Goal Duration Sum (Section 4.6.2) is that when a + Load has (Full-Length) Trials whose Trial Effective Durations when + summed up give a value at least as big as the Goal Duration Sum + value, the Load is guaranteed to be classified either as an Upper + Bound or a Lower Bound for that Search Goal instance. + @@ -2746,12 +2746,6 @@ Konstantynowicz & Polak Expires 6 March 2026 [Page 49] Internet-Draft MLRsearch September 2025 - The meaning of a Goal Duration Sum (Section 4.6.2) is that when a - Load has (Full-Length) Trials whose Trial Effective Durations when - summed up give a value at least as big as the Goal Duration Sum - value, the Load is guaranteed to be classified either as an Upper - Bound or a Lower Bound for that Search Goal instance. - 5.5. Short Trials and Duration Selection MLRsearch requires each Search Goal to specify its Goal Final Trial @@ -2794,6 +2788,12 @@ Internet-Draft MLRsearch September 2025 how to generalize it for Loads with multiple Trials and a non-zero Goal Loss Ratio. + The clearest illustration - and the chief reason for adopting a + generalized throughput definition - is the presence of a hard + performance limit. + + + @@ -2802,10 +2802,6 @@ Konstantynowicz & Polak Expires 6 March 2026 [Page 50] Internet-Draft MLRsearch September 2025 - The clearest illustration - and the chief reason for adopting a - generalized throughput definition - is the presence of a hard - performance limit. - 5.6.1. Hard Performance Limit Even if bandwidth of a medium allows higher traffic forwarding @@ -2851,6 +2847,10 @@ Internet-Draft MLRsearch September 2025 Conditional Throughput values may have up to the Goal Loss Ratio relative difference. + Setting the Goal Width below the Goal Loss Ratio may cause the + Conditional Throughput for a larger Goal Loss Ratio to become smaller + than a Conditional Throughput for a goal with a lower Goal Loss + Konstantynowicz & Polak Expires 6 March 2026 [Page 51] @@ -2858,9 +2858,6 @@ Konstantynowicz & Polak Expires 6 March 2026 [Page 51] Internet-Draft MLRsearch September 2025 - Setting the Goal Width below the Goal Loss Ratio may cause the - Conditional Throughput for a larger Goal Loss Ratio to become smaller - than a Conditional Throughput for a goal with a lower Goal Loss Ratio, which is counter-intuitive, considering they come from the same Search. Therefore, it is RECOMMENDED to set the Goal Width to a value no lower than the Goal Loss Ratio of the higher-loss Search @@ -2887,7 +2884,7 @@ Internet-Draft MLRsearch September 2025 6.1. Load Classification Logic - Note: For explanation clarity variables are taged as (I)nput, + Note: For clarity of explanation, variables are tagged as (I)nput, (T)emporary, (O)utput. * Collect Trial Results: @@ -2905,7 +2902,10 @@ Internet-Draft MLRsearch September 2025 - Short high-loss sum is the sum (T) of Trial Effective Duration values of all short high-loss trials (I). + - Short low-loss sum is the sum (T) of Trial Effective Duration + values of all short low-loss trials (I). + * Derive goal-based ratios: @@ -2914,11 +2914,6 @@ Konstantynowicz & Polak Expires 6 March 2026 [Page 52] Internet-Draft MLRsearch September 2025 - - Short low-loss sum is the sum (T) of Trial Effective Duration - values of all short low-loss trials (I). - - * Derive goal-based ratios: - - Subceed ratio (T) is One minus the Goal Exceed Ratio (I). - Exceed coefficient (T) is the Goal Exceed Ratio divided by the @@ -2961,6 +2956,11 @@ Internet-Draft MLRsearch September 2025 * Classify the Load: + - The load is classified as an Upper Bound (O) if the optimistic + exceed ratio is larger than the Goal Exceed Ratio. + + - The load is classified as a Lower Bound (O) if the pessimistic + exceed ratio is not larger than the Goal Exceed Ratio. @@ -2970,12 +2970,6 @@ Konstantynowicz & Polak Expires 6 March 2026 [Page 53] Internet-Draft MLRsearch September 2025 - - The load is classified as an Upper Bound (O) if the optimistic - exceed ratio is larger than the Goal Exceed Ratio. - - - The load is classified as a Lower Bound (O) if the pessimistic - exceed ratio is not larger than the Goal Exceed Ratio. - - The load is classified as undecided (O) otherwise. 6.2. Conditional Throughput Logic @@ -3019,6 +3013,12 @@ Internet-Draft MLRsearch September 2025 * Compute Conditional Throughput + - Current forwarding ratio (T) is One minus the current loss + ratio. + + - Conditional Throughput (T) is the current forwarding ratio + multiplied by the Load value. + Konstantynowicz & Polak Expires 6 March 2026 [Page 54] @@ -3026,12 +3026,6 @@ Konstantynowicz & Polak Expires 6 March 2026 [Page 54] Internet-Draft MLRsearch September 2025 - - Current forwarding ratio (T) is One minus the current loss - ratio. - - - Conditional Throughput (T) is the current forwarding ratio - multiplied by the Load value. - 6.2.1. Conditional Throughput and Load Classification Conditional Throughput and results of Load Classification overlap but @@ -3074,6 +3068,12 @@ Internet-Draft MLRsearch September 2025 events, even the rare ones, and thus the expert can do probabilistic predictions about future Trial Outputs. + When several outcomes are possible, the expert can assess probability + of each outcome. + + + + @@ -3082,9 +3082,6 @@ Konstantynowicz & Polak Expires 6 March 2026 [Page 55] Internet-Draft MLRsearch September 2025 - When several outcomes are possible, the expert can assess probability - of each outcome. - 6.3.2. Exceed Probability When the Controller selects new Trial Duration and Trial Load, and @@ -3133,6 +3130,9 @@ Internet-Draft MLRsearch September 2025 + + + Konstantynowicz & Polak Expires 6 March 2026 [Page 56] Internet-Draft MLRsearch September 2025 @@ -3180,9 +3180,9 @@ Internet-Draft MLRsearch September 2025 encouragement full of empathy and energizing positive attitude. Al, You are dearly missed. - Thanks to Gabor Lencse, Giuseppe Fioccola and BMWG contributors for - good discussions and thorough reviews, guiding and helping us to - improve the clarity and formality of this document. + Thanks to Gabor Lencse, Giuseppe Fioccola, Carsten Rossenhoevel and + BMWG contributors for good discussions and thorough reviews, guiding + and helping us to improve the clarity and formality of this document. @@ -3198,6 +3198,14 @@ Internet-Draft MLRsearch September 2025 thorough review and numerous useful comments and suggestions in the earlier versions of this document. + We are equally indebted to Mohamed Boucadair for a very thorough and + detailed AD review and providing many good comments and suggestions, + helping us make this document complete. + + Our appreciation is also extended to Shawn Emery, Yoshifumi Nishida, + David Dong, Nabeel Cocker and Lars Eggert for their reviews and + valueable comments. + 10. References 10.1. Normative References @@ -3231,6 +3239,17 @@ Internet-Draft MLRsearch September 2025 . + + + + + + +Konstantynowicz & Polak Expires 6 March 2026 [Page 58] + +Internet-Draft MLRsearch September 2025 + + [Lencze-Kovacs-Shima] "Gaming with the Throughput and the Latency Benchmarking Measurement Procedures of RFC 2544", n.d., @@ -3242,14 +3261,6 @@ Internet-Draft MLRsearch September 2025 . - - - -Konstantynowicz & Polak Expires 6 March 2026 [Page 58] - -Internet-Draft MLRsearch September 2025 - - [Ott-Mathis-Semke-Mahdavi] "The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm", n.d., @@ -3288,6 +3299,13 @@ Internet-Draft MLRsearch September 2025 . + + +Konstantynowicz & Polak Expires 6 March 2026 [Page 59] + +Internet-Draft MLRsearch September 2025 + + [Y.1564] "Y.1564", n.d., . @@ -3299,13 +3317,6 @@ Appendix A. Load Classification Code Any Trial Load value can be classified, according to a given Search Goal (Section 4.6.7) instance. - - -Konstantynowicz & Polak Expires 6 March 2026 [Page 59] - -Internet-Draft MLRsearch September 2025 - - The algorithm uses (some subsets of) the set of all available Trial Results from Trials measured at a given Load at the end of the Search. @@ -3342,6 +3353,15 @@ Internet-Draft MLRsearch September 2025 Duration and with Trial Loss Ratio not higher than the Goal Loss Ratio (across Full-Length Low-Loss Trials). + + + + +Konstantynowicz & Polak Expires 6 March 2026 [Page 60] + +Internet-Draft MLRsearch September 2025 + + * full_length_high_loss_s: Sum of Trial Effective Durations across Trials with Trial Duration at least equal to the Goal Final Trial Duration and with Trial Loss Ratio higher than the Goal Loss Ratio @@ -3352,16 +3372,6 @@ Internet-Draft MLRsearch September 2025 with Trial Loss Ratio not higher than the Goal Loss Ratio (across Short Low-Loss Trials). - - - - - -Konstantynowicz & Polak Expires 6 March 2026 [Page 60] - -Internet-Draft MLRsearch September 2025 - - * short_high_loss_s: Sum of Trial Effective Durations across Trials with Trial Duration shorter than the Goal Final Trial Duration and with Trial Loss Ratio higher than the Goal Loss Ratio (across @@ -3401,6 +3411,13 @@ Appendix B. Conditional Throughput Code The block at the end of this appendix holds pseudocode which computes a value stored as variable conditional_throughput. + + +Konstantynowicz & Polak Expires 6 March 2026 [Page 61] + +Internet-Draft MLRsearch September 2025 + + Although presented as pseudocode, the listing is syntactically valid Python and can be executed without modification. @@ -3409,15 +3426,6 @@ Appendix B. Conditional Throughput Code _sum, and variables holding effective quantities start in effect_ instead of effective_. - - - - -Konstantynowicz & Polak Expires 6 March 2026 [Page 61] - -Internet-Draft MLRsearch September 2025 - - The pseudocode expects the following variables to hold the following values: @@ -3461,14 +3469,6 @@ Internet-Draft MLRsearch September 2025 - - - - - - - - Konstantynowicz & Polak Expires 6 March 2026 [Page 62] Internet-Draft MLRsearch September 2025 @@ -4170,7 +4170,7 @@ Internet-Draft MLRsearch September 2025 * Remaining sum is not larger than zero, exiting the loop. - * Current forwarding ratio was most recently set to 0%. + * Current loss ratio was most recently set to 0%. * Current forwarding ratio is one minus the current loss ratio, so 100%. @@ -4231,7 +4231,7 @@ Internet-Draft MLRsearch September 2025 - Remaining sum is not larger than zero, exiting the loop. - * Current forwarding ratio was most recently set to 0%. + * Current loss ratio was most recently set to 0%. * Current forwarding ratio is one minus the current loss ratio, so 100%. @@ -4295,7 +4295,7 @@ Internet-Draft MLRsearch September 2025 * No more trials (and remaining sum is not larger than zero), exiting loop. - * Current forwarding ratio was most recently set to 0.1%. + * Current loss ratio was most recently set to 0.1%. * Current forwarding ratio is one minus the current loss ratio, so 99.9%. diff --git a/docs/ietf/draft-ietf-bmwg-mlrsearch-12.xml b/docs/ietf/draft-ietf-bmwg-mlrsearch-12.xml index 0598881865..4901ff3ffd 100644 --- a/docs/ietf/draft-ietf-bmwg-mlrsearch-12.xml +++ b/docs/ietf/draft-ietf-bmwg-mlrsearch-12.xml @@ -58,7 +58,7 @@ and comparability. MLRsearch is motivated by the pressing need to address the challenges of evaluating and testing the various data plane solutions, especially in -software- based networking systems based on Commercial Off-the-Shelf +software-based networking systems based on Commercial Off-the-Shelf (COTS) CPU hardware vs purpose-built ASIC / NPU / FPGA hardware. @@ -139,7 +139,7 @@ in Section 3.6.2 of , to initialize bounds. prevent the bounds from narrowing unnecessarily. -Enhacements 1, 2 and partly 4 are formalized as MLRsearch Specification +Enhancements 1, 2 and partly 4 are formalized as MLRsearch Specification within this document, other implementation details are out the scope. The remaining enhancements are treated as implementation details, @@ -207,9 +207,8 @@ mainly a binary search for unconditionally compli
Long Search Duration -The proliferation of software DUTs, with frequent software updates and a - -number of different frame processing modes and configurations, +The proliferation of software DUTs, with frequent software updates +and a number of different frame processing modes and configurations, has increased both the number of performance tests required to verify the DUT update and the frequency of running those tests. This makes the overall test execution time even more important than before. @@ -274,7 +273,7 @@ that share the same CPUs, memory and I/O resources. Given that a SUT is a shared multi-tenant environment, the DUT might inadvertently experience interference from the operating system -or other software operating on the same server. +or from other software operating on the same server. Some of this interference can be mitigated. For instance, in multi-core CPU systems, pinning DUT program threads to @@ -372,7 +371,7 @@ no tolerance of a single frame loss) affect the throughput result as follows:The SUT behavior close to the noiseful end of its performance spectrum consists of rare occasions of significantly low performance, but the long trial duration makes those occasions not so rare on the trial level. -Therefore, the binary search results tend to wander away from the noiseless end +Therefore, the binary search results tend to spread away from the noiseless end of SUT performance spectrum, more frequently and more widely than shorter trials would, thus causing unacceptable throughput repeatability. @@ -423,7 +422,7 @@ non-zero loss ratio as the goal for their load search. Networking protocols tolerate frame loss better, compared to the time when and were specified. - Increased link speeds require trials sending way more frames within the same duration, + Increased link speeds require trials sending more frames within the same duration, increasing the chance of a small SUT performance fluctuation being enough to cause frame loss. Because noise-related drops usually arrive in small bursts, their @@ -537,7 +536,7 @@ complies with MLRsearch Specification. Some terms used in the specification are capitalized. It is just a stylistic choice for this document, reminding the reader this term is introduced, defined or explained -elsewhere in the document. Lowercase variants are equally valid. +elsewhere in the document. Lower case variants are equally valid. This document does not separate terminology from methodology. Terms are fully specified and discussed in their own subsections, under sections @@ -646,7 +645,7 @@ by calling Controller once for each benchmark. The Manager calls a Controller once, and the Controller then invokes the Measurer repeatedly -until Controler decides it has enough information to return outputs. +until Controller decides it has enough information to return outputs. The part during which the Controller invokes the Measurer is termed the Search. Any work the Manager performs either before invoking the @@ -1049,7 +1048,7 @@ this document uses a shorthand Load. interfaces, treating it as the same quantity expressed using different units. Each reported Trial Load value MUST state unambiguously whether it refers to (i) a single interface, (ii) a specified subset of -interfaces (e.g., such as all logical interfaces mapped to one physical +interfaces (such as all logical interfaces mapped to one physical port), or (iii) the total across every interface. For any aggregate load value, the report MUST also give the fixed conversion factor that links the per-interface and multi-interface load values. @@ -1142,7 +1141,7 @@ This is why the traffic profile is not part of the Trial Input.
Specification of traffic properties included in the Traffic Profile is the responsibility of the Manager, but the specific configuration mechanisms -are outside of the scope of this docunment. +are outside of the scope of this document.
 
@@ -1155,8 +1154,7 @@ Typically, Manager and Measurer implementations are tightly integrated.
Integration efforts between independent Manager and Measurer implementations are outside of the scope of this document. -An example standardization effort is , -a draft at the time of writing. +An example standardization effort is .
 
@@ -2327,7 +2325,7 @@ Each instance is either a Regular Goal Result or an Irregular Goal Result.
 
- The Manager MUST be able to distinguish whether the instance is regular or not. + The Manager MUST be able of distinguishing whether the instance is regular or not.
@@ -2523,7 +2521,7 @@ as long as Goal Result instances are regular.
 
- The Manager is a functional element that is reponsible for + The Manager is a functional element that is responsible for provisioning other components, calling a Controller component once, and for creating the test report following the reporting format as defined in Section 26 of . @@ -2740,9 +2738,7 @@ and its variance.
Loss Ratios and Loss Inversion -The biggest - -difference between MLRsearch and binary search +The biggest difference between MLRsearch and binary search is in the goals of the search. has a single goal, based on classifying a single full-length trial as either zero-loss or non-zero-loss. @@ -2761,9 +2757,7 @@ when the search is started with only one Search Goal instance.
Multiple Goals and Loss Inversion -MLRsearch Specification - -supports multiple Search Goals, making the search procedure +MLRsearch Specification supports multiple Search Goals, making the search procedure more complicated compared to binary search with single goal, but most of the complications do not affect the final results much. Except for one phenomenon: Loss Inversion. @@ -2989,7 +2983,7 @@ and uses more intuitive names for the intermediate values.
Load Classification Logic -Note: For explanation clarity variables are taged as (I)nput, +Note: For clarity of explanation, variables are tagged as (I)nput, (T)emporary, (O)utput. @@ -3218,14 +3212,21 @@ guidelines. Thank You Al for the close collaboration over the years, Your Mentor Your continuous unwavering encouragement full of empathy and energizing positive attitude. Al, You are dearly missed. -Thanks to Gabor Lencse, Giuseppe Fioccola and BMWG contributors for good -discussions and thorough reviews, guiding and helping us to improve the -clarity and formality of this document. +Thanks to Gabor Lencse, Giuseppe Fioccola, Carsten Rossenhoevel and BMWG +contributors for good discussions and thorough reviews, guiding and +helping us to improve the clarity and formality of this document. Many thanks to Alec Hothan of the OPNFV NFVbench project for a thorough review and numerous useful comments and suggestions in the earlier versions of this document. +We are equally indebted to Mohamed Boucadair for a very thorough and +detailed AD review and providing many good comments and suggestions, +helping us make this document complete. + +Our appreciation is also extended to Shawn Emery, Yoshifumi Nishida, +David Dong, Nabeel Cocker and Lars Eggert for their reviews and valueable comments. +
@@ -3332,7 +3333,7 @@ versions of this document. - +
Load Classification Code @@ -4283,7 +4284,7 @@ One has Trial Loss Ratio of 0%, the other of 0.1%. For second result (duration 60s, loss 0.1%): Remaining sum is not larger than zero, exiting the loop. - Current forwarding ratio was most recently set to 0%. + Current loss ratio was most recently set to 0%. Current forwarding ratio is one minus the current loss ratio, so 100%. Conditional Throughput is the current forwarding ratio multiplied by the Load value. Conditional Throughput is one million frames per second. @@ -4324,7 +4325,7 @@ They are ordered like this: Remaining sum is not larger than zero, exiting the loop. - Current forwarding ratio was most recently set to 0%. + Current loss ratio was most recently set to 0%. Current forwarding ratio is one minus the current loss ratio, so 100%. Conditional Throughput is the current forwarding ratio multiplied by the Load value. Conditional Throughput is one million frames per second. @@ -4360,7 +4361,7 @@ One has Trial Loss Ratio of 0%, the other of 0.1%. New remaining sum is 36s - 60s = -24s. No more trials (and remaining sum is not larger than zero), exiting loop. - Current forwarding ratio was most recently set to 0.1%. + Current loss ratio was most recently set to 0.1%. Current forwarding ratio is one minus the current loss ratio, so 99.9%. Conditional Throughput is the current forwarding ratio multiplied by the Load value. Conditional Throughput is 999 thousand frames per second. @@ -4377,799 +4378,802 @@ is smaller than Conditional Throughput of the other two goals.