--- /dev/null
+---
+title: Multiple Loss Ratio Search
+abbrev: MLRsearch
+docname: draft-ietf-bmwg-mlrsearch-05
+date: 2023-10-23
+
+ipr: trust200902
+area: ops
+wg: Benchmarking Working Group
+kw: Internet-Draft
+cat: info
+
+coding: us-ascii
+pi: # can use array (if all yes) or hash here
+ toc: yes
+ sortrefs: # defaults to yes
+ symrefs: yes
+
+author:
+ -
+ ins: M. Konstantynowicz
+ name: Maciek Konstantynowicz
+ org: Cisco Systems
+ email: mkonstan@cisco.com
+ -
+ ins: V. Polak
+ name: Vratko Polak
+ org: Cisco Systems
+ email: vrpolak@cisco.com
+
+normative:
+ RFC1242:
+ RFC2285:
+ RFC2544:
+ RFC9004:
+
+informative:
+ TST009:
+ target: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf
+ title: "TST 009"
+ FDio-CSIT-MLRsearch:
+ target: https://csit.fd.io/cdocs/methodology/measurements/data_plane_throughput/mlr_search/
+ title: "FD.io CSIT Test Methodology - MLRsearch"
+ date: 2023-10
+ PyPI-MLRsearch:
+ target: https://pypi.org/project/MLRsearch/0.4.1/
+ title: "MLRsearch 0.4.1, Python Package Index"
+ date: 2021-07
+
+--- abstract
+
+This document proposes extensions to [RFC2544] throughput search by
+defining a new methodology called Multiple Loss Ratio search
+(MLRsearch). The main objectives of MLRsearch are to minimize the
+total search duration, to support searching for multiple loss ratios
+and to improve results repeatability and comparability.
+
+The main motivation behind extending [RFC2544] is the new set of challenges
+and requirements posed by evaluating and testing software based networking
+systems, specifically their data planes.
+
+MLRsearch offers several ways to address these challenges, giving user
+configuration options to select their preferred way.
+
+--- middle
+
+{::comment}
+ As we use kramdown to convert from markdown,
+ we use this way of marking comments not to be visible in rendered draft.
+ https://stackoverflow.com/a/42323390
+ If other engine is used, convert to this way:
+ https://stackoverflow.com/a/20885980
+{:/comment}
+
+# Purpose and Scope
+
+The purpose of this document is to describe Multiple Loss Ratio search
+(MLRsearch), a data plane throughput search methodology optimized for software
+networking DUTs.
+
+Applying vanilla [RFC2544] throughput bisection to software DUTs
+results in a number of problems:
+
+- Binary search takes too long as most of trials are done far from the
+ eventually found throughput.
+- The required final trial duration and pauses between trials
+ prolong the overall search duration.
+- Software DUTs show noisy trial results,
+ leading to a big spread of possible discovered throughput values.
+- Throughput requires loss of exactly zero frames, but the industry
+ frequently allows for small but non-zero losses.
+- The definition of throughput is not clear when trial results are inconsistent.
+
+MLRsearch library aims to address these problems by applying the following set
+of enhancements:
+
+- Allow multiple shorter trials instead of one big trial per load.
+ - Optionally, tolerate a percentage of trial result with higher loss.
+- Allow searching for multiple search goals, with differing loss ratios.
+ - Any trial result can affect each search goal in principle.
+- Insert multiple coarse targets for each search goal, earlier ones need
+ to spend less time on trials.
+ - Earlier targets also aim for lesser precision.
+ - Use Forwarding Rate (FR) at maximum offered load
+ [RFC2285] (section 3.6.2) to initialize the initial targets.
+- Take care when dealing with inconsistent trial results.
+ - Reported throughput is smaller than smallest load with high loss.
+ - Smaller load candidates are measured first.
+- Apply several load selection heuristics to save even more time
+ by trying hard to avoid unnecessarily narrow bounds.
+
+Some of these enhancements are formalized as MLRsearch specification,
+the remaining enhancements are treated as implementation details,
+thus achieving high comparability without limiting future improvements.
+
+MLRsearch configuration options are flexible enough to
+support both conservative settings and aggressive settings.
+Where the conservative settings lead to results
+unconditionally compliant with [RFC2544],
+but longer search duration and worse repeatability.
+Conversely, aggressive settings lead to shorter search duration
+and better repeatability, but the results are not compliant with [RFC2544].
+
+No part of [RFC2544] is intended to be obsoleted by this document.
+
+# Identified Problems
+
+This chapter describes the problems affecting usability
+of various preformance testing methodologies,
+mainly a binary search for [RFC2544] unconditionally compliant throughput.
+
+The last chapter will summarize how the problems are addressed,
+the middle chapters provide explanations and definitions needed for that.
+
+## Long Search Duration
+
+Emergence of software DUTs, with frequent software updates and a
+number of different frame processing modes and configurations,
+has increased both the number of performance tests requred to verify DUT update
+and the frequency of running those tests.
+This makes the overall test execution time even more important than before.
+
+In the context of characterising particular DUT's network performance,
+this calls for improving the time efficiency of throughput search.
+A vanilla bisection (at 60sec trial duration for unconditional [RFC2544]
+compliance) is slow, because most trials spend time quite far from the
+eventual throughput.
+
+[RFC2544] does not specify any stopping condition for throughput search,
+so users can trade-off between search duration and achieved precision.
+But, due to logarithmic nature of bisection, even small improvement
+in search duration needs relatively big sacrifice in the precision of the
+discovered throughput.
+
+## DUT in SUT
+
+[RFC2285] defines:
+- DUT as
+ - The network forwarding device to which stimulus is offered and
+ response measured [RFC2285] (section 3.1.1).
+- SUT as
+ - The collective set of network devices to which stimulus is offered
+ as a single entity and response measured [RFC2285] (section 3.1.2).
+
+[RFC2544] specifies a test setup with an external tester stimulating the
+networking system, treating it either as a single DUT, or as a system
+of devices, an SUT.
+
+In case of software networking, the SUT consists nt only of the DUT
+as a software program processing frames, but also of
+a server hardware and operating system functions,
+with server hardware resources shared across all programs
+and the operating system running on the same server.
+
+The DUT is effectively nested within the rest of the SUT.
+
+Due to a shared multi-tenant nature of SUT, DUT is subject to
+possible interference coming from the operating system and any other
+software running on the same server. Some sources of such interference
+can be to some degree eliminated, e.g. by pinning DUT program threads
+to specific CPU cores and isolating those cores to avoid context switching.
+But some level of adverse effects may remain even after
+all such reasonable precautions are applied.
+These effects affect DUT's network performance negatively.
+As the effects are hard to predict in general, they have impact similar to
+what other engineering disciplines define as a noise.
+Thus, all such effects are called an SUT noise.
+
+DUT can also exhibit fluctuating performance itself, for reasons
+not related to the rest of SUT, for example due to pauses in execution
+as needed for internal stateful processing. In many cases this
+may be an expected per-design behavior, as it would be observable even
+in a hypothetical scenario where all sources of SUT noise are eliminated.
+Such behavior affects trial results in a way similar to SUT noise.
+As the two phenomenons are hard to destinguish,
+this document uses the word noise as a shorthand covering both
+this internal DUT performance fluctuations and genuine SUT noise.
+
+A simple model of SUT performance consists of an idealized noiseless performance,
+and additional noise effects. The noiseless performance is assumed to be constant,
+all observed performance variations are due to noise.
+The impact of the noise can vary in time, sometimes wildly,
+even within a single trial.
+The noise can sometimes be negligible, but frequently
+it lowers the observed SUT performance as observed in trial results.
+
+In this model, SUT does not have a single performance value, it has a spectrum.
+One end of the spectrum is the idealized noiseless performance value,
+the other end can be called a noiseful performance. In practice, trial results
+close to the noiseful end of the spectrum happen only rarely.
+The worse the performance value is, the more rarely it is seen in a trial.
+Therefore, the extreme noiseful end of SUT spectrum is not really observable
+among trial results. Also, the extreme noiseless end of SUT spectrum
+is unlikely to be observable, this time because some small noise effects
+are likely to occur multiple times during a trial.
+
+Unless specified otherwise, this document talks about potentially observable
+ends of the SUT performance spectrum, not about the extreme ones.
+
+Focusing on DUT, the benchmarking effort should aim
+at eliminating only the SUT noise from SUT measurements.
+In practice that is not really possible, as based on authors experience
+and available literature, there are no realistic enough models
+able to distinguish SUT noise from DUT fluctuations.
+
+However, assuming that a well-constructed SUT has the DUT as its
+performance bottleneck, the DUT ideal noiseless performance can be defined
+as the noiseless end of SUT performance spectrum. At least for
+throughput. For other performance quantities such as latency there may be an
+additive difference.
+
+Note that by this definition, DUT noiseless performance
+also minimizes the impact of DUT fluctuations, as much as realistically possible
+for a given trial duration.
+
+In this document, we reduce the DUT in SUT problem to estimating
+the noiseless end of SUT performance spectrum from a limited number of
+trial results.
+
+Any improvements to throughput search algorithm, aimed for better
+dealing with software networking SUT and DUT setup, should employ
+strategies recognizing the presence of SUT noise, and allow discovery of
+(proxies for) DUT noiseless performance
+at different levels of sensitivity to SUT noise.
+
+## Repeatability and Comparability
+
+[RFC2544] does not suggest to repeat throughput search. And from just one
+discovered throughput value, it cannot be determined how repeatable that value is.
+In practice, poor repeatability is also the main cause of poor
+comparability, that is different benchmarking teams can test the same SUT
+but get throughput values differing more than expected from search precision.
+
+[RFC2544] throughput requirements (60 seconds trial and
+no tolerance of a single frame loss) affect the throughput results
+in the following way.
+The SUT behavior close to the noiseful end of its performance spectrum
+consists of rare occasions of significantly low performance,
+but the long trial duration makes those occasions not so rare on the trial level.
+Therefore, the binary search results tend to wander away from the noiseless end
+of SUT performance spectrum, more frequently and more widely than shorter
+trials would, thus resulting in poor throughput repeatability.
+
+The repeatability problem can be addressed by defining a search procedure
+which reports more stable results,
+even if they can no longer be called throughput in [RFC2544] sense.
+According to the SUT performance spectrum model, better repeatability
+will be at the noiseless end of the spectrum.
+Therefore, solutions to the DUT in SUT problem
+will help also with the repeatability problem.
+
+Conversely, any alteration to [RFC2544] throughput search
+that improves repeatability should be considered
+as less dependent on the SUT noise.
+
+An alternative option is to simply run a search multiple times, and report some
+statistics (e.g. average and standard deviation). This can be used
+for a subset of tests deemed more important,
+but it makes the search duration problem even more pronounced.
+
+## Throughput with Non-Zero Loss
+
+[RFC1242] (section 3.17) defines throughput as:
+ The maximum rate at which none of the offered frames
+ are dropped by the device.
+
+Then, it says:
+ Since even the loss of one frame in a
+ data stream can cause significant delays while
+ waiting for the higher level protocols to time out,
+ it is useful to know the actual maximum data
+ rate that the device can support.
+
+Contrary to that, many benchmarking teams settle with small, non-zero
+loss ratio as the goal for a their load search.
+
+Motivations are many:
+
+- Modern protocols tolerate frame loss better,
+ compared to the time when [RFC1242] and [RFC2544] were specified.
+
+- Trials nowadays send way more frames within the same duration,
+ increasing the chance small SUT performance fluctuatios
+ is enough to cause frame loss.
+
+- Small bursts of frame loss caused by noise have otherwise smaller impact
+ on the average frame loss ratio ovserved in the trial,
+ as during other parts of the same trial the SUT may work mroe closely
+ to its noiseless performance, thus perhaps lowering the trial loss ratio
+ below the goal loss ratio value.
+
+- If an approximation of the SUT noise impact on the trial loss ratio is known,
+ it can be set as the goal loss ratio.
+
+Regardless of validity of any and all similar motivations,
+support for non-zero loss goals makes any search algorithm more user friendly.
+[RFC2544] throughput is not user friendly in this regard.
+
+Assuming users are allowed to specify the goal loss ratio value,
+the usefulness is enhanced even more if users can specify multiple
+loss ratio values, especially when a single search can find all relevant bounds.
+
+Searching for multiple search goals also helps to describe the SUT performance
+spectrum better than a single search goal result.
+For example, repeated wide gap between zero and non-zero loss loads
+indicates the noise has a large impact on the observed performance,
+which is not evident from a single goal load search procedure result.
+
+It is easy to modify the vanilla bisection to find a lower bound
+for intended load that satisfies a non-zero goal loss ratio.
+But it is not that obvious how to search for multiple goals at once,
+hence the support for multiple search goals remains a problem.
+
+## Inconsistent Trial Results
+
+While performing throughput search by executing a sequence of
+measurement trials, there is a risk of encountering inconsistencies
+between trial results.
+
+The plain bisection never encounters inconsistent trials.
+But [RFC2544] hints about possibility of inconsistent trial results,
+in two places in its text.
+The first place is section 24, where full trial durations are required,
+presumably because they can be inconsistent with results
+from shorter trial durations.
+The second place is section 26.3, where two successive zero-loss trials
+are recommended, presumably because after one zero-loss trial
+there can be subsequent inconsistent non-zero-loss trial.
+
+Examples include:
+
+- A trial at the same load (same or different trial duration) results
+ in a different trial loss ratio.
+- A trial at higher load (same or different trial duration) results
+ in a smaller trial loss ratio.
+
+Any robust throughput search algorithm needs to decide how to continue
+the search in presence of such inconsistencies.
+Definitions of throughput in [RFC1242] and [RFC2544] are not specific enough
+to imply a unique way of handling such inconsistencies.
+
+Ideally, there will be a definition of a new quantity which both generalizes
+throughput for non-zero-loss (and other possible repeatibility enhancements),
+while being precise enough to force a specific way to resolve trial result
+inconsistencies.
+But until such definition is agreed upon, the correct way to handle
+inconsistent trial results remains an open problem.
+
+# MLRsearch Specification
+
+This chapter focuses on technical definitions needed for evaluating
+whether a particular test procedure adheres to MLRsearch specification.
+
+For motivations, explanations, and other comments see other chapters.
+
+## MLRsearch Architecture
+
+MLRsearch architecture consists of three main components:
+the manager, the controller and the measurer.
+Presence of other components (mainly the SUT) is also implied.
+
+While the manager and the measurer can be seen a abstractions
+present in any testing procedure, the behavior of the controller
+is what distinguishes MLRsearch algorithms from other search procedures.
+
+### Measurer
+
+The measurer is the component which performs one trial
+as described in [RFC2544] section 23, when requested by the controller.
+
+Specifically, one call to the measurer accepts a trial load value
+and trial duration value, performs the trial, and returns
+the measured trial loss ratio, and optionally a different duration value.
+
+It is responsibility of the measurer to uphold any requirements
+and assumptions present in MLRsearch specification
+(e.g. trial forwarding ratio not being larger than one).
+Implementers have some freedom, for example in the way they deal with
+duplicated frames, or what to return if tester sent zero frames towards SUT.
+Implementations are RECOMMENDED to document their behavior
+related to such freedoms in as detailed way as possible
+
+Implemenations MUST document any deviations from RFC documents,
+for example if the wait time around traffic
+is shorter than what [RFC2544] section 23 specifies.
+
+### Controller
+
+The controller is the component of MLRsearch architecture
+that is called by the manager (just once), calls the measurer
+(usually multiple times in succession),
+and returns the result of the search to the manager.
+
+The only required argument in the call to the controller
+is a list of search goals. For the structure of the search result,
+see subsection Search Result.
+
+### Manager
+
+The manager is the component that initializes SUT, traffic generator
+(tester in [RFC2544]), the measurer and the controller
+with intended configurations. It then hands over the execution
+to the controller and receives its result.
+
+Creation of reports of appropriate format can also be understood
+as the responsibility of the manager.
+
+## Units
+
+The specification deals with physical quantities, so it is assumed
+each numeric value is accompanied by an appropriate physical unit.
+
+The specification does not state which unit is appropriate,
+but implementations MUST make it explicit which unit is used
+for each value provided or received by the user.
+
+For example, load quantities (including the conditional throughput)
+returned by the controller are defined to be based on single-interface
+(unidirectional) loads. For bidirectional traffic, users are likely
+to expect bidirectional throughput quantities, so the manager is responsible
+for making its report clear.
+
+## SUT
+
+As defined in [RFC2285]:
+The collective set of network devices to which stimulus is offered
+as a single entity and response measured.
+
+## Trial
+
+A trial is the part of test described in [RFC2544] section 23.
+
+### Trial Load
+
+Trial load is the intended constant load for a trial.
+
+Load is the quantity implied by Constant Load of [RFC1242],
+Data Rate of [RFC2544] and Intended Load of [RFC2285].
+All three specify this value applies to one (input or output) interface.
+
+### Trial Duration
+
+Trial duration is the intended duration of the traffic for a trial.
+
+In general, this general quantity does not include any preparation nor waiting
+described in section 23 of [RFC2544].
+
+However, the measurer MAY return a duration value different
+from the intended duration. This may be useful for users
+who want to control the overal search duration, not just the traffic part of it.
+The manager MUST report how does the measurer computes the returned duration
+values in that case.
+
+### Trial Forwarding Ratio
+
+Trial forwarding ratio is dimensionless floating point value,
+assumed to be between 0.0 and 1.0, both including.
+It is computed as the number of frames forwarded by SUT, divided by
+the number of frames that should have been forwarded during the trial.
+
+Note that, contrary to load, frame counts used to compute
+trial forwarding ratio are aggregates over all SUT ports.
+
+Questions around what is the correct number of frames
+that should have been forwarded it outside of the scope of this document.
+E.g. what should the measurer return when it detects
+that the offered load differs significantly from the intended load.
+
+It is RECOMMENDED implementations return an irregular goal result
+if they detect questionable (in comparability sense) trial results
+affecting their goal result.
+
+### Trial Loss Ratio
+
+Trial loss ratio is equal to one minus the trial forwarding ratio.
+
+### Trial Forwarding Rate
+
+The trial forwarding rate is the trial load multiplied by
+the trial forwarding ratio.
+
+Note that this is very similar, but not identical to Forwarding Rate
+as defined in [RFC2285] section 3.6.1, as that definition
+is specific to one output interface, while trial forwarding ratio
+is based on frame counts aggregated over all SUT interfaces.
+
+## Traffic profile
+
+Any other specifics (besides trial load and trial duration)
+the measurer needs to perform the trial are understood as a composite
+called the traffic profile.
+All its attributes are assumed to be constant during the search,
+and the composite is configured on the measurer by the manager
+before the search starts.
+
+Traffic profile is REQUIRED by [RFC2544] to contain some specific quantities,
+for example frame size.
+Several more specific quantities may be RECOMMENDED.
+
+Depending on SUT configuration, e.g. when testing specific protocols,
+additional values need to be included in the traffic profile
+and in the test report. See other IETF documents.
+
+## Search Goal
+
+A search goal is one item of the list required as an argument
+when the manager calls the controller.
+
+Each search goal is composite consisting of several attributes,
+some of them are required.
+Implementations are free to add their own attributes.
+
+Subsections list all required attributes and one recommended attribute.
+
+The meaning of the attributes is formally given only by their effect
+on the computation of the attributes of the goal result.
+The subsections do contain a short informal description,
+but see other chapters for more in-depth explanations.
+
+### Goal Final Trial Duration
+
+A threshold value for trial durations.
+REQUIRED attribute, MUST be positive.
+
+Informally, the conditional throughput for this goal will be computed
+only from trial results from trials as long as this.
+
+### Goal Duration Sum
+
+A threshold value for a particular sum of trial durations.
+REQUIRED attribute, MUST be positive.
+
+This uses the duration values returned by the measurer,
+in case it returns something else than the intended durations
+for the traffic part of the search.
+
+Informally, even at looking only at trials done at this goal's
+final trial duration, MLRsearch may spend up to this time measuring
+the same load value.
+
+### Goal Loss Ratio
+
+A threshold value for trial loss ratios.
+REQUIRED attribute, MUST be non-negative and smaller than one.
+
+Informally, if a load causes too many trials with trial results larger than this,
+the conditional throughput for this goal will be smaller than that load.
+
+### Goal Exceed Ratio
+
+A threshold value for particular ratio of duration sums.
+REQUIRED attribute, MUST be non-negative and smaller than one.
+
+This uses the duration values returned by the measurer,
+in case it returns something else than the intended durations
+for the traffic part of the search.
+
+Informally, this acts as the q-value for a quantile when selecting
+the forwarding rate of which trial result becomes the conditional throughput.
+For example, when the goal exceed ratio is 0.5 and MLRsearch
+happened to use the whole goal duration sum when determining conditional
+throughput, it means the conditional throughput is the median
+of trial forwarding rates.
+In practice, MLRsearch may stop measuring a load before the goal duration sum
+is reached, and the conditional throughput in that case frequently
+is the worst trial still not exceeding the goal loss ratio.
+If the goal duration sum is no larger than the goal fina trial duration,
+MLRsearch performs only one trial per load (unless other goals need more)
+and the goal exceed ratio has no effect on the search result.
+
+### Goal Width
+
+A value used as a threshold for telling when two trial load values
+are close enough.
+
+RECOMMENDED attribute, positive. Implementations without this attribute
+MUST give the manager other ways to control the search exit condition.
+
+Absolute load difference and relative load difference are two popular choices,
+but implementations may choose a different way to specify width.
+
+Informally, this acts as a stopping condition, controling the precision
+of the search. The search stops if every goal has reached its precision.
+
+## Search Result
+
+Search result is a single composite object returned from the controller
+to the manager.
+It is a mapping from the search goals (see section Search Goal) into goal results
+(see section Goal Result).
+The mapping MUST map from all the search goals present in the controller input.
+
+Each search goal instance is mapped to a goal result instance.
+Multiple search goal instances may map to the same goal result instance.
+
+## Goal Result
+
+Goal result is a composite object consisting of several attributes.
+All attributes are related to the same search goal, the one search goal instance
+the Search Result is mapping into this instance of the Goal Result.
+
+Some of the attributes are required, some are recommended,
+implementations are free to add their own.
+
+The subsections define attributes for regular goal result.
+Implementations are free to define their own irregular goal results,
+but the manager MUST report them clearly as not regular according to this section.
+
+A typical irregular result is when all trials at maximum offered load
+have zero loss, as the relevant upper bound does not exist in that case.
+
+### Relevant Upper Bound
+
+Relevant upper bound is the intended load value that is classified
+at the end of the search as the relevant upper bound (see Appendix A)
+for this goal.
+This is a REQUIRED attribute.
+
+Informally, this is the smallest intended load that failed to uphold
+all the requirements of this search goal, mainly the goal loss ratio
+in combination with the goal exceed ratio.
+
+### Relevant Lower Bound
+
+Relevant lower bound is the intended load value that got classified
+(after all trials) as the relevant lower bound (see Appendix A) for this goal.
+This is a REQUIRED attribute.
+
+The distance between the relevant lower bound and the relevant upper bound
+MUST NOT be larger than the goal width, for a regular goal result,
+if the implementation offers width as a goal attribute.
+
+Informally, this is the smallest intended load that managed to uphold
+all the requirements of this search goal, mainly the goal loss ratio
+in combination with the goal exceed ratio, while not being larger
+than the relevant upper bound.
+
+### Conditional Throughput
+
+The conditional throughput (see Appendix B) as evaluated
+at the relevant lower bound.
+This is a RECOMMENDED attribute.
+
+Informally, a typical forwarding rate expected to be seen
+at the relevant lower bound. But frequently just a conservative estimate thereof,
+as MLRsearch implementations tend to stop gathering more data
+as soon as they confirm this estimate cannot get worse within
+the goal duration sum.
+
+# MLRsearch Explanations
+
+This chapter focuses on intuitions and motivations
+and skips over some important details.
+
+Familiarity with the MLRsearch specification is not required here,
+so this chapter can act as an introduction.
+For example, this chapter start talking about tightest lower bounds
+before it is ready to talk about the relevant lower bound from the specification.
+
+## MLRsearch Versions
+
+The MLRsearch algorithm has been developed in a code-first approach,
+a Python library has been created, debugged and used in production
+before first descriptions (even informal) were published.
+In fact, multiple versions of the library were used in production
+over past few years, and later code was usually not compatible
+with earlier descriptions.
+
+The code in (any version of) MLRsearch library fully determines
+the search process (for given configuration parameters),
+leaving no space for deviations.
+MLRsearch as a name for a broad class of possible algorithms
+leaves plenty of space for future improvements, at the cost
+of poor comparability of results of different MLRsearch implementations.
+
+This document aspires to prescribe a MLRsearch specification
+in a way that restricts the important parts related to comparability,
+while leaving other parts vague enough so implementations can improve freely.
+
+## Exit Condition
+
+[RFC2544] prescribes that after performing one trial at a specific offered load,
+the next offered load should be larger or smaller, based on frame loss.
+
+The usual implementation uses binary search. Here a lossy trial becomes
+a new upper bound, a lossless trial becomes a new lower bound.
+The span of values between the tightest lower bound and the tightest upper bound
+forms an interval of possible results,
+and after each trial the width of that interval halves.
+
+Usually the binary search implementation tracks only the two tightest bounds,
+simply calling them bounds, but the old values still remain valid bounds,
+just not as tight as the new ones.
+
+After some number of trials, the tightest lower bound becomes the throughput.
+[RFC2544] does not specify when (if ever) should the search stop.
+
+MLRsearch library introduces a concept of goal width. The search stops
+when the distance between the tightest upper bound and the tightest lower bound
+is smaller than a user-configured value called goal width from now on.
+In other words, interval width has to be smaller than goal width
+at the end of the search.
+
+This goal width value therefore determines the precision of the result.
+As MLRsearch specification requires a particular structure of the result,
+the result itself does contain enough information to determine its precision,
+thus it is not required to report the goal width value.
+
+This allows MLRsearch implementations to use exit conditions
+different from goal width.
+The MLRsearch specification only REQUIRES the search procedure
+to always finish in a finite time, regardless of possible trial results.
+
+## Load Classification
+
+MLRsearch keeps the basic logic of binary search (tracking tightest bounds,
+measuring at the middle), perhaps with minor technical clarifications.
+The algorithm chooses an intended load (as opposed to offered load),
+the interval between bounds does not need to be split exactly in two equal halves,
+and the final reported structure specifies both bounds
+(optionally also the conditional throughput at the lower bound, defined later).
+
+The biggest difference is that in order to classify a load
+as an upper or lower bound, MLRsearch may need more than one trial
+(depending on configuration options) to be performed at the same intended load.
+
+As a consequence, even if a load already does have few trial results,
+it still may be classified as undecided, neither a lower bound nor an upper bound.
+
+Explanation of the classification logic is given in the next chatper,
+as it relies heavily on other sections of this chapter.
+
+For repeatability and comparability reasons, it is important that
+given a set of trial results, all implementations of MLRsearch
+classify the load in an equivalent way.
+
+## Loss Ratios
+
+Next difference is in goals of the search. [RFC2544] has a single goal,
+based on classifying full-length trials as either loss-less or lossy.
+
+As the name suggests, MLRsearch can seach for multiple goals, differing in their
+loss ratios. Precise definition of goal loss ratio will be given later.
+The [RFC2544] throughput goal then simply becomes a zero goal loss ratio.
+Different goals also may have different goal width.
+
+A set of trial results for one specific intended load value
+can classify the load as an upper bound for some goals, but a lower bound
+for some other goals, and undecided for the rest of the goals.
+
+Therefore, the load classification depends not only on ttrial results,
+but also on the goal. The overall search procedure becomes more complicated
+(compared to binary search with a single goal),
+but most of the complications do not affect the final result,
+except for one phenomenon, loss inversion.
+
+## Loss Inversion
+
+In [RFC2544] throuhput search using bisection, any load with lossy trial
+becomes a hard upper bound, meaning every subsequent trial has smaller
+intended load.
+
+But in MLRsearch, a load that is classified as an upper bound for one goal
+may still be a lower bound for another goal, and due to that other goal
+MLRsearch will probably perform trials at even higher loads.
+What to do when all such higher load trials happen to have zero loss?
+Does it mean the earlier upper bound was not real?
+Does it mean the later lossless trials are not considered a lower bound?
+Surely we do not want to have an upper bound at a load smaller than a lower bound.
+
+MLRsearch is conservative in these situations.
+The upper bound is considered real, and the lossless trials at higher loads
+are considered to be a coincidence, at least when computing the final result.
+
+This is formalized using new notions, the relevant upper bound and
+the relevant lower bound.
+Load classification is still based just on the set of trial results
+at a given intended load (trials at other loads are ignored),
+making it possible to have a lower load classified as an upper bound
+and a higher load classified as a lower bound (for the same goal).
+The relevant upper bound (for a goal) is the smallest load classified
+as an upper bound. But the relevant lower bound is not simply
+the largest among lower bounds. It is the largest load among loads
+that are lower bounds while also being smaller than the relevant upper bound.
+
+With these definitions, the relevant lower bound is always smaller
+than the relevant upper bound (if both exist), and the two relevant bounds
+are used analogously as the two tightest bounds in the binary search.
+When they are less than goal width apart, the relevant bounds are used in output.
+
+One consequence is that every trial result can have an impact on the search result.
+That means if your SUT (or your traffic generator) needs a warmup,
+be sure to warm it up before starting the search.
+
+## Exceed Ratio
+
+The idea of performing multiple trials at the same load comes from
+a model where some trial results (those with high loss) are affected
+by infrequent effects, causing poor repeatability of [RFC2544] throughput results.
+See the discussion about noiseful and noiseless ends of SUT performance spectrum.
+Stable results are closer to the noiseless end of SUT preformance spectrum,
+so MLRsearch may need to allow some frequency of high-loss trials
+to ignore the reare but big effects near the noisefull end.
+
+MLRsearch is able to do such trial result filtering, but it needs
+a configuration option to tell it how much frequent can the infrequent big loss be.
+This option is called exceed ratio. It tells MLRsearch what ratio of trials
+(more exactly what ratio of trial seconds) can have trial loss ratio
+larger than goal loss ratio and still be classified as a lower bound.
+Zero exceed ratio means all trials have to have trial loss ratio
+equal to or smaller than the goal loss ratio.
+
+For explainability reasons, the RECOMMENDED value for exceed ratio is 0.5,
+as it simplifies some later concepts by relating them to the concept of median.
+
+## Duration Sum
+
+When more than one trial is needed to classify a load,
+MLRsearch also needs something that controlls the number of trials needed.
+Therefore, each goal also has an attribute called duration sum.
+
+The meaning of a goal duration sum is that when a load has trials
+(at full trial duration, details later)
+whose trial durations when summed up give a value at least this,
+the load is guaranteed to be classified as an upper bound or a lower bound
+for the goal.
+
+As the duration sum has a big impact on the overall search duration,
+and [RFC2544] prescibes wait intervals around trial traffic,
+the MLRsearch algorithm may sum durations that are different
+from the actual trial traffic durations.
+
+## Short Trials
+
+Section 24 of [RFC2544] already anticipates possible time savings
+when short trials (shorter than full length trials) are used.
+
+MLRsearch requires each goal to specify its final trial duration.
+Full-length trial is the short name for a trial whose intended trial duration
+is equal to the goal final trial duration.
+
+Any MLRsearch implementation may include its own configuration options
+which control when and how MLRsearch chooses to use shorter trial durations.
+
+For explainability reasons, when exceed ratio of 0.5 is used,
+it is recommended for the goal duration sum to be an odd multiple
+of the full trial durations, so conditional throughput becomes identical to
+a median of a particular set of forwarding rates.
+
+Presence of shorter trial results complicates the load classification logic.
+Full details are given later. In short, results from short trials
+may cause a load to be classified as an upper bound.
+This may cause loss inversion, and thus lower the relevant lower bound
+(below what would classification say when considering full-length trials only).
+
+For explainability reasons, it is RECOMMENDED users use such configurations
+that guarantee all trials have the same length.
+Alas, such configurations are usually not compliant with [RFC2544] requirements,
+or not time-saving enough.
+
+## Conditional Throughput
+
+As testing equipment takes intended load as input parameter
+for a trial measurement, any load search algorithm needs to deal
+with intended load values internally.
+
+But in presence of goals with non-zero loss ratio, the intended load
+usually does not match the user intuition of what a throughput is.
+The forwarding rate (as defined in [RFC2285] section 3.6.1) is better,
+but it is not obvious how to generalize it
+for loads with multiple trial results,
+especially with non-zero goal exceed ratio.
+
+MLRsearch defines one such generalization, called the conditional throughput.
+It is the forwarding rate from one of the trials performed at the load
+in question. Specification of which trial exactly is quite technical.
+More detailed explanations are given in the next chapter.
+
+Conditional throughput is partially related to load classification.
+If a load is classified as a lower bound for a goal,
+the conditional throughpt can be calculated,
+and guaranteed to show effective loss ratio no larger than goal loss ratio.
+
+While the conditional throughput gives more intuitive-looking values
+than the relevant lower bound, especially for non-zero goal loss ratio values,
+the actual definition is more complicated than the definition of the relevant
+lower bound. In future, other intuitive values may become popular,
+but they are unlikely to supersede the definition of the relevant lower bound
+as the most fitting value for comparability purposes,
+therefore the relevant lower bound remains a required attribute
+of the goal result structure.
+
+Note that comparing best and worst case, the same relevant lower bound value
+may result in the conditional throughput differing up to the goal loss ratio.
+Therefore it is rarely needed to set the goal width (if expressed
+as relative difference of loads) below the goal loss ratio.
+In other words, setting the goal width below the goal loss ratio
+may cause the conditional throughput for a larger loss ratio to became smaller
+than a conditional throughput for a goal with a smaller goal loss ratio,
+which is counter-intuitive, considering they come from the same search.
+Therefore it is RECOMMENDED to set the goal width to a value no smaller
+than the goal loss ratio.
+
+## Search Time
+
+The main motivation for MLRsearch was to have an algorithm that spends less time
+finding a throughput, either the [RFC2544] compliant one,
+or some generalization thereof. The art of achieving short search times
+is mainly in smart selection of intended loads (and intended durations)
+for the next trial to perform.
+
+While there is an indirect impact of the load selection on the reported values,
+in practice such impact tends to be small,
+even for SUTs with quite broad performance spectrum.
+
+A typical example of two approaches to load selection leading to different
+relevant lower bounds is when the interval is split in a very uneven way.
+An implementation chosing loads very close to the current relevant lower bound
+are quite likely to eventually stumble upon a trial result
+with poor performance (due to SUT noise).
+For an implementation chosing load very close to the current relevant upper bound
+this is unlikely, as it examines more loads that can see a performance
+close to the noiseless end of the SUT performance spectrum.
+The reason why it is unlikely to have two MLRsearch implementation showing
+this kind of preference in load selection is precisely
+in the desire to have short searches.
+Even splits are the best way to achive the desired precision,
+so the more optimized a search algorithm is for the overall search duration,
+the better the repeatability and comparability
+of its results will be, assuming the user configuration remains the same.
+
+Therefore, this document remains quite vague on load selection
+and other optimisation details, and configuration attributes related to them.
+Assuming users prefer libreries that achieve short overall search time,
+the definition of the relevant lower bound
+should be strict enough to ensure result repeatability
+and comparability between different implementations,
+while not restricting future implementations much.
+
+Sadly, different implementations may exhibit their sweet spot of
+best repeatability at given search duration at different goals attribute values,
+especially with respect to optional goal attributes
+such as initial trial duration.
+Thus, this document does not comment much on which configurations
+are good for comparability between different implementations.
+For comparability between different SUTs using the same implementation,
+refer to configurations recommended by that particular implementation.
+
+## [RFC2544] compliance
+
+The following search goal ensures unconditional compliance with
+[RFC2544] throughput search procedure:
+
+- Goal loss ratio: zero.
+
+- Goal final trial duration: 60 seconds.
+
+- Goal duration sum: 60 seconds.
+
+- Goal exceed ratio: zero.
+
+Presence of other search goals does not affect compliance of this goal result.
+The relevant lower bound and the conditional throughput are in this case
+equal to each other, and the value is the [RFC2544] throughput.
+
+If the 60 second quantity is replaced by a smaller quantity in both attributes,
+the conditional throughput is still conditionally compliant with
+[RFC2544] throughput.
+
+# Selected Functional Details
+
+This chapter continues with explanations,
+but this time more precise definitions are needed
+for readers to follow the explanations.
+The definitions here are wordy, implementers can look into the next chapter
+for more concise definitions.
+
+The two areas of focus in this chapter are the load classification
+and the conditional throughput, starting with the latter.
+
+## Performance Spectrum
+
+There are several equivalent ways to define the conditional throughput computation.
+One of the ways relies on an object called the performance spectrum.
+First, two heavy definitions are needed.
+
+Take an intended load value, and a finite set of trial results, all trials
+measured at that load value. The performance spectrum is the function that maps
+any non-negative real number into a sum of trial durations among all trials
+in the set that have that number as their forwarding rate,
+e.g. map to zero if no trial has that particular forwarding rate.
+
+A related function, defined if there is at least one trial in the set,
+is the performance spectrum divided by sum of durations of all trials in the set.
+That function is called the performance probability function, as it satisfies
+all the requirements for probability mass function function
+of a discrete probability distribution,
+the one-dimensional random variable being the trial forwarding rate.
+
+These functions are related to the SUT performance spectrum,
+as sampled by the trials in the set.
+
+As for any other probability function, we can talk about percentiles,
+of the performance probability function, and bout other quantiles
+such as the median. The conditional throughput will be
+one such quantile value for a specifically chosen set of trials.
+
+Take a set of all full-length trials performed at the load in question.
+The sum of durations of those trials may be less than goal duration sum, or not.
+If it is less, add an imaginary trial result with zero forwarding rate
+such that the new sum of durations is equal to the goal duration sum.
+This is the set of trials to use. The q-value for the quantile
+is the goal exceed ratio. If the quantile touches two trials,
+the larger forwarding rate is used.
+
+First example. For zero exceed ratio when goal duration sum
+has been reached. The conditional throughput is the smallest forwarding
+rate among the trials.
+
+Second example. For zero exceed ratio when goal duration sum
+has not been reached yet. Due to the missing duration sum,
+the worst case may still happen, so the conditional througput
+is zero. This is not reported to the user, as this load
+cannot become the relevant lower bound yet.
+
+Third example. Exceed ratio 50%, goal duration sum two seconds,
+one trial present with duration one second and zero loss.
+An imaginary trial is added with duration one second and zero forwarding rate.
+Median would touch both trials, so the conditional throughput
+is the forwarding rate of the one non-imaginary trial.
+As that had zero loss, the value is equal to the offered load.
+
+The classification does not need the whole performance spectrum,
+only few duration sums.
+
+A trial is called bad (according to a goal) if its trial loss ratio
+is larger than the goal loss ratio. Trial that is not bad is called good.
+
+## Single Trial Duration
+
+When goal attributes are chosen in such a way that every trial has the same
+intended duration, the load classification is sipler.
+
+The following description looks technical, but it follows the motivation
+of goal loss ratio, goal exceed ratio and goal duration sum.
+If sum of durations of all trials (at given load) is less than the goal
+duration sum, imagine best case scenario (all subsequent trials having zero loss)
+and worst case scenario (all subsequent trials having 100% loss).
+Here we assume there is as many subsequent trials as needed
+to make the sum of all trials to become equal to the goal duration sum.
+As the exceed ratio is defined just using sums of durations
+(number of trials does not matter), it does not matter whether
+the "subsequent trials" can consist of integer number of full-length trials.
+
+If even in the best case scenario the load exceed ratio would be larger
+than the goal exceed ratio, the load is an upper bound.
+If even in the worst case scenario the load exceed ratio would not be larger
+than the goal exceed ratio, the load is a lower bound.
+
+Even more specifically.
+Take all trials measured at a given load.
+Sum of durations of all bad full-length trials is called the bad sum.
+Sum of durations of all good full-length trials is called the good sum.
+The result of adding bad sum plus the good sum is called the measured sum.
+The larger of the measured sum and the goal duration sum is called the whole sum.
+The whole sum minus the measured sum is called the missing sum.
+Optimistic exceed ratio is the bad sum divided by the whole sum.
+Pessimistic exceed ratio is the bad sum plus the missing sum, that divided by
+the whole sum.
+If optimistic exceed ratio is larger than the goal exceed ratio,
+the load is classified as an upper bound.
+If pessimistic exceed ratio is not larger than the goal exceed ratio,
+the load is classified as a lower bound.
+Else, the load is classified as undecided.
+
+The definition of pessimistic exceed ratio matches the logic in
+the conditional throughput computation, so a load is a lower bound
+if and only if the conditional throughput effective loss ratio
+is not larger than the goal loss ratio.
+If it is larger, the load is either an upper bound or undecided.
+
+## Short Trial Scenarios
+
+Trials with intended duration smaller than the goal final trial duration
+are called short trials.
+The motivation for load classification logic in presence of short trials
+is based around a counter-factual case: What would the trial result be
+if a short trial has been measured as a full-length trial instead?
+
+There are three main scenarios where human intuition guides
+the intended behavior of load classification.
+
+Scenario one. The user had their reason for not configuring shorter goal
+final trial duration. Perhaps SUT has buffers that may get full at longer
+trial durations. Perhaps SUT shows periodic decreases of performance
+the user does not want to treat as noise. In any case, many good short trials
+may became bad full-length trial in the counter-factual case.
+In extreme case, there are no bad short trials.
+In this scenario, we want the load classification NOT to classify the load
+as a lower bound, despite the abundance of good short trials.
+Effectively, we want the good short trials to be ignored, so they
+do not contribute to comparisons with the goal duration sum.
+
+Scenario two. When there is a frame loss in a short trial,
+the counter-factual full-length trial is expected to lose at least as many
+frames. And in practice, bad short trials are rarely turning into
+good full-length trials. In extreme case, there are no good short trials.
+In this scenario, we want the load classification
+to classify the load as an upper bound just based on abundance
+of short bad trials. Effectively we want the bad short trials
+to contribute to comparisons with the goal duration sum,
+so the load can be classified sooner.
+
+Scenario three. Some SUTs are quite indifferent to trial duration.
+Performance probability function constructed from short trial results
+is likely to be similar to performance probability function constructed
+from full-length trial results (perhaps with smaller dispersion,
+but overall without big impact on the median quantiles).
+For moderate goal exceed ratio values, this may mean there are both
+good short trials and bad short trials.
+This scenario is there just to invalidate a simple heuristic
+of always ignoring good short trials and never ignoring bad short trials.
+That simple heuristic would be too biased. Yes, the short bad trials
+are likely to turn into full-length bad trials in the counter-factual case,
+but there is no information on what would the good short trials turn into.
+The only way to decide is to do more trials at full length,
+the same as in scenario one.
+
+## Short Trial Logic
+
+MLRsearch picks a particular logic for load classification
+in presence of short trials, but it is still RECOMMENDED to use configurations
+that imply no short trials, so the possible inefficiencies in that logic
+do not affect the result, and the result has better explainability.
+
+With thas said, the logic differs from the single trial duration case
+only in different definition of bad sum.
+Good sum is still the sum across all good full-length trials.
+
+Few more notions are needed for definig the new bad sum.
+Sum of durations of all bad full-length trials is called the bad long sum.
+Sum of durations of all bad short trials is called the bad short sum.
+Sum of durations of all good short trials is called the good short sum.
+One minus the goal exceed ratio is called the inceed ratio.
+The goal exceed ratio divided by the inceed ratio is called the exceed coefficient.
+The good short sum multiplied by the exceed coefficient is called the balancing sum.
+The bad short sum minus the balancing sum is called the excess sum.
+If the excess sum is negative, the bad sum is equal to the bad long sum.
+Else, the bad sum is equal to the bad long sum plus the excess sum.
+
+Here is how the new definition of the bad sum fares in the three scenarios,
+where the load is close to what would relevant bounds be
+if only full-length trials were used for the search.
+
+Scenario one. If duration is too short, we expect to see higher frequency
+of good short trials. This could lead to negative excess sum,
+which has no impact, hence the load classification is given just by
+full-length trials.
+Thus, MLRsearch using too short trials has no detrimental effect
+on result comparability in this scenario.
+But also using short trials does not help with overall search duration,
+proably making it worse.
+
+Scenario two. Settings with small exceed ratio have small exceed coefficient,
+so the impact of good short sum is small and the bad short sum
+is almost wholly converted into excess sum, thus bad short trials
+have almost as big impact as full-length bad trials.
+The same conclusion applies for moderate exceed ratio values
+when the good short sum is small.
+Thus, short trials can cause a load to get classified as an upper bound earlier
+bringing time savings (while not affecting comparability).
+
+Scenario three. Here excess sum is small in absolute value, as balancing sum
+is expected to be be similar to the bad short sum.
+Once again, full-length trials are needed for final load classification,
+but usage of short trials probably means MLRsearch needed shorter search time
+before selecting this load for measurement, bringing time savings
+(while not affecting comparability).
+
+## Longer Trial Durations
+
+If there are trial results with intended duration larger
+than the goal trial duration, the classification logic is intentionally undefined.
+
+The implementations MAY treat such longer trials as if they were full-length.
+In any case, presence of such longer trials in either the relevant lower bound
+or the relevant upper bound SHOULD be mentioned, as for sume SUTs
+it is likely to affect comparability.
+
+
+TODO: Here will be a chapter summarizing how MLRsearch library
+adresses the problems from the Identified Problems chapter.
+
+{::comment}
+
+ # Problems after MLRsearch
+
+ Now when MLRsearch is clearly specified and explained,
+ it is possible to summarize how does MLRsearch specification help with problems.
+
+ Here, "multiple trials" is a shorthand for having the goal final trial duration
+ significantly smaller than the goal duration sum.
+ This results in MLRsearch performing multiple trials at the same load,
+ which may not be the case with other configurations.
+
+ ## Long Test Duration
+
+ As shortening the overall search duration is the main motivation
+ of MLRsearch library development, the library implements
+ multiple improvements on this front, both big and small.
+ Most of implementation details are not part of the MLRsearch specification,
+ so that future implementations may keep shortening the search duration even more.
+
+ TODO: The rest is about attributes.
+
+ From the required goal attributes, the goal duration sum
+ remains the best way to get even shorter searches.
+
+ Usage of multiple trials can also save time,
+ depending on wait times around trial traffic.
+
+ The farther the goal exceed ratio is from 0.5 towards either side,
+ the less predictable the overal search duration becomes in practice.
+
+ Width parameter does not change search duration much in practice
+ (compared to other, mainly optional goal attributes).
+
+ ## DUT in SUT
+
+ Practice shows big improvements when multiple trials
+ and moderate exceed ratios are used. Mainly when it comes to result
+ repeatability, as sometimes it is not easy to distinguish
+ SUT noise from DUT instability.
+
+ Conditional throughput has intuitive meaning when described
+ using the performance spectrum, so this is an improvement,
+ especially when compared to search procedures which use non-zero
+ goal loss ratio but return only the intended load at a lower bound.
+
+ Multiple trials can save time also when the noisy end of
+ the preformance spectrum needs to be examined, e.g. for [RFC9004].
+
+ Under some circumstances, testing the same DUT and SUT setup with different
+ DUT configurations can give some hints on what part of noise us SUT noise
+ and what part is DUT performance fluctuations.
+ In practice, both types of noise tend to be too complicated for that analysis.
+ MLRsearch does not offer additional tools in this regard,
+ apart of giving users ability to search for more goals,
+ hoping to get more insight at the cost of longer overall search time.
+
+ ## Repeatability and Comparability
+
+ Multiple trials improve repeatability, depending on exceed ratio.
+
+ In practice, 1s goal final trial duration with exceed ratio 0.5
+ is good enough for modern SUTs (but that usually requires
+ smaller wait times around the traffic part of the trial,
+ otherwise too much search time is wasted waiting).
+
+ It is not clear whether exceed ratios higher than 0.5 are better
+ for repeatability.
+ The 0.5 value is still preferred due to explainability using median.
+ TODO: Stress single value is for comparability, which one is due explainability.
+
+ It is possible that the conditional throughput values (with non-zero
+ goal loss ratio) are better for repeatability than the relevant
+ lower bound values, especially for implementations
+ which pick load from a small set of discrete values.
+
+ Implementations focusing on shortening the overall search time
+ are automatically forced to avoid comparability issues
+ due to load selection, as they must prefer even splits wherever possible.
+ But this conclusion only holds when the same goals are used.
+ Larger adoption is needed before any further claims on comparability
+ between MLRsearch implementations can be made.
+
+ ## Throughput with Non-Zero Loss
+
+ Suported by the goal loss ratio attribute.
+ Improves repeatability as expected.
+
+ ## Inconsistent Trial Results
+
+ MLRsearch is conservative wherever possible,
+ this is built into the definition of conditional throughput,
+ and into the treatment of short trial results for load classification.
+
+ This is consistent with [RFC2544] zero loss tolerance motivation.
+
+ If the very best (noiseless) part of the SUT performance spectrum
+ is of interest, it should be enough to set small value for
+ the goal final trial duration, and perhaps also a large value
+ for the goal exceed ratio.
+
+ Implementations may offer other (optional) configuration attributes
+ (and optional goal result attributes)
+ to become less conservative, but currently it is not clear
+ what impact would that have on repeatability.
+
+{:/comment}
+
+# IANA Considerations
+
+No requests of IANA.
+
+# Security Considerations
+
+Benchmarking activities as described in this memo are limited to
+technology characterization of a DUT/SUT using controlled stimuli in a
+laboratory environment, with dedicated address space and the constraints
+specified in the sections above.
+
+The benchmarking network topology will be an independent test setup and
+MUST NOT be connected to devices that may forward the test traffic into
+a production network or misroute traffic to the test management network.
+
+Further, benchmarking is performed on a "black-box" basis, relying
+solely on measurements observable external to the DUT/SUT.
+
+Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
+benchmarking purposes. Any implications for network security arising
+from the DUT/SUT SHOULD be identical in the lab and in production
+networks.
+
+# Acknowledgements
+
+Many thanks to Alec Hothan of OPNFV NFVbench project for thorough
+review and numerous useful comments and suggestions.
+
+Special wholehearted gratitude and thanks to late Al Morton for his
+thorough reviews filled with very specific feedback and constructive
+guidelines. Thank you Al for the close collaboration over the years,
+for your continuous unwavering encouragements full of empathy and
+positive attitude.
+Al, you are dearly missed.
+
+# Appendix A
+
+This is a specification of load classification.
+
+The block at the end of this appendix holds pseudocode
+which computes two values, stored in variables named optimistic and pessimistic.
+The pseudocode happens to be a valid Python code.
+
+If both values are computed to be true, the load in question
+is classified as a lower bound according to the goal in question.
+If both values are false, the load is classified as an upper bound.
+Otherwise, the load is classifies as undecided.
+
+The pseudocole expects the following variables hold values as follows:
+
+- goal_duration_sum: The goal duration sum value.
+
+- goal_exceed_ratio: The goal exceed ratio value.
+
+- good_long_sum: Sum of durations across trials with trial duration
+ at least equal to the goal final trial duration and with trial loss ratio
+ not higher than the goal loss ratio.
+
+- bad_long_sum: Sum of durations across trials with trial duration
+ at least equal to the goal final trial duration and with trial loss ratio
+ higher than the goal loss ratio.
+
+- good_short_sum: Sum of durations across trials with trial duration
+ shorter than the goal final trial duration and with trial loss ratio
+ not higher than the goal loss ratio.
+
+- bad_short_sum: Sum of durations across trials with trial duration
+ shorter than the goal final trial duration and with trial loss ratio
+ higher than the goal loss ratio.
+
+Here the implicit set of available trial results consists of all trials
+measured at given intended load at the end of search.
+
+The code works correctly also when there are no trial results at given load.
+
+```
+balancing_sum = good_short_sum * goal_exceed_ratio / (1.0 - goal_exceed_ratio)
+effective_bad_sum = bad_long_sum + max(0.0, bad_short_sum - balancing_sum)
+effective_whole_sum = max(good_long_sum + effective_bad_sum, goal_duration_sum)
+quantile_duration_sum = effective_whole_sum * goal_exceed_ratio
+optimistic = effective_bad_sum <= quantile_duration_sum
+pessimistic = (effective_whole_sum - good_long_sum) <= quantile_duration_sum
+```
+
+# Appendix B
+
+This is a specification of conditional throughput.
+
+The block at the end of this appendix holds pseudocode
+which computes a value stored as variable conditional_throughput.
+The pseudocode happens to be a valid Python code.
+
+The pseudocole expects the following variables hold values as follows:
+
+- goal_duration_sum: The goal duration sum value.
+
+- goal_exceed_ratio: The goal exceed ratio value.
+
+- good_long_sum: Sum of durations across trials with trial duration
+ at least equal to the goal final trial duration and with trial loss ratio
+ not higher than the goal loss ratio.
+
+- bad_long_sum: Sum of durations across trials with trial duration
+ at least equal to the goal final trial duration and with trial loss ratio
+ higher than the goal loss ratio.
+
+- long_trials: An iterable of all trial results from trials with trial duration
+ at least equal to the goal final trial duration,
+ sorted by increasing trial loss ratio.
+ A trial result is a composite with the following two attributes available:
+
+ - trial.loss_ratio: The trial loss ratio as measured for this trial.
+
+ - trial.duration: The trial duration of this trial.
+
+Here the implicit set of available trial results consists of all trials
+measured at given intended load at the end of search.
+
+The code works correctly only when there if there is at leas one
+trial result measured at given load.
+
+```
+all_long_sum = max(goal_duration_sum, good_long_sum + bad_long_sum)
+remaining = all_long_sum * (1.0 - goal_exceed_ratio)
+quantile_loss_ratio = None
+for trial in long_trials:
+ if quantile_loss_ratio is None or remaining > 0.0:
+ quantile_loss_ratio = trial.loss_ratio
+ remaining -= trial.duration
+ else:
+ break
+else:
+ if remaining > 0.0:
+ quantile_loss_ratio = 1.0
+
+conditional_throughput = intended_load * (1.0 - quantile_loss_ratio)
+```
+
+--- back
Code Review / csit.git / blobdiff