-Test Effectiveness Comparison
------------------------------
-
-Introduction
-````````````
-
-CSIT release 1804 contains two test suites that use the new MDR search
-to enable comparison against existing CSIT NDR and PDR binary searches.
-The suites got chosen based on the level of consistency of their
-historical NDR/PDR results:
-
-#. *10Ge2P1X520-Ethip4-Ip4Base-Ndrpdr* - yielding very consistent binary
- search results.
-#. *10Ge2P1X520-Eth-L2Bdbasemaclrn-Eth-2Vhostvr1024-1Vm-Ndrpdr* - yielding
- somewhat inconsistent results.
-
-Here "inconsistent" means the values found differ between runs,
-even though the setup and the test are exactly the same.
-
-The search part of CSIT binary search tests requires a single 5-second warmup
-and each trial measurement is set to 10 seconds.
-
-New tests with MDR search do not have any warmup, as initial measurements
-are not critical to the final result.
-
-Fairness of the following comparison has been achieved
-by setting MDR final relative width to values causing the width to match
-the binary NDR/PDR result.
-Each search algorithm has been run with three different
-(final) trial durations: 10s, 30s and 60s.
-
-Tables below compares overall test duration between the search tests.
-For simplicity only data for single thread 64B packet tests is listed,
-as it takes the longest in all cases.
-
-Data in tables is based on result of 6 runs.
-
-Tables
-``````
-
-.. note:: The comparison was done for the MDR code
- before https://gerrit.fd.io/r/12761
-
-.. table:: Table 1. Search part of test duration.
-
- ==================== ========== =========== =========== ========== =========== ===========
- Duration+-avgdev [s] IP4 10s IP4 30s IP4 60s Vhost 10s Vhost 30s Vhost 60s
- ==================== ========== =========== =========== ========== =========== ===========
- MDR (both intervals) 50.8+-1.2 109.0+-10.0 202.8+-11.7 80.5+-9.0 201.9+-20.6 474.9+-58.2
- NDR binary 98.9+-0.1 278.6+-0.1 548.8+-0.1 119.8+-0.1 339.3+-0.1 669.6+-0.2
- PDR binary 98.9+-0.1 278.6+-0.1 548.8+-0.1 119.7+-0.1 339.3+-0.1 669.5+-0.1
- NDR+PDR sum 197.8+-0.1 557.2+-0.2 1097.6+-0.1 239.5+-0.1 678.7+-0.1 1339.2+-0.1
- ==================== ========== =========== =========== ========== =========== ===========
-
-.. note:: Here "avgdev" is the estimated difference between
- the average duration computed from the limited sample
- and a true average duration as its hypothetical limit for infinite samples.
- To get the usual "standard deviation" of duration, "avgdev" has to be multiplied
- by the square root of the number of samples.
- For the subtle details see `estimation of standard deviation`_,
- we used zero ACF and c4==1.
-
-.. table:: Table 2. MDR duration as percentage of NDR duration.
-
- ==================================== ========= ========= ========= ========= ========= =========
- Fraction+-stdev [%] IP4 10s IP4 30s IP4 60s Vhost 10s Vhost 30s Vhost 60s
- ==================================== ========= ========= ========= ========= ========= =========
- MDR duration divided by NDR duration 51.4+-1.2 39.1+-3.6 37.0+-2.1 67.2+-7.5 59.5+-6.1 70.9+-8.7
- ==================================== ========= ========= ========= ========= ========= =========
-
-.. note:: Here "stdev" is standard deviation as computed by the
- `simplified error propagation formula`_.
-
-Conclusions
-```````````
-
-In consistent tests, MDR is on average more than 50% faster
-than a single NDR binary search (even though MDR also detects PDR).
-
-One exception is 10 second final trial duration,
-where MDR is (only) almost 50% faster than NDR binary search.
-Most probably presence of 2 intermediate phases (instead of just 1) hurts there.
-
-In inconsistent tests MDR is still somewhat faster than NDR binary search,
-but it is not by 50%, and it is hard to quantify as MDR samples have wildly
-varying durations.
-
-Search Time Graphs
-------------------
-
-The following graphs were created from the data gathered from comparison runs,
-for the vhost tests.
-The vertical axis has always the same values,
-zoomed into the interesting part of the search space.
-The faint blue vertical lines separate the phases of MDR search.
-The bound lines are sloped just to help locate the previous value,
-in reality the bounds are updated instantly at the end of the measurement.
-
-The graphs do not directly show when a particular bound is invalid.
-However this can be gleaned indirectly by identifying
-that the measurement does not satisfy that bound's validity conditions
-(see point 2a).
-Also, the external search follows, and the measurement previously acting
-as and invalid upper or lower bound starts acting instead
-as a valid lower or upper bound, respectively.
-
-The following three graphs are for MDR with 10 second final trial duration,
-showing different behavior in this inconsistent test,
-and different amount of "work" done by each phase.
-Also the horizontal axis has the same scaling here.
-
-.. image:: MDR_10_1.svg
-.. image:: MDR_10_2.svg
-.. image:: MDR_10_3.svg
-
-The next graph is for MDR with 60 second final trial duration,
-to showcase the final phase takes the most of the overall search time.
-The scaling of the horizontal axis is different.
-
-.. image:: MDR_60.svg
-
-Finally, here are two graphs showing NDR and PDR binary searches.
-The trial duration is again 60 seconds,
-but scaling of horizontal axis is once again different.
-This shows the binary search spends most time measuring outside
-the interesting rate region.
-
-.. image:: NDR_60.svg
-.. image:: PDR_60.svg
-