report: edits of methodology throughput sections

Change-Id: I9ed1c2d67bc53f98d36ad17afb9153de5e944b7f
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>

diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst
index e9d8707..45cd6bc 100644 (file)
--- a/docs/report/introduction/methodology.rst
+++ b/docs/report/introduction/methodology.rst
@@ -11,7 +11,8 @@ Test Methodology
     methodology_vpp_features
     methodology_data_plane_throughput
     methodology_mlrsearch_tests
     methodology_vpp_features
     methodology_data_plane_throughput
     methodology_mlrsearch_tests

-    methodology_bmrr_throughput
+    methodology_mrr_throughput
+    methodology_plrsearch

     methodology_packet_latency
     methodology_multi_core_speedup
     methodology_vpp_startup_settings
     methodology_packet_latency
     methodology_multi_core_speedup
     methodology_vpp_startup_settings


@@ -22,7 +23,6 @@ Test Methodology
     methodology_vpp_device_functional
     methodology_ipsec_on_intel_qat
     methodology_trex_traffic_generator
     methodology_vpp_device_functional
     methodology_ipsec_on_intel_qat
     methodology_trex_traffic_generator

-    methodology_plrsearch

 
 ..
     methodology_http_tcp_with_wrk_tool
 
 ..
     methodology_http_tcp_with_wrk_tool

diff --git a/docs/report/introduction/methodology_data_plane_throughput.rst b/docs/report/introduction/methodology_data_plane_throughput.rst
index 10d85e9..2b5b085 100644 (file)
--- a/docs/report/introduction/methodology_data_plane_throughput.rst
+++ b/docs/report/introduction/methodology_data_plane_throughput.rst
@@ -1,33 +1,86 @@
 Data Plane Throughput
 ---------------------
 
 Data Plane Throughput
 ---------------------
 

-Network data plane packet and bandwidth throughput are measured in
-accordance with :rfc:`2544`, using FD.io CSIT Multiple Loss Ratio search
-(MLRsearch), an optimized throughput search algorithm, that measures
-SUT/DUT packet throughput rates at different Packet Loss Ratio (PLR)
-values.
+Network data plane packet and bandwidth throughput are measured using
+multiple methods in order to obtain representative and repeatable
+results across the large set of performance test cases implemented and
+executed within CSIT. Following throughput test methods are used:

 
 

-Following MLRsearch values are measured across a range of L2 frame sizes
-and reported:
+#. MLRsearch: Multiple Loss Ratio search

 
 

-- NON DROP RATE (NDR): packet and bandwidth throughput at PLR=0%.
+   - **Description**: MLRsearch discovers multiple packet throughput
+     rates in a single search, reducing the overall test execution
+     time compared to a binary search. Each rate associated with a
+     distinct Packet Loss Ratio (PLR) criteria. In FD.io CSIT two
+     throughput rates are discovered: Non-Drop Rate (NDR, with zero
+     packet loss, PLR=0) and Partial Drop Rate (PDR, with PLR<0.5%).
+     MLRsearch is compliant with :rfc:`2544`.
+   - **Usage**: MLRsearch tests are run to discover NDR and PDR rates
+     for each VPP and DPDK release covered by CSIT report. Results for
+     small frame sizes (64b/78B, IMIX) are presented in packet
+     throughput graphs (Box-and-Whisker Plots) with NDR and PDR rates
+     plotted against the test cases covering popular VPP packet paths.
+     Each test is executed at least 10 times to verify measurements
+     repeatability and results are compared between releases and test
+     environments. NDR and PDR packet and bandwidth throughput results
+     for all frame sizes and for all tests are presented in detailed
+     results tables.
+   - **References**: See :ref:`mlrsearch_algorithm` for more detailed
+     description of MLRsearch tests. MLRsearch is being standardized
+     in IETF with `draft-vpolak-mkonstan-mlrsearch
+     <https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch>`_.

 
 

-  - **Aggregate packet rate**: NDR_LOWER <bi-directional packet rate>
-    pps.
-  - **Aggregate bandwidth rate**: NDR_LOWER <bi-directional bandwidth
-    rate> Gbps.
+#. MRR Measurements: Maximum Receive Rate

 
 

-- PARTIAL DROP RATE (PDR): packet and bandwidth throughput at PLR=0.5%.
+   - **Description**: MRR tests are complementary to MLRsearch tests,
+     as they provide a maximum “raw” throughput benchmark for
+     development and testing community. MRR tests measure the packet
+     forwarding rate under the maximum load offered by traffic
+     generator (dependent on link type and NIC model) over a set trial
+     duration, regardless of packet loss. Maximum load for specified
+     Ethernet frame size is set to the bi-directional link rate.
+   - **Usage**: MRR tests are much faster than MLRsearch as they rely
+     on a single trial or a small set of trials with very short
+     duration. It is this property that makes them suitable for
+     continuous execution in daily performance trending jobs enabling
+     detection of performance anomalies (regressions, progressions)
+     resulting from data plane code changes. MRR tests are also used
+     for VPP per patch performance jobs verifying patch performance
+     vs. parent. CSIT reports include MRR throughput comparisons
+     between releases and test environments. Small frame sizes only
+     (64b/78B, IMIX).
+   - **References**: See :ref:`mrr_throughput` for more detailed
+     description of MRR tests configuration used for daily performance
+     trending jobs. VPP per patch test methodology is available on
+     `FD.io CSIT trending pages
+     <https://docs.fd.io/csit/master/trending/methodology/perpatch_performance_tests.html>`_.

 
 

-  - **Aggregate packet rate**: PDR_LOWER <bi-directional packet rate>
-    pps.
-  - **Aggregate bandwidth rate**: PDR_LOWER <bi-directional bandwidth
-    rate> Gbps.
+#. PLRsearch: Probabilistic Loss Ratio search

 
 

-NDR and PDR are measured for the following L2 frame sizes (untagged
-Ethernet):
+   - **Description**: PLRsearch discovers a packet throughput rate
+     associated with configured Packet Loss Ratio (PLR) criteria for
+     tests run over an extended period of time a.k.a. soak testing.
+     PLRsearch assumes that system under test is probabilistic in
+     nature, and not deterministic.
+   - **Usage**: PLRsearch are run to discover a sustained throughput
+     for PLR=10^-7 (close to NDR) for VPP release covered by CSIT
+     report. Results for small frame sizes (64b/78B) are presented in
+     packet throughput graphs (Box Plots) for a small subset of
+     baseline tests. Each soak test lasts 2hrs and is executed at
+     least twice. Results are compared against NDR and PDR rates
+     discovered with MLRsearch.
+   - **References**: See :ref:`plrsearch_algorithm` for more detailed
+     description of PLRsearch tests. PLRsearch is being standardized
+     in IETF with `draft-vpolak-bmwg-plrsearch
+     <https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch>`_.

 
 

-- IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B.
-- IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B.
+All of the listed data plane throughput test methodologies share
+following properties:

 
 

-All rates are reported from external Traffic Generator perspective.
+- Tested L2 frame sizes (untagged Ethernet):
+
+  - IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B.
+  - IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B.
+
+- All measured rates are aggregate bi-directional rates reported from
+  external Traffic Generator perspective.
\ No newline at end of file

diff --git a/docs/report/introduction/methodology_mlrsearch_tests.rst b/docs/report/introduction/methodology_mlrsearch_tests.rst
index 15b9ee0..6c2adfc 100644 (file)
--- a/docs/report/introduction/methodology_mlrsearch_tests.rst
+++ b/docs/report/introduction/methodology_mlrsearch_tests.rst
@@ -1,4 +1,3 @@
-

 .. _mlrsearch_algorithm:
 
 MLRsearch Tests
 .. _mlrsearch_algorithm:
 
 MLRsearch Tests

diff --git a/docs/report/introduction/methodology_bmrr_throughput.rst b/docs/report/introduction/methodology_mrr_throughput.rst
similarity index 93%
rename from docs/report/introduction/methodology_bmrr_throughput.rst
rename to docs/report/introduction/methodology_mrr_throughput.rst
index 7bef3a4..824ac88 100644 (file)
--- a/docs/report/introduction/methodology_bmrr_throughput.rst
+++ b/docs/report/introduction/methodology_mrr_throughput.rst
@@ -1,5 +1,7 @@
-(B)MRR Throughput
------------------
+.. _mrr_throughput:
+
+MRR Throughput
+--------------

 
 Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests,
 as they provide a maximum "raw" throughput benchmark for development and
 
 Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests,
 as they provide a maximum "raw" throughput benchmark for development and


@@ -10,10 +12,9 @@ size is set to the bi-directional link rate.
 
 In |csit-release| MRR test code has been updated with a configurable
 burst MRR parameters: trial duration and number of trials in a single
 
 In |csit-release| MRR test code has been updated with a configurable
 burst MRR parameters: trial duration and number of trials in a single

-burst. This enabled a new Burst MRR (BMRR) methodology for more precise
-performance trending.
+burst. This enabled more precise performance trending.

 
 

-Current parameters for BMRR tests:
+Current parameters for MRR trending tests:

 
 - Ethernet frame sizes: 64B (78B for IPv6), IMIX, 1518B, 9000B; all
   quoted sizes include frame CRC, but exclude per frame transmission
 
 - Ethernet frame sizes: 64B (78B for IPv6), IMIX, 1518B, 9000B; all
   quoted sizes include frame CRC, but exclude per frame transmission

diff --git a/docs/report/introduction/methodology_plrsearch.rst b/docs/report/introduction/methodology_plrsearch.rst
index 65ac590..fdd0347 100644 (file)
--- a/docs/report/introduction/methodology_plrsearch.rst
+++ b/docs/report/introduction/methodology_plrsearch.rst
@@ -1,4 +1,4 @@
-.. _`PLRsearch algorithm`:
+.. _plrsearch_algorithm:

 
 PLRsearch
 ^^^^^^^^^
 
 PLRsearch
 ^^^^^^^^^