methodology_vpp_forwarding_modes
methodology_tunnel_encapsulations
methodology_vpp_features
- methodology_data_plane_throughput
+ methodology_data_plane_throughput/index
methodology_mlrsearch_tests
methodology_mrr_throughput
methodology_plrsearch
+++ /dev/null
-Data Plane Throughput
----------------------
-
-Network data plane packet and bandwidth throughput are measured using
-multiple methods in order to obtain representative and repeatable
-results across the large set of performance test cases implemented and
-executed within CSIT. Following throughput test methods are used:
-
-#. MLRsearch: Multiple Loss Ratio search
-
- - **Description**: MLRsearch discovers multiple packet throughput
- rates in a single search, reducing the overall test execution
- time compared to a binary search. Each rate associated with a
- distinct Packet Loss Ratio (PLR) criteria. In FD.io CSIT two
- throughput rates are discovered: Non-Drop Rate (NDR, with zero
- packet loss, PLR=0) and Partial Drop Rate (PDR, with PLR<0.5%).
- MLRsearch is compliant with :rfc:`2544`.
- - **Usage**: MLRsearch tests are run to discover NDR and PDR rates
- for each VPP and DPDK release covered by CSIT report. Results for
- small frame sizes (64b/78B, IMIX) are presented in packet
- throughput graphs (Box-and-Whisker Plots) with NDR and PDR rates
- plotted against the test cases covering popular VPP packet paths.
- Each test is executed at least 10 times to verify measurements
- repeatability and results are compared between releases and test
- environments. NDR and PDR packet and bandwidth throughput results
- for all frame sizes and for all tests are presented in detailed
- results tables.
- - **References**: See :ref:`mlrsearch_algorithm` for more detailed
- description of MLRsearch tests. MLRsearch is being standardized
- in IETF with `draft-vpolak-mkonstan-mlrsearch
- <https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch>`_.
-
-#. MRR Measurements: Maximum Receive Rate
-
- - **Description**: MRR tests are complementary to MLRsearch tests,
- as they provide a maximum “raw” throughput benchmark for
- development and testing community. MRR tests measure the packet
- forwarding rate under the maximum load offered by traffic
- generator (dependent on link type and NIC model) over a set trial
- duration, regardless of packet loss. Maximum load for specified
- Ethernet frame size is set to the bi-directional link rate.
- - **Usage**: MRR tests are much faster than MLRsearch as they rely
- on a single trial or a small set of trials with very short
- duration. It is this property that makes them suitable for
- continuous execution in daily performance trending jobs enabling
- detection of performance anomalies (regressions, progressions)
- resulting from data plane code changes. MRR tests are also used
- for VPP per patch performance jobs verifying patch performance
- vs. parent. CSIT reports include MRR throughput comparisons
- between releases and test environments. Small frame sizes only
- (64b/78B, IMIX).
- - **References**: See :ref:`mrr_throughput` for more detailed
- description of MRR tests configuration used for daily performance
- trending jobs. VPP per patch test methodology is available on
- `FD.io CSIT trending pages
- <https://docs.fd.io/csit/master/trending/methodology/perpatch_performance_tests.html>`_.
-
-#. PLRsearch: Probabilistic Loss Ratio search
-
- - **Description**: PLRsearch discovers a packet throughput rate
- associated with configured Packet Loss Ratio (PLR) criteria for
- tests run over an extended period of time a.k.a. soak testing.
- PLRsearch assumes that system under test is probabilistic in
- nature, and not deterministic.
- - **Usage**: PLRsearch are run to discover a sustained throughput
- for PLR=10^-7 (close to NDR) for VPP release covered by CSIT
- report. Results for small frame sizes (64b/78B) are presented in
- packet throughput graphs (Box Plots) for a small subset of
- baseline tests. Each soak test lasts 2hrs and is executed at
- least twice. Results are compared against NDR and PDR rates
- discovered with MLRsearch.
- - **References**: See :ref:`plrsearch_algorithm` for more detailed
- description of PLRsearch tests. PLRsearch is being standardized
- in IETF with `draft-vpolak-bmwg-plrsearch
- <https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch>`_.
-
-All of the listed data plane throughput test methodologies share
-following properties:
-
-- Tested L2 frame sizes (untagged Ethernet):
-
- - IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B.
- - IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B.
-
-- All measured rates are aggregate bi-directional rates reported from
- external Traffic Generator perspective.
\ No newline at end of file
--- /dev/null
+Data Plane Throughput
+=====================
+
+.. toctree::
+
+ methodology_data_plane_throughput
+ methodology_mlrsearch_tests
+ methodology_mrr_throughput
+ methodology_plrsearch
+
--- /dev/null
+Data Plane Throughput Tests
+---------------------------
+
+Network data plane throughput is measured using multiple test methods in
+order to obtain representative and repeatable results across the large
+set of performance test cases implemented and executed within CSIT.
+
+Following throughput test methods are used:
+
+- MLRsearch - Multiple Loss Ratio search
+- MRR - Maximum Receive Rate
+- PLRsearch - Probabilistic Loss Ratio search
+
+Description of each test method is followed by generic test properties
+shared by all methods.
+
+MLRsearch Tests
+^^^^^^^^^^^^^^^
+
+Description
+~~~~~~~~~~~
+
+Multiple Loss Ratio search (MLRsearch) tests discover multiple packet
+throughput rates in a single search, reducing the overall test execution
+time compared to a binary search. Each rate associated with a distinct
+Packet Loss Ratio (PLR) criteria. In FD.io CSIT two throughput rates are
+discovered: Non-Drop Rate (NDR, with zero packet loss, PLR=0) and
+Partial Drop Rate (PDR, with PLR<0.5%). MLRsearch is compliant with
+:rfc:`2544`.
+
+Usage
+~~~~~
+
+MLRsearch tests are run to discover NDR and PDR rates for each VPP and
+DPDK release covered by CSIT report. Results for small frame sizes
+(64b/78B, IMIX) are presented in packet throughput graphs
+(Box-and-Whisker Plots) with NDR and PDR rates plotted against the test
+cases covering popular VPP packet paths.
+
+Each test is executed at least 10 times to verify measurements
+repeatability and results are compared between releases and test
+environments. NDR and PDR packet and bandwidth throughput results for
+all frame sizes and for all tests are presented in detailed results
+tables.
+
+Details
+~~~~~~~
+
+See :ref:`mlrsearch_algorithm` section for more detail. MLRsearch is
+being standardized in IETF in `draft-vpolak-mkonstan-mlrsearch
+<https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch>`_.
+
+MRR Tests
+^^^^^^^^^
+
+Description
+~~~~~~~~~~~
+
+Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests,
+as they provide a maximum “raw” throughput benchmark for development and
+testing community.
+
+MRR tests measure the packet forwarding rate under the maximum load
+offered by traffic generator (dependent on link type and NIC model) over
+a set trial duration, regardless of packet loss. Maximum load for
+specified Ethernet frame size is set to the bi-directional link rate.
+
+MRR tests are much faster than MLRsearch as they rely on a single trial
+or a small set of trials with very short duration. It is this property
+that makes them suitable for continuous execution in daily performance
+trending jobs enabling detection of performance anomalies (regressions,
+progressions) resulting from data plane code changes.
+
+MRR tests are also used for VPP per patch performance jobs verifying
+patch performance vs. parent. CSIT reports include MRR throughput
+comparisons between releases and test environments. Small frame sizes
+only (64b/78B, IMIX).
+
+Details
+~~~~~~~
+
+See :ref:`mrr_throughput` section for more detail about MRR tests
+configuration.
+
+FD.io CSIT performance dashboard includes complete description of
+`daily performance trending tests
+<https://docs.fd.io/csit/master/trending/methodology/performance_tests.html>`_
+and `VPP per patch tests
+<https://docs.fd.io/csit/master/trending/methodology/perpatch_performance_tests.html>`_.
+
+PLRsearch Tests
+^^^^^^^^^^^^^^^
+
+Description
+~~~~~~~~~~~
+
+Probabilistic Loss Ratio search (PLRsearch) tests discovers a packet
+throughput rate associated with configured Packet Loss Ratio (PLR)
+criteria for tests run over an extended period of time a.k.a. soak
+testing. PLRsearch assumes that system under test is probabilistic in
+nature, and not deterministic.
+
+Usage
+~~~~~
+
+PLRsearch are run to discover a sustained throughput for PLR=10^-7
+(close to NDR) for VPP release covered by CSIT report. Results for small
+frame sizes (64b/78B) are presented in packet throughput graphs (Box
+Plots) for a small subset of baseline tests.
+
+Each soak test lasts 2hrs and is executed at least twice. Results are
+compared against NDR and PDR rates discovered with MLRsearch.
+
+Details
+~~~~~~~
+
+See :ref:`plrsearch_algorithm` section for more detail. PLRsearch is
+being standardized in IETF in `draft-vpolak-bmwg-plrsearch
+<https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch>`_.
+
+Generic Test Properties
+^^^^^^^^^^^^^^^^^^^^^^^
+
+All data plane throughput test methodologies share following generic
+properties:
+
+- Tested L2 frame sizes (untagged Ethernet):
+
+ - IPv4 payload: 64B, IMIX (28x64B, 16x570B, 4x1518B), 1518B, 9000B.
+ - IPv6 payload: 78B, IMIX (28x78B, 16x570B, 4x1518B), 1518B, 9000B.
+ - All quoted sizes include frame CRC, but exclude per frame
+ transmission overhead of 20B (preamble, inter frame gap).
+
+- Offered packet load is always bi-directional and symmetric.
+- All measured and reported packet and bandwidth rates are aggregate
+ bi-directional rates reported from external Traffic Generator
+ perspective.
--- /dev/null
+.. _mrr_throughput:
+
+MRR Throughput
+--------------
+
+Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests,
+as they provide a maximum "raw" throughput benchmark for development and
+testing community. MRR tests measure the packet forwarding rate under
+the maximum load offered by traffic generator over a set trial duration,
+regardless of packet loss.
+
+MRR tests are currently used for following test jobs:
+
+- Report performance comparison: 64B, IMIX for vhost, memif.
+- Daily performance trending: 64B, IMIX for vhost, memif.
+- Per-patch performance verification: 64B.
+- PLRsearch soaking tests: 64B.
+
+Maximum offered load for specific L2 Ethernet frame size is set to
+either the maximum bi-directional link rate or tested NIC model
+capacity, as follows:
+
+- For 10GE NICs the maximum packet rate load is 2x14.88 Mpps for 64B, a
+ 10GE bi-directional link rate.
+- For 25GE NICs the maximum packet rate load is 2x18.75 Mpps for 64B, a
+ 25GE bi-directional link sub-rate limited by 25GE NIC used on TRex TG,
+ XXV710.
+- For 40GE NICs the maximum packet rate load is 2x18.75 Mpps for 64B, a
+ 40GE bi-directional link sub-rate limited by 40GE NIC used on TRex
+ TG,XL710. Packet rate for other tested frame sizes is limited by
+ PCIeGen3 x8 bandwidth limitation of ~50Gbps.
+
+MRR test code implements multiple bursts of offered packet load and has
+two configurable burst parameters: individual trial duration and number
+of trials in a single burst. This enables more precise performance
+trending by providing more results data for analysis.
+
+Burst parameter settings vary between different tests using MRR:
+
+- MRR individual trial duration:
+
+ - Report performance comparison: 1 sec.
+ - Daily performance trending: 1 sec.
+ - Per-patch performance verification: 10 sec.
+ - PLRsearch soaking tests: 5.2 sec.
+
+- Number of MRR trials per burst:
+
+ - Report performance comparison: 10.
+ - Daily performance trending: 10.
+ - Per-patch performance verification: 5.
+ - PLRsearch soaking tests: 1.
\ No newline at end of file
+++ /dev/null
-.. _mrr_throughput:
-
-MRR Throughput
---------------
-
-Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests,
-as they provide a maximum "raw" throughput benchmark for development and
-testing community. MRR tests measure the packet forwarding rate under
-the maximum load offered by traffic generator over a set trial duration,
-regardless of packet loss. Maximum load for specified Ethernet frame
-size is set to the bi-directional link rate.
-
-In |csit-release| MRR test code has been updated with a configurable
-burst MRR parameters: trial duration and number of trials in a single
-burst. This enabled more precise performance trending.
-
-Current parameters for MRR trending tests:
-
-- Ethernet frame sizes: 64B (78B for IPv6), IMIX, 1518B, 9000B; all
- quoted sizes include frame CRC, but exclude per frame transmission
- overhead of 20B (preamble, inter frame gap).
-
-- Maximum load offered: 10GE, 25GE and 40GE link (sub-)rates depending on NIC
- tested, with the actual packet rate depending on frame size,
- transmission overhead and traffic generator NIC forwarding capacity.
-
- - For 10GE NICs the maximum packet rate load is 2* 14.88 Mpps for 64B,
- a 10GE bi-directional link rate.
- - For 25GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B,
- a 25GE bi-directional link sub-rate limited by TG 25GE NIC used,
- XXV710.
- - For 40GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B,
- a 40GE bi-directional link sub-rate limited by TG 40GE NIC used,
- XL710. Packet rate for other tested frame sizes is limited by PCIe
- Gen3 x8 bandwidth limitation of ~50Gbps.
-
-- Trial duration: 1 sec.
-
-- Number of trials per burst: 10.
-
-Similarly to NDR/PDR throughput tests, MRR test should be reporting
-bi-directional link rate (or NIC rate, if lower) if tested VPP
-configuration can handle the packet rate higher than bi-directional link
-rate, e.g. large packet tests and/or multi-core tests.
-
-MRR tests are currently used for FD.io CSIT continuous performance
-trending and for comparison between releases. Daily trending job tests
-subset of frame sizes, focusing on 64B (78B for IPv6) for all tests and
-IMIX for selected tests (vhost, memif).
-
-MRR-like measurements are being used to establish starting conditions
-for experimental Probabilistic Loss Ratio Search (PLRsearch) used for
-soak testing, aimed at verifying continuous system performance over an
-extended period of time, hours, days, weeks, months. PLRsearch code is
-currently in experimental phase in FD.io CSIT project.
Code Review / csit.git / commitdiff