-Packet Throughput Graphs
-========================
-
-Plotted results are generated by multiple executions of the same CSIT
-performance tests across three physical testbeds within LF FD.io labs.
-To provide a descriptive summary view, Box-and-Whisker plots are used to
-display variation in measured throughput values, without making any
-assumptions of the underlying statistical distribution.
-
-For each plotted test case, Box-and-Whisker plots show the quartiles
-(Min, 1st quartile / 25th percentile, 2nd quartile / 50th percentile /
-mean, 3rd quartile / 75th percentile, Max) across collected data set
-(data set size stated in the note below). Outliers are plotted as
-individual points. Min and max values are plotted as bottom and top
-Whiskers respectively. 2nd and 3rd quartiles are plotted as bottom and
-top edge of the box. If multiple samples match only two values, and all
-samples fall between them, then no whiskers are plotted. If all samples
-have the same value, only a horizontal line is plotted.
-
-*Title of each graph* is a regex (regular expression) matching all
-throughput test cases plotted on this graph, *X-axis labels* are indices
-of individual test suites executed by
-`FD.io test executor vpp performance jobs`_ jobs that created result output
-files used as data sources for the graph, *Y-axis labels* are measured Packets
-Per Second [pps] values, and the *Graph legend* lists the plotted test suites
-and their indices.
+.. raw:: latex
+
+ \clearpage
+
+.. _VPP_Packet_Throughput:
+
+Packet Throughput
+=================
+
+Throughput graphs are generated based on the results data obtained from
+the |csit-release| test jobs. In order to verify benchmark results
+repeatibility selected, CSIT performance tests are executed multiple
+times (target: 10 times) on each physical testbed type. Box-and-Whisker
+plots are used to display variations in measured throughput values.
+
+Lists of tests selected for multiple execution and graphing are captured
+per testbed type in `test_select_list_{testbed_type}.md
+<https://git.fd.io/csit/tree/docs/job_specs>`_ files.
+
+Graphs are split into sections as follows:
+
+#. **Header 1**: VPP packet path and lookup types
+
+ - **L2 Ethernet Switching**: L2 bridge-doman, L2 cross-connect and L2 patch
+ - **IPv4 Routing**: IPv4 routing with /32 prefixes
+ - **IPv6 Routing**: IPv6 routing with /128 prefixes
+ - **SRv6 Routing**: SRv6 with IPv6 routing
+ - **IPv4 Tunnels**: IPv4 overlay tunnels
+ - **KVM VMs vhost-user**: KVM VMs connected over virtio and vhost-user interfaces
+ - **LXC/DRC Container Memif**: Linux containers and Docker containers connected over Memif interfaces
+ - **IPsec IPv4 Routing**: IPsec encryption/decryption with IPv4 routing
+
+#. **Header 2**: testbeds and NIC models
+
+ - section name format:
+
+ - {**testbed_type**}-{**nic_model**}
+
+ - **testbed_type**:
+
+ - 2n-skx: 2-node Xeon Skylake
+ - 3n-skx: 3-node Xeon Skylake
+ - 2n-clx: 2-node Xeon Cascade Lake
+ - 3n-tsh: 3-node Arm TaiShan
+ - 2n-tx2: 2-node Arm ThunderX2
+ - 2n zn2: 2-node AMD Zen2
+ - 2n-dnv: 2-node Atom Denverton
+ - 3n-dnv: 3-node Atom Denverton
+ - 2n-aws: 2-node AWS
+ - 3n-aws: 3-node AWS
+ - 2n-icx: 2-node Xeon Icelake
+ - 3n-icx: 3-node Xeon Icelake
+
+ - **nic_model**:
+
+ - xxv710: xxv710 2p25GE Intel (Fortville)
+ - x710: x710 4p10GE Intel (Fortville)
+ - xl710: xl710 2p40GE Intel (Fortville)
+ - x520: x520 2p10GE Intel (Niantic)
+ - x553: x553 2p10GE Intel (Niantic)
+ - cx556a: cx556a-edat 2p100GE Mellanox ConnectX5
+ - e810cq: E810-2CQDA2 2p100GE Intel Columbiaville
+
+#. **Header 3**: test group names
+
+ - section name format:
+
+ - {**frame_size**}-{**worker_thread_core_cfg**}-{**vpp_functionality**}-{**vpp_lookup_type**}-{**baseline_scale**}-{**nic_driver**}
+
+ - **frame_size**:
+
+ - 64b: 64 byte frames, smallest frame size for untagged IPv4 packets
+ - 78b: 78 byte frames, smallest frame size for untagged IPv6 packets
+ - 114b: VXLAN encapsulated L2 frames
+ - imix: a sequence of (7x64B, 4x570, 1x1518) byte frames
+
+ - **worker_thread_core_cfg**:
+
+ - 1t1c: 1 worker thread on 1 core, hyper-threading not used
+ - 2t1c: 2 worker threads on 1 core, hyper-threading used
+
+ - **vpp_functionality** (optional):
+
+ - features: including input-acl, output-acl, macip-iacl, nat44
+ - srv6: srv6 encap/decap, proxy
+ - link-bonding: L2 link aggregation with 1 or 2 bonded links
+ - ipsec: IPsec encryption/decryption with different ciphers
+ - vts: Virtual Topology System specific tests
+
+ - **vpp_lookup_type**:
+
+ - l2switching, ip4routing, ip6routing, ip4tunnel, vhost, memif
+
+ - **baseline_scale**:
+
+ - base: baseline tests with less than 10 forwarding entries
+ - scale: scale tests with up to 2 million forwarding entries
+ - base-scale: both baseline and scale tests grouped together
+
+ - **nic_driver**:
+
+ - avf: VPP native avf driver for Intel Fortville NICs
+ - i40e: dpdk poll mode driver for Intel Fortville NICs
+ - ixgbe: dpdk poll mode driver for Intel Niantic NICs
+
+For each test case, Box-and-Whisker plots show the quartiles (Min, 1st
+quartile / 25th percentile, 2nd quartile / 50th percentile / mean, 3rd
+quartile / 75th percentile, Max) across collected data set. Outliers are
+plotted as individual points.
+
+Additional information about graph data:
+
+#. **Graph Title**: describes tested packet path, testbed topology,
+ processor model, NIC model, packet size, number of cores and
+ threads used by data plane workers and indication of VPP DUT
+ configuration.
+
+#. **X-axis Labels**: indices of individual test suites as listed in
+ Graph Legend.
+
+#. **Y-axis Labels**: measured Packets Per Second [pps] throughput
+ values.
+
+#. **Graph Legend**: lists X-axis indices with associated CSIT test
+ suites executed to generate graphed test results.
+
+#. **Hover Information**: lists minimum, first quartile, median,
+ third quartile, and maximum. If either type of outlier is present the
+ whisker on the appropriate side is taken to 1.5×IQR from the quartile
+ (the "inner fence") rather than the max or min, and individual outlying
+ data points are displayed as unfilled circles (for suspected outliers)
+ or filled circles (for outliers). (The "outer fence" is 3×IQR from the
+ quartile.)
.. note::
- Test results have been generated by
- `FD.io test executor vpp performance jobs`_ with Robot Framework result
- files csit-vpp-perf-\*.zip `archived here <../../_static/archive/>`_.
- Plotted data set size per test case is equal to the number of job executions
- presented in this report version: **10**.
+ Test results are stored in
+ `build logs from FD.io vpp performance job 2n-aws`_,
+ `build logs from FD.io vpp performance job 3n-aws`_,
+ `build logs from FD.io vpp performance job 2n-skx`_,
+ `build logs from FD.io vpp performance job 3n-skx`_,
+ `build logs from FD.io vpp performance job 2n-clx`_,
+ `build logs from FD.io vpp performance job 2n-zn2`_,
+ `build logs from FD.io vpp performance job 3n-tsh`_,
+ `build logs from FD.io vpp performance job 2n-tx2`_,
+ `build logs from FD.io vpp performance job 2n-dnv`_ and
+ `build logs from FD.io vpp performance job 3n-dnv`_ with RF
+ result files csit-vpp-perf-|srelease|-\*.zip
+ `archived here <../../_static/archive/>`_.
+ Required per test case data set size is **10**, but for VPP tests
+ the actual size varies per test case and is <=10.
.. toctree::
+ :maxdepth: 3
l2
ip4
ip6
+ srv6
ip4_tunnels
- ip6_tunnels
+ nat44
vm_vhost
+ container_memif
ipsec
Code Review / csit.git / blobdiff