X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Freport%2Fvpp_performance_tests%2Fpacket_throughput_graphs%2Findex.rst;h=186a058177c80aa7fdbef5bec4d3521f4f8ae046;hp=7c2a1a9e263b70e5360ee6475cef9ce0eeedb68f;hb=9dd388775bc42b334223b3a98909e9ee0e663dc7;hpb=7b213e217377d9862c1aef426fd0b6b27b3e6d8d diff --git a/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst b/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst index 7c2a1a9e26..186a058177 100644 --- a/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst +++ b/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst @@ -1,47 +1,165 @@ -Packet Throughput Graphs -======================== - -Plotted results are generated by multiple executions of the same CSIT -performance tests across three physical testbeds within LF FD.io labs. -To provide a descriptive summary view, Box-and-Whisker plots are used to -display variation in measured throughput values, without making any -assumptions of the underlying statistical distribution. - -For each plotted test case, Box-and-Whisker plots show the quartiles -(Min, 1st quartile / 25th percentile, 2nd quartile / 50th percentile / -mean, 3rd quartile / 75th percentile, Max) across collected data set -(data set size stated in the note below). Outliers are plotted as -individual points. Min and max values are plotted as bottom and top -Whiskers respectively. 2nd and 3rd quartiles are plotted as bottom and -top edge of the box. If multiple samples match only two values, and all -samples fall between them, then no whiskers are plotted. If all samples -have the same value, only a horizontal line is plotted. - -*Title of each graph* is a regex (regular expression) matching all -throughput test cases plotted on this graph, *X-axis labels* are indices -of individual test suites executed by -`FD.io test executor vpp performance jobs`_ jobs that created result output -files used as data sources for the graph, *Y-axis labels* are measured Packets -Per Second [pps] values, and the *Graph legend* lists the plotted test suites -and their indices. +.. raw:: latex + + \clearpage + +.. _VPP_Packet_Throughput: + +Packet Throughput +================= + +Throughput graphs are generated based on the results data obtained from +the |csit-release| test jobs. In order to verify benchmark results +repeatibility selected, CSIT performance tests are executed multiple +times (target: 10 times) on each physical testbed type. Box-and-Whisker +plots are used to display variations in measured throughput values. + +Lists of tests selected for multiple execution and graphing are captured +per testbed type in `test_select_list_{testbed_type}.md +`_ files. + +Graphs are split into sections as follows: + +#. **Header 1**: VPP packet path and lookup types + + - **L2 Ethernet Switching**: L2 bridge-doman, L2 cross-connect and L2 patch + - **IPv4 Routing**: IPv4 routing with /32 prefixes + - **IPv6 Routing**: IPv6 routing with /128 prefixes + - **SRv6 Routing**: SRv6 with IPv6 routing + - **IPv4 Tunnels**: IPv4 overlay tunnels + - **KVM VMs vhost-user**: KVM VMs connected over virtio and vhost-user interfaces + - **LXC/DRC Container Memif**: Linux containers and Docker containers connected over Memif interfaces + - **IPsec IPv4 Routing**: IPsec encryption/decryption with IPv4 routing + +#. **Header 2**: testbeds and NIC models + + - section name format: + + - {**testbed_type**}-{**nic_model**} + + - **testbed_type**: + + - 2n-skx: 2-node Xeon Skylake + - 3n-skx: 3-node Xeon Skylake + - 2n-clx: 2-node Xeon Cascade Lake + - 3n-tsh: 3-node Arm TaiShan + - 2n-tx2: 2-node Arm ThunderX2 + - 2n zn2: 2-node AMD Zen2 + - 2n-dnv: 2-node Atom Denverton + - 3n-dnv: 3-node Atom Denverton + - 2n-aws: 2-node AWS + - 3n-aws: 3-node AWS + - 2n-icx: 2-node Xeon Icelake + - 3n-icx: 3-node Xeon Icelake + + - **nic_model**: + + - xxv710: xxv710 2p25GE Intel (Fortville) + - x710: x710 4p10GE Intel (Fortville) + - xl710: xl710 2p40GE Intel (Fortville) + - x520: x520 2p10GE Intel (Niantic) + - x553: x553 2p10GE Intel (Niantic) + - cx556a: cx556a-edat 2p100GE Mellanox ConnectX5 + - e810cq: E810-2CQDA2 2p100GE Intel Columbiaville + +#. **Header 3**: test group names + + - section name format: + + - {**frame_size**}-{**worker_thread_core_cfg**}-{**vpp_functionality**}-{**vpp_lookup_type**}-{**baseline_scale**}-{**nic_driver**} + + - **frame_size**: + + - 64b: 64 byte frames, smallest frame size for untagged IPv4 packets + - 78b: 78 byte frames, smallest frame size for untagged IPv6 packets + - 114b: VXLAN encapsulated L2 frames + - imix: a sequence of (7x64B, 4x570, 1x1518) byte frames + + - **worker_thread_core_cfg**: + + - 1t1c: 1 worker thread on 1 core, hyper-threading not used + - 2t1c: 2 worker threads on 1 core, hyper-threading used + + - **vpp_functionality** (optional): + + - features: including input-acl, output-acl, macip-iacl, nat44 + - srv6: srv6 encap/decap, proxy + - link-bonding: L2 link aggregation with 1 or 2 bonded links + - ipsec: IPsec encryption/decryption with different ciphers + - vts: Virtual Topology System specific tests + + - **vpp_lookup_type**: + + - l2switching, ip4routing, ip6routing, ip4tunnel, vhost, memif + + - **baseline_scale**: + + - base: baseline tests with less than 10 forwarding entries + - scale: scale tests with up to 2 million forwarding entries + - base-scale: both baseline and scale tests grouped together + + - **nic_driver**: + + - avf: VPP native avf driver for Intel Fortville NICs + - i40e: dpdk poll mode driver for Intel Fortville NICs + - ixgbe: dpdk poll mode driver for Intel Niantic NICs + +For each test case, Box-and-Whisker plots show the quartiles (Min, 1st +quartile / 25th percentile, 2nd quartile / 50th percentile / mean, 3rd +quartile / 75th percentile, Max) across collected data set. Outliers are +plotted as individual points. + +Additional information about graph data: + +#. **Graph Title**: describes tested packet path, testbed topology, + processor model, NIC model, packet size, number of cores and + threads used by data plane workers and indication of VPP DUT + configuration. + +#. **X-axis Labels**: indices of individual test suites as listed in + Graph Legend. + +#. **Y-axis Labels**: measured Packets Per Second [pps] throughput + values. + +#. **Graph Legend**: lists X-axis indices with associated CSIT test + suites executed to generate graphed test results. + +#. **Hover Information**: lists minimum, first quartile, median, + third quartile, and maximum. If either type of outlier is present the + whisker on the appropriate side is taken to 1.5×IQR from the quartile + (the "inner fence") rather than the max or min, and individual outlying + data points are displayed as unfilled circles (for suspected outliers) + or filled circles (for outliers). (The "outer fence" is 3×IQR from the + quartile.) .. note:: - Test results have been generated by - `FD.io test executor vpp performance jobs`_ with Robot Framework result - files csit-vpp-perf-\*.zip `archived here <../../_static/archive/>`_. - Plotted data set size per test case is equal to the number of job executions - presented in this report version: **10**. + Test results are stored in + `build logs from FD.io vpp performance job 2n-aws`_, + `build logs from FD.io vpp performance job 3n-aws`_, + `build logs from FD.io vpp performance job 2n-skx`_, + `build logs from FD.io vpp performance job 3n-skx`_, + `build logs from FD.io vpp performance job 2n-clx`_, + `build logs from FD.io vpp performance job 2n-zn2`_, + `build logs from FD.io vpp performance job 3n-tsh`_, + `build logs from FD.io vpp performance job 2n-tx2`_, + `build logs from FD.io vpp performance job 2n-dnv`_ and + `build logs from FD.io vpp performance job 3n-dnv`_ with RF + result files csit-vpp-perf-|srelease|-\*.zip + `archived here <../../_static/archive/>`_. + Required per test case data set size is **10**, but for VPP tests + the actual size varies per test case and is <=10. .. toctree:: + :maxdepth: 3 l2 ip4 ip6 + srv6 ip4_tunnels - ip6_tunnels + nat44 vm_vhost container_memif - container_orchestrated ipsec