X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Freport%2Fvpp_performance_tests%2Fpacket_throughput_graphs%2Findex.rst;h=186a058177c80aa7fdbef5bec4d3521f4f8ae046;hp=8020809f7160ce3fd031d91d83dbd690fa7df781;hb=9dd388775bc42b334223b3a98909e9ee0e663dc7;hpb=b6d247550a5ff8c3893fb918ea328fb0f1f9797d diff --git a/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst b/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst index 8020809f71..186a058177 100644 --- a/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst +++ b/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst @@ -3,14 +3,106 @@ \clearpage +.. _VPP_Packet_Throughput: + Packet Throughput ================= -Throughput graphs are generated by multiple executions of the same -performance tests across physical testbeds hosted LF FD.io labs: 3n-hsw, -2n-skx, 2n-skx. Box-and-Whisker plots are used to display variations in -measured throughput values, without making any assumptions of the -underlying statistical distribution. +Throughput graphs are generated based on the results data obtained from +the |csit-release| test jobs. In order to verify benchmark results +repeatibility selected, CSIT performance tests are executed multiple +times (target: 10 times) on each physical testbed type. Box-and-Whisker +plots are used to display variations in measured throughput values. + +Lists of tests selected for multiple execution and graphing are captured +per testbed type in `test_select_list_{testbed_type}.md +`_ files. + +Graphs are split into sections as follows: + +#. **Header 1**: VPP packet path and lookup types + + - **L2 Ethernet Switching**: L2 bridge-doman, L2 cross-connect and L2 patch + - **IPv4 Routing**: IPv4 routing with /32 prefixes + - **IPv6 Routing**: IPv6 routing with /128 prefixes + - **SRv6 Routing**: SRv6 with IPv6 routing + - **IPv4 Tunnels**: IPv4 overlay tunnels + - **KVM VMs vhost-user**: KVM VMs connected over virtio and vhost-user interfaces + - **LXC/DRC Container Memif**: Linux containers and Docker containers connected over Memif interfaces + - **IPsec IPv4 Routing**: IPsec encryption/decryption with IPv4 routing + +#. **Header 2**: testbeds and NIC models + + - section name format: + + - {**testbed_type**}-{**nic_model**} + + - **testbed_type**: + + - 2n-skx: 2-node Xeon Skylake + - 3n-skx: 3-node Xeon Skylake + - 2n-clx: 2-node Xeon Cascade Lake + - 3n-tsh: 3-node Arm TaiShan + - 2n-tx2: 2-node Arm ThunderX2 + - 2n zn2: 2-node AMD Zen2 + - 2n-dnv: 2-node Atom Denverton + - 3n-dnv: 3-node Atom Denverton + - 2n-aws: 2-node AWS + - 3n-aws: 3-node AWS + - 2n-icx: 2-node Xeon Icelake + - 3n-icx: 3-node Xeon Icelake + + - **nic_model**: + + - xxv710: xxv710 2p25GE Intel (Fortville) + - x710: x710 4p10GE Intel (Fortville) + - xl710: xl710 2p40GE Intel (Fortville) + - x520: x520 2p10GE Intel (Niantic) + - x553: x553 2p10GE Intel (Niantic) + - cx556a: cx556a-edat 2p100GE Mellanox ConnectX5 + - e810cq: E810-2CQDA2 2p100GE Intel Columbiaville + +#. **Header 3**: test group names + + - section name format: + + - {**frame_size**}-{**worker_thread_core_cfg**}-{**vpp_functionality**}-{**vpp_lookup_type**}-{**baseline_scale**}-{**nic_driver**} + + - **frame_size**: + + - 64b: 64 byte frames, smallest frame size for untagged IPv4 packets + - 78b: 78 byte frames, smallest frame size for untagged IPv6 packets + - 114b: VXLAN encapsulated L2 frames + - imix: a sequence of (7x64B, 4x570, 1x1518) byte frames + + - **worker_thread_core_cfg**: + + - 1t1c: 1 worker thread on 1 core, hyper-threading not used + - 2t1c: 2 worker threads on 1 core, hyper-threading used + + - **vpp_functionality** (optional): + + - features: including input-acl, output-acl, macip-iacl, nat44 + - srv6: srv6 encap/decap, proxy + - link-bonding: L2 link aggregation with 1 or 2 bonded links + - ipsec: IPsec encryption/decryption with different ciphers + - vts: Virtual Topology System specific tests + + - **vpp_lookup_type**: + + - l2switching, ip4routing, ip6routing, ip4tunnel, vhost, memif + + - **baseline_scale**: + + - base: baseline tests with less than 10 forwarding entries + - scale: scale tests with up to 2 million forwarding entries + - base-scale: both baseline and scale tests grouped together + + - **nic_driver**: + + - avf: VPP native avf driver for Intel Fortville NICs + - i40e: dpdk poll mode driver for Intel Fortville NICs + - ixgbe: dpdk poll mode driver for Intel Niantic NICs For each test case, Box-and-Whisker plots show the quartiles (Min, 1st quartile / 25th percentile, 2nd quartile / 50th percentile / mean, 3rd @@ -33,13 +125,27 @@ Additional information about graph data: #. **Graph Legend**: lists X-axis indices with associated CSIT test suites executed to generate graphed test results. +#. **Hover Information**: lists minimum, first quartile, median, + third quartile, and maximum. If either type of outlier is present the + whisker on the appropriate side is taken to 1.5×IQR from the quartile + (the "inner fence") rather than the max or min, and individual outlying + data points are displayed as unfilled circles (for suspected outliers) + or filled circles (for outliers). (The "outer fence" is 3×IQR from the + quartile.) .. note:: - Test results have been generated by - `FD.io test executor vpp performance job 3n-hsw`_, - `FD.io test executor vpp performance job 3n-skx`_ and - `FD.io test executor vpp performance job 2n-skx`_ with RF + Test results are stored in + `build logs from FD.io vpp performance job 2n-aws`_, + `build logs from FD.io vpp performance job 3n-aws`_, + `build logs from FD.io vpp performance job 2n-skx`_, + `build logs from FD.io vpp performance job 3n-skx`_, + `build logs from FD.io vpp performance job 2n-clx`_, + `build logs from FD.io vpp performance job 2n-zn2`_, + `build logs from FD.io vpp performance job 3n-tsh`_, + `build logs from FD.io vpp performance job 2n-tx2`_, + `build logs from FD.io vpp performance job 2n-dnv`_ and + `build logs from FD.io vpp performance job 3n-dnv`_ with RF result files csit-vpp-perf-|srelease|-\*.zip `archived here <../../_static/archive/>`_. Required per test case data set size is **10**, but for VPP tests @@ -53,9 +159,7 @@ Additional information about graph data: ip6 srv6 ip4_tunnels - ip6_tunnels + nat44 vm_vhost container_memif - container_orchestrated ipsec - vts