6 .. _VPP_Packet_Throughput:
11 Throughput graphs are generated based on the results data obtained from
12 the |csit-release| test jobs. In order to verify benchmark results
13 repeatibility selected, CSIT performance tests are executed multiple
14 times (target: 10 times) on each physical testbed type. Box-and-Whisker
15 plots are used to display variations in measured throughput values.
17 Lists of tests selected for multiple execution and graphing are captured
18 per testbed type in `test_select_list_{testbed_type}.md
19 <https://git.fd.io/csit/tree/docs/job_specs>`_ files.
21 Graphs are split into sections as follows:
23 #. **Header 1**: VPP packet path and lookup types
25 - **L2 Ethernet Switching**: L2 bridge-doman, L2 cross-connect and L2 patch
26 - **IPv4 Routing**: IPv4 routing with /32 prefixes
27 - **IPv6 Routing**: IPv6 routing with /128 prefixes
28 - **SRv6 Routing**: SRv6 with IPv6 routing
29 - **IPv4 Tunnels**: IPv4 overlay tunnels
30 - **KVM VMs vhost-user**: KVM VMs connected over virtio and vhost-user interfaces
31 - **LXC/DRC Container Memif**: Linux containers and Docker containers connected over Memif interfaces
32 - **IPsec IPv4 Routing**: IPsec encryption/decryption with IPv4 routing
34 #. **Header 2**: testbeds and NIC models
36 - section name format:
38 - {**testbed_type**}-{**nic_model**}
42 - 2n-icx: 2-node Xeon Icelake
43 - 3n-icx: 3-node Xeon Icelake
45 - 2n-skx: 2-node Xeon Skylake
46 - 3n-skx: 3-node Xeon Skylake
47 - 2n-clx: 2-node Xeon Cascade Lake
48 - 2n-zn2: 2-node AMD Zen2
49 - 3n-alt: 2-node Arm Altra
50 - 3n-tsh: 3-node Arm TaiShan
51 - 2n-tx2: 2-node Arm ThunderX2
52 - 2n-dnv: 2-node Atom Denverton
53 - 3n-dnv: 3-node Atom Denverton
57 - xxv710: xxv710 2p25GE Intel (Fortville)
58 - x710: x710 4p10GE Intel (Fortville)
59 - xl710: xl710 2p40GE Intel (Fortville)
60 - x520: x520 2p10GE Intel (Niantic)
61 - x553: x553 2p10GE Intel (Niantic)
62 - cx556a: cx556a-edat 2p100GE Mellanox ConnectX5
63 - e810cq: E810-2CQDA2 2p100GE Intel Columbiaville
65 #. **Header 3**: test group names
67 - section name format:
69 - {**frame_size**}-{**worker_thread_core_cfg**}-{**vpp_functionality**}-{**vpp_lookup_type**}-{**baseline_scale**}-{**nic_driver**}
73 - 64b: 64 byte frames, smallest frame size for untagged IPv4 packets
74 - 78b: 78 byte frames, smallest frame size for untagged IPv6 packets
75 - 114b: VXLAN encapsulated L2 frames
76 - imix: a sequence of (7x64B, 4x570, 1x1518) byte frames
78 - **worker_thread_core_cfg**:
80 - 1t1c: 1 worker thread on 1 core, hyper-threading not used
81 - 2t1c: 2 worker threads on 1 core, hyper-threading used
83 - **vpp_functionality** (optional):
85 - features: including input-acl, output-acl, macip-iacl, nat44
86 - srv6: srv6 encap/decap, proxy
87 - link-bonding: L2 link aggregation with 1 or 2 bonded links
88 - ipsec: IPsec encryption/decryption with different ciphers
89 - vts: Virtual Topology System specific tests
91 - **vpp_lookup_type**:
93 - l2switching, ip4routing, ip6routing, ip4tunnel, vhost, memif
97 - base: baseline tests with less than 10 forwarding entries
98 - scale: scale tests with up to 2 million forwarding entries
99 - base-scale: both baseline and scale tests grouped together
103 - avf: VPP native avf driver for Intel Fortville NICs
104 - i40e: dpdk poll mode driver for Intel Fortville NICs
105 - ixgbe: dpdk poll mode driver for Intel Niantic NICs
107 For each test case, Box-and-Whisker plots show the quartiles (Min, 1st
108 quartile / 25th percentile, 2nd quartile / 50th percentile / mean, 3rd
109 quartile / 75th percentile, Max) across collected data set. Outliers are
110 plotted as individual points.
112 Additional information about graph data:
114 #. **Graph Title**: describes tested packet path, testbed topology,
115 processor model, NIC model, packet size, number of cores and
116 threads used by data plane workers and indication of VPP DUT
119 #. **X-axis Labels**: indices of individual test suites as listed in
122 #. **Y-axis Labels**: measured Packets Per Second [pps] throughput
125 #. **Graph Legend**: lists X-axis indices with associated CSIT test
126 suites executed to generate graphed test results.
128 #. **Hover Information**: lists minimum, first quartile, median,
129 third quartile, and maximum. If either type of outlier is present the
130 whisker on the appropriate side is taken to 1.5×IQR from the quartile
131 (the "inner fence") rather than the max or min, and individual outlying
132 data points are displayed as unfilled circles (for suspected outliers)
133 or filled circles (for outliers). (The "outer fence" is 3×IQR from the
138 Test results are stored in
139 `build logs from FD.io vpp performance job 2n-icx`_,
140 `build logs from FD.io vpp performance job 3n-icx`_,
141 `build logs from FD.io vpp performance job 2n-aws`_,
142 `build logs from FD.io vpp performance job 2n-skx`_,
143 `build logs from FD.io vpp performance job 3n-skx`_,
144 `build logs from FD.io vpp performance job 2n-clx`_,
145 `build logs from FD.io vpp performance job 2n-zn2`_,
146 `build logs from FD.io vpp performance job 3n-alt`_,
147 `build logs from FD.io vpp performance job 3n-tsh`_,
148 `build logs from FD.io vpp performance job 2n-tx2`_,
149 `build logs from FD.io vpp performance job 2n-dnv`_ and
150 `build logs from FD.io vpp performance job 3n-dnv`_ with RF
151 result files csit-vpp-perf-|srelease|-\*.zip
152 `archived here <../../_static/archive/>`_.
153 Required per test case data set size is **10**, but for VPP tests
154 the actual size varies per test case and is <=10.