To execute performance tests, there are three identical testbeds, each testbed\r
consists of two SUTs and one TG.\r
\r
+Naming Convention\r
+-----------------\r
+\r
+Following naming convention is used within this page to specify physical\r
+connectivity and wiring across defined CSIT testbeds:\r
+\r
+- testbedname: testbedN.\r
+- hostname:\r
+\r
+ - traffic-generator: tN-tgW.\r
+ - system-under-testX: tN-sutX.\r
+\r
+- portnames:\r
+\r
+ - tN-tgW-cY/pZ.\r
+ - tN-sutX-cY/pZ.\r
+\r
+- where:\r
+\r
+ - N - testbed number.\r
+ - tgW - server acts as traffic-generator with W index.\r
+ - sutX - server acts as system-under-test with X index.\r
+ - Y - PCIe slot number denoting a NIC card number within the host.\r
+\r
+ - Y=1,2,3 - slots in Riser 1, Right PCIe Riser Board, NUMA node 0.\r
+ - Y=4,5,6 - slots in Riser 2, Left PCIe Riser Board, NUMA node 1.\r
+ - Y=m - the MLOM slot.\r
+\r
+ - Z - port number on the NIC card.\r
+\r
+Server HW Configuration\r
+-----------------------\r
+\r
+CSIT testbed contains following three HW configuration types of UCS x86 servers,\r
+across total of ten servers provided:\r
+\r
+#. Type-1: Purpose - VPP functional and performance conformance testing.\r
+\r
+ - Quantity: 6 computers as SUT hosts (Systems Under Test).\r
+ - Physical connectivity:\r
+\r
+ - CIMC and host management ports.\r
+ - NIC ports connected in 3-node topologies.\r
+\r
+ - Main HW configuration:\r
+\r
+ - Chassis: UCSC-C240-M4SX with 6 PCIe3.0 slots.\r
+ - Processors: 2* E5-2699 2.3 GHz.\r
+ - RAM Memory: 16* 32GB DDR4-2133MHz.\r
+ - Disks: 2* 2TB 12G SAS 7.2K RPM SFF HDD.\r
+\r
+ - NICs configuration:\r
+\r
+ - Right PCIe Riser Board (Riser 1) (x8, x8, x8 PCIe3.0 lanes)\r
+\r
+ - PCIe Slot1: Cisco VIC 1385 2p40GE.\r
+\r
+ - PCIe Slot2: Intel NIC x520 2p10GE.\r
+ - PCIe Slot3: empty.\r
+\r
+ - Left PCIe Riser Board (Riser 2) (x8, x16, x8 PCIe3.0 lanes)\r
+\r
+ - PCIe Slot4: Intel NIC xl710 2p40GE.\r
+ - PCIe Slot5: Intel NIC x710 2p10GE.\r
+ - PCIe Slot6: Intel QAT 8950 50G (Walnut Hill)\r
+\r
+ - MLOM slot: Cisco VIC 1227 2p10GE (x8 PCIe2.0 lanes).\r
+\r
+#. Type-2: Purpose - VPP functional and performance conformance testing.\r
+\r
+ - Quantity: 3 computers as TG hosts (Traffic Generators).\r
+ - Physical connectivity:\r
+\r
+ - CIMC and host management ports.\r
+ - NIC ports connected in 3-node topologies.\r
+\r
+ - Main HW configuration:\r
+\r
+ - Chassis: UCSC-C240-M4SX with 6 PCIe3.0 slots.\r
+ - Processors: 2* E5-2699 2.3 GHz.\r
+ - RAM Memory: 16* 32GB DDR4-2133MHz.\r
+ - Disks: 2* 2TB 12G SAS 7.2K RPM SFF HDD.\r
+\r
+ - NICs configuration:\r
+\r
+ - Right PCIe Riser Board (Riser 1) (x8, x8, x8 lanes)\r
+\r
+ - PCIe Slot1: Intel NIC xl710 2p40GE.\r
+ - PCIe Slot2: Intel NIC x710 2p10GE.\r
+ - PCIe Slot3: Intel NIC x710 2p10GE.\r
+\r
+ - Left PCIe Riser Board (Riser 2) (x8, x16, x8 lanes)\r
+\r
+ - PCIe Slot4: Intel NIC xl710 2p40GE.\r
+ - PCIe Slot5: Intel NIC x710 2p10GE.\r
+ - PCIe Slot6: Intel NIC x710 2p10GE.\r
+\r
+ - MLOM slot: empty.\r
+\r
+#. Type-3: Purpose - VIRL functional conformance.\r
+\r
+ - Quantity: 3 computers as VIRL hosts.\r
+ - Physical connectivity:\r
+\r
+ - CIMC and host management ports.\r
+ - no NIC ports, standalone setup.\r
+\r
+ - Main HW configuration:\r
+\r
+ - Chassis: UCSC-C240-M4SX with 6 PCIe3.0 slots.\r
+ - Processors: 2* E5-2699 2.3 GHz.\r
+ - RAM Memory: 16* 32GB DDR4-2133MHz.\r
+ - Disks: 2* 480 GB 2.5inch 6G SATA SSD.\r
+\r
+ - NICs configuration:\r
+\r
+ - Right PCIe Riser Board (Riser 1) (x8, x8, x8 lanes)\r
+\r
+ - no cards.\r
+\r
+ - Left PCIe Riser Board (Riser 2) (x8, x16, x8 lanes)\r
+\r
+ - no cards.\r
+\r
+ - MLOM slot: empty.\r
+\r
SUT Configuration - Host HW\r
---------------------------\r
Host hardware details (CPU, memory, NIC layout) and physical topology are\r
-described in detail in\r
-`LF FDio CSIT testbed wiki page <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.\r
+described in detail in `LF FDio CSIT testbed wiki page\r
+<https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.\r
\r
**Host configuration**\r
\r
-- All hosts are Cisco UCS C240-M4 (2x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz,\r
- 18c, 512GB RAM)\r
- ::\r
+All hosts are Cisco UCS C240-M4 (2x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz,\r
+18c, 512GB RAM)\r
+\r
+::\r
\r
$ lscpu\r
Architecture: x86_64\r
NUMA node1 CPU(s): 18-35\r
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts\r
\r
-- BIOS settings\r
- ::\r
+**BIOS settings**\r
+\r
+::\r
\r
C240 /bios # show advanced detail\r
Set-up parameters:\r
CDN Support for VIC: Disabled\r
Out-of-Band Management: Disabled\r
\r
-- In addition to CIMC and Management, each TG has 4x Intel X710 10GB NIC\r
- (=8 ports) and 2x Intel XL710 40GB NIC (=4 ports), whereas each SUT has:\r
+**NIC models and placement**\r
\r
- - 1x Intel X520 NIC (10GB, 2 ports),\r
- - 1x Cisco VIC 1385 (40GB, 2 ports),\r
- - 1x Intel XL710 NIC (40GB, 2 ports),\r
- - 1x Intel X710 NIC (10GB, 2 ports),\r
- - 1x Cisco VIC 1227 (10GB, 2 ports).\r
- - This allows for a total of five ring topologies, each using ports on\r
- specific NIC model, enabling per NIC model benchmarking.\r
+In addition to CIMC and Management, each TG has 4x Intel X710 10GB NIC\r
+(=8 ports) and 2x Intel XL710 40GB NIC (=4 ports), whereas each SUT has:\r
\r
-**NIC models and placement**\r
+- 1x Intel X520 NIC (10GB, 2 ports),\r
+- 1x Cisco VIC 1385 (40GB, 2 ports),\r
+- 1x Intel XL710 NIC (40GB, 2 ports),\r
+- 1x Intel X710 NIC (10GB, 2 ports),\r
+- 1x Cisco VIC 1227 (10GB, 2 ports).\r
+\r
+This allows for a total of five ring topologies, each using ports on specific\r
+NIC model, enabling per NIC model benchmarking.\r
\r
- 0a:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+\r
Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server\r
SUT Configuration - Host OS Linux\r
---------------------------------\r
\r
-Software details (OS, configuration) are described in\r
-`CSIT/CSIT_LF_testbed <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.\r
+Software details (OS, configuration) are described in FD.io wiki `LF FDio CSIT\r
+testbed wiki page <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.\r
\r
-Below a subset of the configuration:\r
+System provisioning is done by combination of PXE boot unattented install and\r
+`Ansible <https://www.ansible.com>`_ described in `CSIT Testbed Setup`_.\r
+\r
+Below a subset of the running configuration:\r
\r
::\r
\r
Node 1 HugePages_Free: 2048\r
Node 1 HugePages_Surp: 0\r
\r
+**Kernel boot parameters used in CSIT performance testbeds**\r
+\r
+- **isolcpus=<cpu number>-<cpu number>** used for all cpu cores apart from\r
+ first core of each socket used for running VPP worker threads and Qemu/LXC\r
+ processes https://www.kernel.org/doc/Documentation/kernel-parameters.txt\r
+- **intel_pstate=disable** - [X86] Do not enable intel_pstate as the default\r
+ scaling driver for the supported processors. Intel P-State driver decide what\r
+ P-state (CPU core power state) to use based on requesting policy from the\r
+ cpufreq core. [X86 - Either 32-bit or 64-bit x86]\r
+ https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt\r
+- **nohz_full=<cpu number>-<cpu number>** - [KNL,BOOT] In kernels built with\r
+ CONFIG_NO_HZ_FULL=y, set the specified list of CPUs whose tick will be stopped\r
+ whenever possible. The boot CPU will be forced outside the range to maintain\r
+ the timekeeping. The CPUs in this range must also be included in the\r
+ rcu_nocbs= set. Specifies the adaptive-ticks CPU cores, causing kernel to\r
+ avoid sending scheduling-clock interrupts to listed cores as long as they have\r
+ a single runnable task. [KNL - Is a kernel start-up parameter, SMP - The\r
+ kernel is an SMP kernel].\r
+ https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt\r
+- **rcu_nocbs** - [KNL] In kernels built with CONFIG_RCU_NOCB_CPU=y, set the\r
+ specified list of CPUs to be no-callback CPUs, that never queue RCU callbacks\r
+ (read-copy update).\r
+ https://www.kernel.org/doc/Documentation/kernel-parameters.txt\r
+\r
+**Applied command line boot parameters:**\r
+\r
::\r
\r
$ cat /proc/cmdline\r
BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=UUID=35ea11e4-e44f-4f67-8cbe-12f09c49ed90 ro isolcpus=1-17,19-35 nohz_full=1-17,19-35 rcu_nocbs=1-17,19-35 intel_pstate=disable console=tty0 console=ttyS0,115200n8\r
\r
+**Mount listing**\r
+\r
::\r
\r
$ cat /proc/mounts\r
none /mnt/huge hugetlbfs rw,relatime,pagesize=2048k 0 0\r
lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0\r
\r
+**Package listing**\r
+\r
::\r
\r
$ dpkg -l\r
ii zlib1g:amd64 1:1.2.8.dfsg-2ubuntu4 amd64 compression library - runtime\r
ii zlib1g-dev:amd64 1:1.2.8.dfsg-2ubuntu4 amd64 compression library - development\r
\r
+**Kernel module listing**\r
+\r
::\r
\r
$ lsmod | sort\r
xt_CHECKSUM 16384 1\r
xt_tcpudp 16384 5\r
\r
+**Sysctl listing**\r
+\r
::\r
\r
$ sysctl -a\r
vm.vfs_cache_pressure = 100\r
vm.zone_reclaim_mode = 0\r
\r
+**Services listing**\r
+\r
::\r
\r
$ service --status-all\r
[ + ] uuidd\r
[ - ] x11-common\r
\r
+**Host CFS optimizations (QEMU+VPP)**\r
+\r
+Applying CFS scheduler tuning on all Qemu vcpu worker threads (those are\r
+handling testpmd - pmd threads) and VPP PMD worker threads. List of VPP PMD\r
+threads can be obtained e.g. from:\r
+\r
+::\r
+\r
+ $ for psid in $(pgrep vpp)\r
+ $ do\r
+ $ for tid in $(ps -Lo tid --pid $psid | grep -v TID)\r
+ $ do\r
+ $ echo $tid\r
+ $ done\r
+ $ done\r
+\r
+Or:\r
+\r
+::\r
+\r
+ $ cat /proc/`pidof vpp`/task/*/stat | awk '{print $1" "$2" "$39}'\r
+\r
+Applying Round-robin scheduling with highest priority\r
+\r
+::\r
+\r
+ $ for psid in $(pgrep vpp)\r
+ $ do\r
+ $ for tid in $(ps -Lo tid --pid $psid | grep -v TID)\r
+ $ do\r
+ $ chrt -r -p 1 $tid\r
+ $ done\r
+ $ done\r
+\r
+More information about Linux CFS can be found in: `Sched manual pages\r
+<http://man7.org/linux/man-pages/man7/sched.7.html>`_.\r
+\r
+\r
+**Host IRQ affinity**\r
+\r
+Changing the default pinning of every IRQ to core 0. (Same does apply on both\r
+guest VM and host OS)\r
+\r
+::\r
+\r
+ $ for l in `ls /proc/irq`; do echo 1 | sudo tee /proc/irq/$l/smp_affinity; done\r
+\r
+**Host RCU affinity**\r
+\r
+Changing the default pinning of RCU to core 0. (Same does apply on both guest VM\r
+and host OS)\r
+\r
+::\r
+\r
+ $ for i in `pgrep rcu[^c]` ; do sudo taskset -pc 0 $i ; done\r
+\r
+**Host Writeback affinity**\r
+\r
+Changing the default pinning of writebacks to core 0. (Same does apply on both\r
+guest VM and host OS)\r
+\r
+::\r
+\r
+ $ echo 1 | sudo tee /sys/bus/workqueue/devices/writeback/cpumask\r
+\r
DUT Configuration - DPDK\r
------------------------\r
\r
\r
.. code-block:: bash\r
\r
- testpmd -c 0x403 -n 4 -- --numa --nb-ports=2 --portmask=0x3 --nb-cores=2 --max-pkt-len=9000 --txqflags=0 --forward-mode=io --rxq=1 --txq=1 --burst=64 --burst=64 --rxd=1024 --txd=1024 --disable-link-check --auto-start\r
+ testpmd -c 0x403 -n 4 -- --numa --nb-ports=2 --portmask=0x3 --nb-cores=2 --max-pkt-len=9000 --txqflags=0 --forward-mode=io --rxq=1 --txq=1 --burst=64 --rxd=1024 --txd=1024 --disable-link-check --auto-start\r
\r
Tagged by **4T4C**\r
\r
.. code-block:: bash\r
\r
- testpmd -c 0xc07 -n 4 -- --numa --nb-ports=2 --portmask=0x3 --nb-cores=4 --max-pkt-len=9000 --txqflags=0 --forward-mode=io --rxq=2 --txq=2 --burst=64 --burst=64 --rxd=1024 --txd=1024 --disable-link-check --auto-start\r
+ testpmd -c 0xc07 -n 4 -- --numa --nb-ports=2 --portmask=0x3 --nb-cores=4 --max-pkt-len=9000 --txqflags=0 --forward-mode=io --rxq=2 --txq=2 --burst=64 --rxd=1024 --txd=1024 --disable-link-check --auto-start\r
\r
**L3FWD Startup Configuration**\r
\r
- dest_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf4]\r
src_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf5]\r
\r
+**TG Startup Command**\r
+\r
+::\r
+\r
+ $ sh -c 'cd <t-rex-install-dir>/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --iom 0 > /dev/null 2>&1 &'> /dev/null\r
+\r
**TG common API - pointer to driver**\r
\r
`TRex driver`_\r
-\r
Code Review / csit.git / blobdiff