docs/lab/Testbeds_Xeon_Skx_Arm_Atom.md

   1 ## FD.io CSIT testbeds - Xeon Skylake, Arm, Atom
   2
   3 This is a low-level design implemented as an extensions of FD.io CSIT lab to
   4 accommodate the new Intel Xeon Skylake, Arm AArch64 and Atom devices.
   5
   6 ## Testbeds Overview
   7
   8 ### Testbeds Type Breakdown
   9
  10 ```
  11  #.  CSIT_tb.           Purpose.    SUT.  TG.   #tb.  #SUTs.  #TGs. #skx_node.
  12  1.  1-node Xeon.       func.       skx.  n/a.  2.    2.      0.    2.
  13  2.  2-node Xeon.       perf.       skx.  skx.  4.    4.      4.    8.
  14  3.  3-node Xeon.       perf.       skx.  skx.  2.    4.      2.    6.
  15  4.  tcp-l47.           tcp-stack.  skx.  ps1.  1.    1.      1.    1.
  16  5.  atom-netgate.      perf+func.  net.  skx.  1.    3.      1.    1.
  17  6.  aarch64-d05        perf        arm.  skx.  1.    2.      1.    1/2.
  18  7.  aarch64-mcbin      perf        arm.  skx.  1.    2.      1.    1/2.
  19  8.  1-node aarch64     func        arm.  n/a.  1.    1.      0.    0.
  20                                                  Total skx_node:   19.
  21 ```
  22
  23 ### 1-Node Xeon Testbeds
  24
  25 One 1-node Xeon testbed for VPP_Device tests is built using one SUT (Type-6
  26 server), with NIC ports connected back-to-back.
  27
  28 ### 2-Node Xeon Testbeds
  29
  30 Four 2-node Xeon testbeds (are expected to be built|are built), with each
  31 testbed using one SUTs (Type-1 server) and one TG (Type-2 server) connected
  32 back-to-back. NIC cards placement into slots and NIC ports connectivity is
  33 following the testbed specification included in next sections.
  34
  35 ### 3-Node Xeon Testbeds
  36
  37 Two 3-node Xeon testbeds (are expected to be built|are built), with each testbed
  38 using two SUTs (Type-1 server) and one TG (Type-2 server) connected in full-mesh
  39 triangle. NIC cards placement into slots and NIC ports connectivity is following
  40 the testbed specification included in next sections.
  41
  42 ### 1-Node Arm Marvell ThunderX2 Testbed
  43
  44 One 1-node ThunderX2 testbed for VPP_Device tests is expected to be built using
  45 one SUT (Type-9 server), with NIC ports connected back-to-back.
  46
  47 ### 3-Node Arm Huawei TaiShan Testbed
  48
  49 One 3-node TaiShan testbed is built, with each testbed using two SUTs (Type-3
  50 server) and one TG (Type-2 server) connected in full-mesh triangle. NIC cards
  51 placement into slots and NIC ports connectivity is following the testbed
  52 specification included in next sections.
  53
  54 ### 3-Node Arm Marvell MACCHIATObin Testbed
  55
  56 One 3-node MACCHIATObin testbeds is built, with each testbed using two SUTs
  57 (Type-4 server) and one TG (Type-2 server) connected in full-mesh triangle.
  58 Built-in NIC ports connectivity is following the testbed specification included
  59 in next sections.
  60
  61 ### TCP/IP and L47 Testbeds
  62
  63 One 2-node Ixia PS One and Xeon server testbed, for TCP/IP host stack tests.
  64
  65 ### Atom Testbeds
  66 One 3-node Atom (Netgate based) testbed is built consisting of three SUTs
  67 (Type-5 Netgate device.) NIC cards placement into slots and NIC ports
  68 connectivity is following the testbed specification included in the next
  69 section.
  70
  71 ## Inventory
  72
  73 ### Appliances
  74
  75 ```
  76 1. Ixia PerfectStorm One Appliance
  77     - 1 * PS10GE4NG
  78         - Chassis: PS10GE4NG.
  79         - Motherboard: SuperMicro X11DPG-QT.
  80         - Processors: Quad-Core, Intel Processor.
  81         - HW accelerators: FPGA offload.
  82         - RAM Memory: 64GB.
  83         - Disks: 1 * 1 TB, Enterprise Class, High MTBF.
  84         - Physical Interfaces: 4 * 10GE SFP+.
  85         - Operating System: Native IxOS.
  86 ```
  87
  88 ### Arm Servers
  89
  90 ```
  91 1. Arm Cortex A-72 servers
  92     - 1 * ThunderX2
  93         - Chassis: Marvell ThunderX2
  94         - Processors: 2* ThunderX2 CN9975 ~ 112* ThunderX2.
  95         - RAM Memory: 4* 32GB RDIMM
  96         - Disks: 1* 480GB SSD Micron, 1* 1000GB HDD Seagate_25
  97     - 2 * Huawei TaiShan 2280.
  98         - Chassis: Huawei TaiShan 2280.
  99         - Processors: 1* hip07-d05 ~ 64* Arm Cortex-A72.
 100         - RAM Memory: 8* 16GB DDR4-2400MT/s.
 101         - Disks: 1* 4TB SATA HDD.
 102     - 3 * MACCHIATObin
 103         - Chassis: MACCHIATObin.
 104         - Processors: 1* Armada 8040 ~ 4* Arm Cortex-A72.
 105         - RAM Memory: 1* 16GB DDR4.
 106         - Disks: 1* 128GB(?) SATA SDD.
 107 ```
 108
 109 Platform Name and Specification | Role | Status | Hostname | IP | IPMI | Cores | RAM | Ethernet | Distro
 110 ------------------------------- | ---- | ------ | -------- | -- | ---- | ----- | --- | -------- | ------
 111 [SoftIron OverDrive 1000](https://softiron.com/development-tools/overdrive-1000/) | CI build server | Up, Not Running Jobs | softiron-1 | 10.30.51.12 | N/A | 4 | 8GB |  | openSUSE
 112   | | CI build server | Up, Not Running Jobs | softiron-2 | 10.30.51.13 | N/A | 4 | 8GB |   | openSUSE
 113   | | CI build server | Up, Not Running Jobs | softiron-3 | 10.30.51.14 | N/A | 4 | 8GB |   | openSUSE
 114 [Marvell ThunderX](https://www.marvell.com/server-processors/thunderx-arm-processors/) | CI build server | Up, Running VPP CI | nomad3arm | 10.30.51.38 | 10.30.50.38 | 96 | 128GB | 3x40GbE QSFP+ / 4x10GbE SFP+ | Ubuntu 16.04
 115   | | CI build server | Up, Running VPP CI | nomad4arm | 10.30.51.39 | 10.30.50.39 | 96 | 128GB | 3x40GbE QSFP+ / 4x10GbE SFP+ | Ubuntu 16.04
 116   | | CI build server | Up, Running VPP CI | nomad5arm | 10.30.51.40 | 10.30.50.40 | 96 | 128GB | 3x40GbE QSFP+ / 4x10GbE SFP+ | Ubuntu 16.04
 117   | | CI build server | Up, Not Running Jobs, USB_NIC broken, QSFP wiring to be added | nomad6arm | 10.30.51.65 | 10.30.50.65 | 96 | 256GB | 2xQSFP+ / USB Ethernet | Ubuntu 18.04.1
 118   | | VPP dev debug | Up | nomad7arm | 10.30.51.66 | 10.30.50.66 | 96 | 256GB | 2xQSFP+ / USB Ethernet | Ubuntu 18.04.1
 119   | | CI build server | Up, Not Running Jobs, USB_NIC broken, QSFP wiring to be added | nomad8arm | 10.30.51.67 | 10.30.50.67 | 96 | 256GB | 2xQSFP+ / USB Ethernet | Ubuntu 16.04.1
 120   | | VPP dev debug | Up | nomad9arm | 10.30.51.68 | 10.30.50.68 | 96 | 256GB | 2xQSFP+ / USB Ethernet | Ubuntu 16.04.1
 121 [Marvell ThunderX2](https://www.marvell.com/server-processors/thunderx2-arm-processors/) | VPP device server | Being Provisioned | s27-t13-sut1 | 10.30.51.69 | 10.30.50.69 | 112 | 128GB | 3x40GbE QSFP+ XL710-QDA2 | Ubuntu 18.04.1
 122 Huawei TaiShan 2280 | CSIT Performance | Up, Manual perf experiments | s17-t33-sut1 | 10.30.51.36 | 10.30.50.36 | 64 | 128GB | 2x10GbE SFP+ Intel 82599 / 2x25GbE SFP28 Mellanox CX-4 | Ubuntu 17.10
 123   | | CSIT Performance | Up, Manual perf experiments | s18-t33-sut2 | 10.30.51.37 | 10.30.50.37 | 64 | 128GB | 2x10GbE SFP+ Intel 82599 / 2x25GbE SFP28 Mellanox CX-4 | Ubuntu 17.10
 124 [Marvell MACCHIATObin](http://macchiatobin.net/) | CSIT Performance | Up, Manual experiments, Full Skx TG too much for it - suggest to use LXC/DRC TG(!) | s20-t34-sut1 | 10.30.51.41 | 10.30.51.49, then connect to /dev/ttyUSB0 | 4 | 16GB | 2x10GbE SFP+ | Ubuntu 16.04.4
 125   | | CSIT Performance | Up, Manual experiments, Full Skx TG too much for it - suggest to use LXC/DRC TG(!) | s21-t34-sut2 | 10.30.51.42 | 10.30.51.49, then connect to /dev/ttyUSB1 | 4 | 16GB | 2x10GbE SFP+ | Ubuntu 16.04.5
 126   | | VPP dev debug | Up, Manual VPP Device experiments, Full Skx TG too much for it - suggest to use LXC/DRC TG(!) | fdio-mcbin3 | 10.30.51.43 | 10.30.51.49, then connect to /dev/ttyUSB2 | 4 | 16GB | 2x10GbE SFP+ | Ubuntu 16.04.5
 127
 128 ### Xeon and Atom Servers
 129
 130 ```
 131 1. Intel Xeon servers:
 132     - 20 * SuperMicro SYS-7049GP-TRT with Xeon Skylake processors.
 133         - Chassis: SuperMicro SYS-7049GP-TRT.
 134         - Motherboard: SuperMicro X11DPG-QT.
 135         - Processors: 2* Intel Platinum 8180 2.3 GHz.
 136         - RAM Memory: 16* 16GB DDR4-2666MHz.
 137         - Disks: 2* 1.6TB 6G SATA SSD.
 138 2. Intel Atom servers with Rangely processors.
 139     - 3 * Netgate XG-2758-1u
 140         - Chassis: Netgate XG-2758-1u
 141         - Processors: 1* Rangely (Atom) C2758 2.4 GHz
 142         - RAM Memory: 16GB ECC
 143         - Disks: 150 GB
 144 ```
 145
 146 ### Network Interface Cards
 147
 148 ```
 149 1. 10GE NICs
 150     - 14 * Intel® Ethernet Converged Network Adapter X710-DA4
 151     - 6 * Intel® Ethernet Converged Network Adapter X710-DA2
 152     - 6 * Intel® Ethernet Converged Network Adapter X520-DA2
 153 2. 25GE NICs
 154     - 12 * Intel® Ethernet Network Adapter XXV710-DA2
 155 3. 40GE NICs
 156     - 2 * Intel® Ethernet Converged Network Adapter XL710-QDA2
 157 4. 100GE NICs
 158     - 4 * mcx556a-edat NICs (not on site yet, in transit)
 159 ```
 160
 161 ### Pluggables and Cables
 162
 163 Pluggables:
 164
 165 ```
 166 1. 10GE SFP+
 167     - 16 * Intel E10GSFPSR Ethernet SFP+ SR Optics
 168     - 80 * 10G SR optic (generic, "Intel" compatible branded)
 169 2. 25GE SFP28
 170     - None
 171 3. 40GE QSFP+
 172     - None
 173 4. 100GE
 174     - 8 * mcp1600-c002 qsfp28 pluggables and cables (not on site yet, in transit)
 175 ```
 176
 177 Standalone cables:
 178
 179 ```
 180 1. 10GE
 181     - None
 182 2. 25GE
 183     - None
 184 3. 40GE QSFP+
 185     - 20 * Intel XLDACBL5 40G QSFP+ Passive DAC Cable
 186 4. 100GE
 187     - None
 188 ```
 189
 190 ### Other Network Cards
 191
 192 Any QATs?
 193
 194 ## Installation Status
 195
 196 Lab installation status is tracked by LF IT team in
 197 [FD.io Server Status](https://docs.google.com/document/d/16TdvGC73wuNQjkP355MTXRckv7yqnaxJkWwX7G7izEo/edit?ts=5b10411b#heading=h.dprb64shku8u).
 198
 199 ## Server/Device Management and Naming
 200
 201 ### Server Management Requirements
 202
 203 Total of 20 SM SYS-7049GP-TRT servers are made available for FD.IO CSIT testbed.
 204 For management purposes, each server must have following two ports connected to
 205 the management network:
 206
 207 ```
 208 - 1GE IPMI port
 209   - IPMI - Intelligent Platform Management Interface.
 210   - Required for access to embedded server management with WebUI, CLI, SNMPv3,
 211     IPMIv2.0, for firmware (BIOS) and OS updates.
 212 - 1GE/10GE management port
 213   - hostOS management port for general system management.
 214 ```
 215
 216 ### Server and Port Naming Convention
 217
 218 Following naming convention is used within this page to specify physical
 219 connectivity and wiring across defined CSIT testbeds:
 220
 221 ```
 222 - testbedname: testbedN.
 223 - hostname:
 224     - traffic-generator: tN-tgW.
 225     - system-under-testX: tN-sutX.
 226 - portnames:
 227     - tN-tgW-cY/pZ.
 228     - tN-sutX-cY/pZ.
 229 - where:
 230     - N - testbed number.
 231     - tgW - server acts as traffic-generator with W index.
 232     - sutX - server acts as system-under-test with X index.
 233     - Y - PCIe slot number denoting a NIC card number within the host.
 234         - Y=2,4,9 - slots connected to NUMA node 0.
 235         - Y=6,8,10 - slots connected to NUMA node 1.
 236     - Z - port number on the NIC card.
 237 ```
 238
 239 ### Server Management - Addressing
 240
 241 Each server has a LOM (Lights-Out-Management e.g. SM IPMI) and a Management
 242 port, which are connected to two different VLANs.
 243
 244 ```
 245 1. LOM (IPMI) VLAN:
 246     - Subnet: 10.30.50.0/24
 247     - Gateway: 10.30.50.1
 248     - Broadcast: 10.30.50.255
 249     - DNS1: 199.204.44.24
 250     - DNS2: 199.204.47.54
 251 2. Management Vlan:
 252     - Subnet: 10.30.51.0/24
 253     - Gateway: 10.30.51.1
 254     - Broadcast: 10.30.51.255
 255     - DNS1: 199.204.44.24
 256     - DNS2: 199.204.47.54
 257 ```
 258
 259 To access these hosts, an VPN connection is required.
 260
 261 ### LOM (IPMI) VLAN IP Addresses
 262 ..
 263
 264 ### Management VLAN IP Addresses
 265 ..
 266
 267 ## Testbeds Specification - Target Build
 268
 269 ### Server/Ports Naming, NIC Placement
 270
 271 #### 1-Node Xeon
 272
 273 Each server in 1-node Xeon topology has its NIC cards placed, and NIC cards and
 274 ports indexed per following specification:
 275
 276 ```
 277 - Server1 [Type-6]:
 278     - testbedname: testbed11.
 279     - hostname: s1-t11-sut1.
 280     - IPMI IP: 10.30.50.47
 281     - Host IP: 10.30.51.50
 282     - portnames:
 283         - s1-t11-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
 284         - s1-t11-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
 285         - s1-t11-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
 286         - s1-t11-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
 287         - s1-t11-sut1-c4/p1 - 10GE-port1 x710-4p10GE.
 288         - s1-t11-sut1-c4/p2 - 10GE-port2 x710-4p10GE.
 289         - s1-t11-sut1-c4/p3 - 10GE-port3 x710-4p10GE.
 290         - s1-t11-sut1-c4/p4 - 10GE-port4 x710-4p10GE.
 291 - Server2 [Type-6]:
 292     - testbedname: testbed12.
 293     - hostname: s2-t12-sut1.
 294     - IPMI IP: 10.30.50.48
 295     - Host IP: 10.30.51.51
 296     - portnames:
 297         - s2-t12-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
 298         - s2-t12-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
 299         - s2-t12-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
 300         - s2-t12-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
 301         - s2-t12-sut1-c4/p1 - 10GE-port1 x710-4p10GE.
 302         - s2-t12-sut1-c4/p2 - 10GE-port2 x710-4p10GE.
 303         - s2-t12-sut1-c4/p3 - 10GE-port3 x710-4p10GE.
 304         - s2-t12-sut1-c4/p4 - 10GE-port4 x710-4p10GE.
 305 ```
 306
 307 #### 2-Node Xeon
 308
 309 Each server in 2-node Xeon topology has its NIC cards placed, and NIC cards and
 310 ports indexed per following specification:
 311
 312 ```
 313 - Server3 [Type-1]:
 314     - testbedname: testbed21.
 315     - hostname: s3-t21-sut1.
 316     - IPMI IP: 10.30.50.41
 317     - Host IP: 10.30.51.44
 318     - portnames:
 319         - s3-t21-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
 320         - s3-t21-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
 321         - s3-t21-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
 322         - s3-t21-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
 323         - s3-t21-sut1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 324         - s3-t21-sut1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 325         - s3-t21-sut1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 326         - s3-t21-sut1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 327 - Server4 [Type-2]:
 328     - testbedname: testbed21.
 329     - hostname: s4-t21-tg1.
 330     - IPMI IP: 10.30.50.42
 331     - Host IP: 10.30.51.45
 332     - portnames:
 333         - s4-t21-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
 334         - s4-t21-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
 335         - s4-t21-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
 336         - s4-t21-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
 337         - s4-t21-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 338         - s4-t21-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 339         - s4-t21-tg1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 340         - s4-t21-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 341 - Server5 [Type-1]:
 342     - testbedname: testbed22.
 343     - hostname: s5-t22-sut1.
 344     - IPMI IP: 10.30.50.49
 345     - Host IP: 10.30.51.52
 346     - portnames:
 347         - s5-t22-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
 348         - s5-t22-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
 349         - s5-t22-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
 350         - s5-t22-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
 351         - s5-t22-sut1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 352         - s5-t22-sut1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 353         - s5-t22-sut1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 354         - s5-t22-sut1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 355 - Server6 [Type-2]:
 356     - testbedname: testbed22.
 357     - hostname: s6-t22-tg1.
 358     - IPMI IP: 10.30.50.50
 359     - Host IP: 10.30.51.53
 360     - portnames:
 361         - s6-t22-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
 362         - s6-t22-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
 363         - s6-t22-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
 364         - s6-t22-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
 365         - s6-t22-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 366         - s6-t22-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 367         - s6-t22-tg1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 368         - s6-t22-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 369 - Server7 [Type-1]:
 370     - testbedname: testbed23.
 371     - hostname: s7-t23-sut1.
 372     - IPMI IP: 10.30.50.51
 373     - Host IP: 10.30.51.54
 374     - portnames:
 375         - s7-t23-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
 376         - s7-t23-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
 377         - s7-t23-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
 378         - s7-t23-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
 379         - s7-t23-sut1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 380         - s7-t23-sut1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 381         - s7-t23-sut1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 382         - s7-t23-sut1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 383 - Server8 [Type-2]:
 384     - testbedname: testbed23.
 385     - hostname: s8-t23-tg1.
 386     - IPMI IP: 10.30.50.52
 387     - Host IP: 10.30.51.55
 388     - portnames:
 389         - s8-t23-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
 390         - s8-t23-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
 391         - s8-t23-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
 392         - s8-t23-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
 393         - s8-t23-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 394         - s8-t23-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 395         - s8-t23-tg1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 396         - s8-t23-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 397 - Server9 [Type-1]:
 398     - testbedname: testbed24.
 399     - hostname: s9-t24-sut1.
 400     - IPMI IP: 10.30.50.53
 401     - Host IP: 10.30.51.56
 402     - portnames:
 403         - s9-t24-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
 404         - s9-t24-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
 405         - s9-t24-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
 406         - s9-t24-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
 407         - s9-t24-sut1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 408         - s9-t24-sut1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 409         - s9-t24-sut1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 410         - s9-t24-sut1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 411 - Server10 [Type-2]:
 412     - testbedname: testbed24.
 413     - hostname: s10-t24-tg1.
 414     - IPMI IP: 10.30.50.54
 415     - Host IP: 10.30.51.57
 416     - portnames:
 417         - s10-t24-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
 418         - s10-t24-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
 419         - s10-t24-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
 420         - s10-t24-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
 421         - s10-t24-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 422         - s10-t24-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 423         - s10-t24-tg1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 424         - s10-t24-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 425 ```
 426
 427 #### 3-Node Xeon
 428
 429 Each server in 3-node Xeon topology has its NIC cards placed, and NIC cards and
 430 ports indexed per following specification:
 431
 432 ```
 433 - Server11 [Type-1]:
 434     - testbedname: testbed31.
 435     - hostname: s11-t31-sut1.
 436     - IPMI IP: 10.30.50.43
 437     - Host IP: 10.30.51.46
 438     - portnames:
 439         - s11-t31-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
 440         - s11-t31-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
 441         - s11-t31-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
 442         - s11-t31-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
 443         - s11-t31-sut1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 444         - s11-t31-sut1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 445         - s11-t31-sut1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 446         - s11-t31-sut1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 447 - Server12 [Type-1]:
 448     - testbedname: testbed31.
 449     - hostname: s12-t31-sut2.
 450     - IPMI IP: 10.30.50.44
 451     - Host IP: 10.30.51.47
 452     - portnames:
 453         - s12-t31-sut2-c2/p1 - 10GE-port1 x710-4p10GE.
 454         - s12-t31-sut2-c2/p2 - 10GE-port2 x710-4p10GE.
 455         - s12-t31-sut2-c2/p3 - 10GE-port3 x710-4p10GE.
 456         - s12-t31-sut2-c2/p4 - 10GE-port4 x710-4p10GE.
 457         - s12-t31-sut2-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 458         - s12-t31-sut2-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 459         - s12-t31-sut2-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 460         - s12-t31-sut2-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 461 - Server13 [Type-2]:
 462     - testbedname: testbed31.
 463     - hostname: s13-t31-tg1.
 464     - IPMI IP: 10.30.50.45
 465     - Host IP: 10.30.51.48
 466     - portnames:
 467         - s13-t31-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
 468         - s13-t31-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
 469         - s13-t31-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
 470         - s13-t31-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
 471         - s13-t31-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 472         - s13-t31-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 473         - s13-t31-tg1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 474         - s13-t31-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 475 - Server14 [Type-1]:
 476     - testbedname: testbed32.
 477     - hostname: s14-t32-sut1.
 478     - IPMI IP: 10.30.50.55
 479     - Host IP: 10.30.51.58
 480     - portnames:
 481         - s14-t32-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
 482         - s14-t32-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
 483         - s14-t32-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
 484         - s14-t32-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
 485         - s14-t32-sut1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 486         - s14-t32-sut1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 487         - s14-t32-sut1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 488         - s14-t32-sut1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 489 - Server15 [Type-1]:
 490     - testbedname: testbed32.
 491     - hostname: s15-t32-sut2.
 492     - IPMI IP: 10.30.50.56
 493     - Host IP: 10.30.51.59
 494     - portnames:
 495         - s15-t32-sut2-c2/p1 - 10GE-port1 x710-4p10GE.
 496         - s15-t32-sut2-c2/p2 - 10GE-port2 x710-4p10GE.
 497         - s15-t32-sut2-c2/p3 - 10GE-port3 x710-4p10GE.
 498         - s15-t32-sut2-c2/p4 - 10GE-port4 x710-4p10GE.
 499         - s15-t32-sut2-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 500         - s15-t32-sut2-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 501         - s15-t32-sut2-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 502         - s15-t32-sut2-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 503 - Server16 [Type-2]:
 504     - testbedname: testbed32.
 505     - hostname: s16-t32-tg1.
 506     - IPMI IP: 10.30.50.57
 507     - Host IP: 10.30.51.60
 508     - portnames:
 509         - s16-t32-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
 510         - s16-t32-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
 511         - s16-t32-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
 512         - s16-t32-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
 513         - s16-t32-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 514         - s16-t32-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 515         - s16-t32-tg1-c9/p1 - FUTURE 100GE-port1 ConnectX5-2p100GE.
 516         - s16-t32-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE.
 517 ```
 518
 519 #### 1-Node Arm
 520
 521 ```
 522 - Server17 [Type-3]:
 523     - testbedname: testbed13.
 524     - hostname: s27-t13-sut1.
 525     - IPMI IP: 10.30.50.69
 526     - Host IP: 10.30.51.69
 527     - portnames:
 528         - s27-t13-sut1-c1/p1 - 40GE-port1 XL710-QDA2-2p40GE.
 529         - s27-t13-sut1-c1/p2 - 40GE-port2 XL710-QDA2-2p40GE.
 530         - s27-t13-sut1-c3/p1 - 40GE-port1 XL710-QDA2-2p40GE.
 531         - s27-t13-sut1-c3/p2 - 40GE-port2 XL710-QDA2-2p40GE.
 532         - s27-t13-sut1-c6/p1 - 40GE-port1 XL710-QDA2-2p40GE.
 533         - s27-t13-sut1-c6/p2 - 40GE-port2 XL710-QDA2-2p40GE.
 534 ```
 535
 536 #### 3-Node Arm
 537
 538 Note: Server19 (TG) is shared between testbed33 & testbed34
 539
 540 ```
 541 - Server17 [Type-3]:
 542     - testbedname: testbed33.
 543     - hostname: s17-t33-sut1.
 544     - IPMI IP: 10.30.50.36
 545     - Host IP: 10.30.51.36
 546     - portnames:
 547         - s17-t33-sut1-c6/p1 - 10GE-port1 82599-2p10GE.
 548         - s17-t33-sut1-c6/p2 - 10GE-port2 82599-2p10GE.
 549         - s17-t33-sut1-c4/p1 - 25GE-port1 cx4-2p25GE.
 550         - s17-t33-sut1-c4/p2 - 25GE-port2 cx4-2p25GE.
 551 - Server18 [Type-3]:
 552     - testbedname: testbed33.
 553     - hostname: s18-t33-sut2.
 554     - IPMI IP: 10.30.50.37
 555     - Host IP: 10.30.51.37
 556     - portnames:
 557         - s18-t33-sut2-c6/p1 - 10GE-port1 82599-2p10GE.
 558         - s18-t33-sut2-c6/p2 - 10GE-port2 82599-2p10GE.
 559         - s18-t33-sut2-c4/p1 - 25GE-port1 cx4-2p25GE.
 560         - s18-t33-sut2-c4/p2 - 25GE-port2 cx4-2p25GE.
 561 - Server19 [Type-2]:
 562     - testbednames: testbed33 and testbed34.
 563     - hostname: s19-t33t34-tg1.
 564     - IPMI IP: 10.30.50.46
 565     - Host IP: 10.30.51.49
 566     - portnames:
 567         - s19-t33t34-tg1-c2/p1 - 10GE-port1 x710-4p10GE.
 568         - s19-t33t34-tg1-c2/p2 - 10GE-port2 x710-4p10GE.
 569         - s19-t33t34-tg1-c2/p3 - 10GE-port3 x710-4p10GE.
 570         - s19-t33t34-tg1-c2/p4 - 10GE-port4 x710-4p10GE.
 571         - s19-t33t34-tg1-c4/p1 - 25GE-port1 xxv710-DA2-2p25GE.
 572         - s19-t33t34-tg1-c4/p2 - 25GE-port2 xxv710-DA2-2p25GE.
 573         - s19-t33t34-tg1-c10/p1 - 10GE-port1 x710-4p10GE.
 574         - s19-t33t34-tg1-c10/p2 - 10GE-port2 x710-4p10GE.
 575         - s19-t33t34-tg1-c10/p3 - 10GE-port3 x710-4p10GE.
 576         - s19-t33t34-tg1-c10/p4 - 10GE-port4 x710-4p10GE.
 577 - Server20 [Type-4]:
 578     - testbedname: testbed34.
 579     - hostname: s20-t34-sut1.
 580     - IPMI IP: N/A
 581     - Host IP: 10.30.51.41
 582     - portnames:
 583         - s20-t34-sut1-ca/p1 - 10GE-port1 Marvell.
 584         - s20-t34-sut1-ca/p2 - 10GE-port2 Marvell.
 585 - Server21 [Type-4]:
 586     - testbedname: testbed34.
 587     - hostname: s21-t34-sut2.
 588     - IPMI IP: N/A
 589     - Host IP: 10.30.51.42
 590     - portnames:
 591         - s21-t34-sut2-ca/p1 - 10GE-port1 Marvell.
 592         - s21-t34-sut2-ca/p2 - 10GE-port2 Marvell.
 593 ```
 594
 595 #### TCP/IP and L47
 596
 597 Each server (appliance) in 2-node TCP/IP topology has its NIC cards placed, and
 598 NIC cards and ports indexed per following specification:
 599
 600 ```
 601 - Server25 [Type-8]:
 602     - testbedname: testbed25.
 603     - hostname: s25-t25-sut1.
 604     - IPMI IP: 10.30.50.58
 605     - Host IP: 10.30.51.61
 606     - portnames:
 607         - s25-t25-sut1-c2/p1 - 10GE-port1 x710-4p10GE.
 608         - s25-t25-sut1-c2/p2 - 10GE-port2 x710-4p10GE.
 609         - s25-t25-sut1-c2/p3 - 10GE-port3 x710-4p10GE.
 610         - s25-t25-sut1-c2/p4 - 10GE-port4 x710-4p10GE.
 611 - Server26 [Type-7]:
 612     - testbedname: testbed25.
 613     - hostname: s26-t25-tg1.
 614     - IPMI IP: 10.30.50.59
 615     - Host IP: 10.30.51.62
 616     - portnames:
 617         - s26-t25-tg1-p1 - 10GE-port1.
 618         - s26-t25-tg1-p2 - 10GE-port2.
 619         - s26-t25-tg1-p3 - 10GE-port3.
 620         - s26-t25-tg1-p4 - 10GE-port4.
 621 ```
 622
 623 #### 3-Node Atom
 624
 625 Note: There is no IPMI. Serial console is accessible via VIRL2 and VIRL3 USB.
 626
 627 ```
 628 - Server22 [Type-5]:
 629     - testbedname: testbed35.
 630     - hostname: s22-t35-sut1 (vex-yul-rot-netgate-1).
 631     - IPMI IP: 10.30.51.29 - screen -r /dev/ttyUSB0
 632     - Host IP: 10.30.51.9
 633     - portnames:
 634         - s22-t35-sut1-p1 - 10GE-port1 ix0 82599.
 635         - s22-t35-sut1-p2 - 10GE-port2 ix1 82599.
 636     - 1GB ports (tbd)
 637 - Server23 [Type-5]:
 638     - testbedname: testbed35.
 639     - hostname: s23-t35-sut2 (vex-yul-rot-netgate-2).
 640     - IPMI IP: 10.30.51.30 - screen -r /dev/ttyUSB1
 641     - Host IP: 10.30.51.10
 642     - portnames:
 643         - s23-t35-sut1-p1 - 10GE-port1 ix0 82599.
 644         - s23-t35-sut1-p2 - 10GE-port2 ix1 82599.
 645     - 1GB ports (tbd)
 646 - Server24 [Type-5]:
 647     - testbedname: testbed35.
 648     - hostname: s24-t35-sut3 (vex-yul-rot-netgate-3).
 649     - IPMI IP: 10.30.51.30 - screen -r /dev/ttyUSB2
 650     - Host IP: 10.30.51.11
 651     - portnames:
 652         - s24-t35-sut1-p1 - 10GE-port1 ix0 82599.
 653         - s24-t35-sut1-p2 - 10GE-port2 ix1 82599.
 654     - 1GB ports (tbd)
 655 ```
 656
 657 ### Physical Connectivity within Testbeds
 658
 659 #### 1-Node Xeon
 660
 661 Two 1-Node testbeds are constructed by connecting 2 Xeon servers using below
 662 specification:
 663
 664 ```
 665 - testbed11:
 666     - ring1 10GE-ports x710-4p10GE:
 667         - s1-t11-sut1-c2/p1 to s1-t11-sut1-c4/p1.
 668     - ring2 10GE-ports x710-4p10GE:
 669         - s1-t11-sut1-c2/p2 to s1-t11-sut1-c4/p2.
 670     - ring3 10GE-ports x710-4p10GE:
 671         - s1-t11-sut1-c2/p3 to s1-t11-sut1-c4/p3.
 672     - ring4 10GE-ports x710-4p10GE:
 673         - s1-t11-sut1-c2/p3 to s1-t11-sut1-c4/p3.
 674 - testbed12:
 675     - ring1 10GE-ports x710-4p10GE:
 676         - s2-t12-sut1-c2/p1 to s2-t12-sut1-c4/p1.
 677     - ring2 10GE-ports x710-4p10GE:
 678         - s2-t12-sut1-c2/p2 to s2-t12-sut1-c4/p2.
 679     - ring3 10GE-ports x710-4p10GE:
 680         - s2-t12-sut1-c2/p3 to s2-t12-sut1-c4/p3.
 681     - ring4 10GE-ports x710-4p10GE:
 682         - s2-t12-sut1-c2/p3 to s2-t12-sut1-c4/p3.
 683 ```
 684
 685 #### 2-Node Xeon
 686
 687 Four 2-Node testbeds are constructed by connecting 8 Xeon servers using below
 688 specification:
 689
 690 ```
 691 - testbed21:
 692     - ring1 10GE-ports x710-4p10GE on SUT:
 693         - s4-t21-tg1-c2/p1 to s3-t21-sut1-c2/p1.
 694         - s3-t21-sut1-c2/p2 to s4-t21-tg1-c2/p2.
 695     - ring2 10GE-ports x710-4p10GE on SUT:
 696         - s4-t21-tg1-c2/p3 to s3-t21-sut1-c2/p3.
 697         - s3-t21-sut1-c2/p4 to s4-t21-tg1-c2/p4.
 698     - ring3 25GE-ports xxv710-DA2-2p25GE on SUT
 699         - s4-t21-tg1-c4/p1 to s3-t21-sut1-c4/p1.
 700         - s3-t21-sut1-c4/p2 to s4-t21-tg1-c4/p2.
 701     - FUTURE ring4 100GE-ports ConnectX5-2p100GE on SUT:
 702         - s4-t21-tg1-c9/p1 to s3-t21-sut1-c9/p1.
 703         - s3-t21-sut1-c9/p2 to s4-t21-tg1-c9/p2.
 704     - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
 705         - s4-t21-tg1-c10/p1 to s4-t21-tg1-c10/p2.
 706         - s4-t21-tg1-c10/p3 to s4-t21-tg1-c10/p4.
 707 - testbed22:
 708     - ring1 10GE-ports x710-4p10GE on SUT:
 709         - s6-t22-tg1-c2/p1 to s5-t22-sut1-c2/p1.
 710         - s5-t22-sut1-c2/p2 to s6-t22-tg1-c2/p2.
 711     - ring2 10GE-ports x710-4p10GE on SUT:
 712         - s6-t22-tg1-c2/p3 to s5-t22-sut1-c2/p3.
 713         - s5-t22-sut1-c2/p4 to s6-t22-tg1-c2/p4.
 714     - ring3 25GE-ports xxv710-DA2-2p25GE on SUT
 715         - s6-t22-tg1-c4/p1 to s5-t22-sut1-c4/p1.
 716         - s5-t22-sut1-c4/p2 to s6-t22-tg1-c4/p2.
 717     - FUTURE ring4 100GE-ports ConnectX5-2p100GE on SUT:
 718         - t22-tg1-c9/p1 to s5-t22-sut1-c9/p1.
 719         - s5-t22-sut1-c9/p2 to s6-t22-tg1-c9/p2.
 720     - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
 721         - s6-t22-tg1-c10/p1 to s6-t22-tg1-c10/p2.
 722         - s6-t22-tg1-c10/p3 to s6-t22-tg1-c10/p4.
 723 - testbed23:
 724     - ring1 10GE-ports x710-4p10GE on SUT:
 725         - s8-t23-tg1-c2/p1 to s7-t23-sut1-c2/p1.
 726         - s7-t23-sut1-c2/p2 to s8-t23-tg1-c2/p2.
 727     - ring2 10GE-ports x710-4p10GE on SUT:
 728         - s8-t23-tg1-c2/p3 to s7-t23-sut1-c2/p3.
 729         - s7-t23-sut1-c2/p4 to s8-t23-tg1-c2/p4.
 730     - ring3 25GE-ports xxv710-DA2-2p25GE on SUT
 731         - s8-t23-tg1-c4/p1 to s7-t23-sut1-c4/p1.
 732         - s7-t23-sut1-c4/p2 to s8-t23-tg1-c4/p2.
 733     - FUTURE ring4 100GE-ports ConnectX5-2p100GE on SUT:
 734         - s8-t23-tg1-c9/p1 to s7-t23-sut1-c9/p1.
 735         - s7-t23-sut1-c9/p2 to s8-t23-tg1-c9/p2.
 736     - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
 737         - s8-t23-tg1-c10/p1 to s8-t23-tg1-c10/p2.
 738         - s8-t23-tg1-c10/p3 to s8-t23-tg1-c10/p4.
 739 - testbed24:
 740     - ring1 10GE-ports x710-4p10GE on SUT:
 741         - s10-t24-tg1-c2/p1 to s9-t24-sut1-c2/p1.
 742         - s9-t24-sut1-c2/p2 to s10-t24-tg1-c2/p2.
 743     - ring2 10GE-ports x710-4p10GE on SUT:
 744         - s10-t24-tg1-c2/p3 to s9-t24-sut1-c2/p3.
 745         - s9-t24-sut1-c2/p4 to s10-t24-tg1-c2/p4.
 746     - ring3 25GE-ports xxv710-DA2-2p25GE on SUT
 747         - s10-t24-tg1-c4/p1 to s9-t24-sut1-c4/p1.
 748         - s9-t24-sut1-c4/p2 to s10-t24-tg1-c4/p2.
 749     - FUTURE ring4 100GE-ports ConnectX5-2p100GE on SUT:
 750         - s10-t24-tg1-c9/p1 to s9-t24-sut1-c9/p1.
 751         - s9-t24-sut1-c9/p2 to s10-t24-tg1-c9/p2.
 752     - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
 753         - s10-t24-tg1-c10/p1 to s10-t24-tg1-c10/p2.
 754         - s10-t24-tg1-c10/p3 to s10-t24-tg1-c10/p4.
 755 ```
 756
 757 #### 3-Node Xeon
 758
 759 Two 3-Node testbeds are constructed by connecting 6 Xeon servers using below
 760 specification:
 761
 762 ```
 763 - testbed31:
 764     - ring1 10GE-ports x710-4p10GE on SUTs:
 765         - s13-t31-tg1-c2/p1 to s11-t31-sut1-c2/p1.
 766         - s11-t31-sut1-c2/p2 to s12-t31-sut2-c2/p2.
 767         - s12-t31-sut2-c2/p1 to s13-t31-tg1-c2/p2.
 768     - ring2 10GE-ports x710-4p10GE on SUT:
 769         - s13-t31-tg1-c2/p3 to s11-t31-sut1-c2/p3.
 770         - s11-t31-sut1-c2/p4 to s12-t31-sut2-c2/p4.
 771         - s12-t31-sut2-c2/p3 to s13-t31-tg1-c2/p4.
 772     - ring3 25GE-ports xxv710-DA2-2p25GE on SUT
 773         - s13-t31-tg1-c4/p1 to s11-t31-sut1-c4/p1.
 774         - s11-t31-sut1-c4/p2 to s12-t31-sut2-c4/p2.
 775         - s12-t31-sut2-c4/p1 to s13-t31-tg1-c4/p2.
 776     - FUTURE ring4 100GE-ports ConnectX5-2p100GE on SUT:
 777         - s13-t31-tg1-c9/p1 to s11-t31-sut1-c9/p1.
 778         - s11-t31-sut1-c9/p2 to s12-t31-sut2-c9/p2.
 779         - s12-t31-sut2-c9/p1 to s13-t31-tg1-c9/p2.
 780     - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
 781         - s13-t31-tg1-c10/p1 to s13-t31-tg1-c10/p2.
 782         - s13-t31-tg1-c10/p3 to s13-t31-tg1-c10/p4.
 783 - testbed32:
 784     - ring1 10GE-ports x710-4p10GE on SUTs:
 785         - s16-t32-tg1-c2/p1 to s14-t32-sut1-c2/p1.
 786         - s14-t32-sut1-c2/p2 to s15-t32-sut2-c2/p2.
 787         - s15-t32-sut2-c2/p1 to s16-t32-tg1-c2/p2.
 788     - ring2 10GE-ports x710-4p10GE on SUT:
 789         - s16-t32-tg1-c2/p3 to s14-t32-sut1-c2/p3.
 790         - s14-t32-sut1-c2/p4 to s15-t32-sut2-c2/p4.
 791         - s15-t32-sut2-c2/p3 to s16-t32-tg1-c2/p4.
 792     - ring3 25GE-ports xxv710-DA2-2p25GE on SUT
 793         - s16-t32-tg1-c4/p1 to s14-t32-sut1-c4/p1.
 794         - s14-t32-sut1-c4/p2 to s15-t32-sut2-c4/p2.
 795         - s15-t32-sut2-c4/p1 to s16-t32-tg1-c4/p2.
 796     - FUTURE ring4 100GE-ports ConnectX5-2p100GE on SUT:
 797         - s16-t32-tg1-c9/p1 to s14-t32-sut1-c9/p1.
 798         - s14-t32-sut1-c9/p2 to s15-t32-sut2-c9/p2.
 799         - s15-t32-sut2-c9/p1 to s16-t32-tg1-c9/p2.
 800     - ring5 10GE-ports x710-4p10GE loopbacks on TG for self-tests:
 801         - s16-t32-tg1-c10/p1 to s16-t32-tg1-c10/p2.
 802         - s16-t32-tg1-c10/p3 to s16-t32-tg1-c10/p4.
 803 ```
 804
 805
 806 #### 1-Node Arm
 807
 808 One 1-Node testbed is constructed by connecting 1 Arm server using below
 809 specification:
 810
 811 ```
 812 - testbed13:
 813     - ring1 40GE-ports XL710-QDA2-2p40GE on SUTs:
 814         - s27-t13-sut1-c1/p2 - s27-t13-sut1-c3/p1.
 815     - ring2 40GE-ports XL710-QDA2-2p40GE on SUTs:
 816         - s27-t13-sut1-c3/p2 - s27-t13-sut1-c6/p1.
 817     - ring3 40GE-ports XL710-QDA2-2p40GE on SUTs:
 818         - s27-t13-sut1-c6/p2 - s27-t13-sut1-c1/p1.
 819 ```
 820
 821 #### 3-Node Arm
 822
 823 One 3-Node testbed is constructed by connecting 2 TaiShan servers and one Xeon
 824 server using below specification:
 825
 826 ```
 827 - testbed33:
 828     - ring1 10GE-ports 82599-2p10GE on SUTs:
 829         - t33t34-tg1-c2/p2 - t33-sut1-c6/p2.
 830         - t33-sut1-c6/p1 - t33-sut2-c6/p2.
 831         - t33-sut2-c6/p1 - t33t34-tg1-c2/p1.
 832     - ring2 25GE-ports cx4-2p25GE on SUTs:
 833         - t33t34-tg1-c4/p2 - t33-sut1-c4/p2.
 834         - t33-sut1-c4/p1 - t33-sut2-c4/p2.
 835         - t33-sut2-c4/p1 - t33t34-tg1-c4/p1.
 836 ```
 837
 838 One 3-Node testbed is constructed by connecting 2 MACCHIATObin servers and one
 839 Xeon server using below specification:
 840
 841 ```
 842 - testbed34:
 843     - ring1 10GE-ports Marvell on SUTs:
 844         - t33t34-tg1-c2/p3 - t34-sut1-ca/p1.
 845         - t34-sut1-ca/p2 - t34-sut2-ca/p1.
 846         - t34-sut2-ca/p2 - t33t34-tg1-c2/p4.
 847 ```
 848
 849 #### TCP/IP and L47
 850
 851 One 2-Node TCP/IP testbed is constructed by connecting Ixia PSOne and 1 Xeon
 852 server using below specification:
 853
 854 ```
 855 - testbed25:
 856     - link1 10GE-port x710-4p10GE on SUT:
 857         - t25-tg1-p1 to t25-sut1-c2/p1.
 858     - link2 10GE-port x710-4p10GE on SUT:
 859         - t25-tg1-p2 to t25-sut1-c2/p2.
 860     - link3 10GE-port x710-4p10GE on SUT:
 861         - t25-tg1-p3 to t25-sut1-c2/p3.
 862     - link4 10GE-port x710-4p10GE on SUT:
 863         - t25-tg1-p4 to t25-sut1-c2/p4.
 864 ```
 865
 866 #### 3-Node Atom
 867
 868 ..
 869
 870 ## Server Specification
 871
 872 ### Hardware Configuration
 873
 874 The new FD.io CSIT-CPL lab (is expected to contain|contains) following hardware
 875 server configurations:
 876
 877 ```
 878 1. Type-1: Purpose - (Intel Xeon Processor) SUT for SW Data Plane Workload i.e.
 879    VPP, testpmd.
 880     - Quantity: TBD based on testbed allocation.
 881     - Physical connectivity:
 882         - IPMI and host management ports.
 883         - NIC ports connected into 2-node and 3-node topologies.
 884     - Main HW configuration:
 885         - Chassis: SuperMicro SYS-7049GP-TRT.
 886         - Motherboard: SuperMicro X11DPG-QT.
 887         - Processors: 2* Intel Platinum 8180 2.3 GHz.
 888         - RAM Memory: 16* 16GB DDR4-2666MHz.
 889         - Disks: 2* 1.6TB 6G SATA SSD.
 890     - NICs configuration:
 891         - Numa0: (x16, x16, x16 PCIe3.0 lanes)
 892             - PCIe Slot2 18:00.xx: x710-4p10GE Intel.
 893             - PCIe Slot4 3b:00.xx: xxv710-DA2-2p25GE Intel.
 894             - PCIe Slot9 5e:00.xx: FUTURE ConnectX5-2p100GE Mellanox.
 895         - Numa1: (x16, x16, x16 PCIe3.0 lanes)
 896             - PCIe Slot6 86:00.xx: empty.
 897             - PCIe Slot8 af:00.xx: empty.
 898             - PCIe Slot10 d8:00.xx: empty.
 899 2. Type-2: Purpose - (Intel Xeon Processor) TG for T-Rex.
 900     - Quantity: TBD based on testbed allocation.
 901     - Physical connectivity:
 902         - IPMI and host management ports.
 903         - NIC ports connected into 2-node and 3-node topologies.
 904     - Main HW configuration:
 905         - Chassis: SuperMicro SYS-7049GP-TRT.
 906         - Motherboard: SuperMicro X11DPG-QT.
 907         - Processors: 2* Intel Platinum 8180 2.3 GHz.
 908         - RAM Memory: 16* 16GB DDR4-2666MHz.
 909         - Disks: 2* 1.6TB 6G SATA SSD.
 910     - NICs configuration:
 911         - Numa0: (x16, x16, x16 PCIe3.0 lanes)
 912             - PCIe Slot2 18:00.xx: x710-4p10GE Intel.
 913             - PCIe Slot4 3b:00.xx: xxv710-DA2 2p25GE Intel.
 914             - PCIe Slot9 5e:00.xx: FUTURE ConnectX5-2p100GE Mellanox.
 915         - Numa1: (x16, x16, x16 PCIe3.0 lanes)
 916             - PCIe Slot6 86:00.xx: empty.
 917             - PCIe Slot8 af:00.xx: empty.
 918             - PCIe Slot10 d8:00.xx: x710-4p10GE Intel.
 919 3. Type-3: Purpose - (Arm hip07-d05 Processor) SUT for SW Data Plane Workload
 920    i.e. VPP, testpmd.
 921     - Quantity: 2
 922     - Physical connectivity:
 923         - IPMI(?) and host management ports.
 924         - NIC ports connected into 3-node topology.
 925     - Main HW configuration:
 926         - Chassis: Huawei TaiShan 2280.
 927         - Processors: 1* hip07-d05 ~ 64* Arm Cortex-A72
 928         - RAM Memory: 8* 16GB DDR4-2400MT/s
 929         - Disks: 1* 4TB SATA HDD
 930     - NICs configuration:
 931         - PCIe Slot4 e9:00.xx: connectx4-2p25GE Mellanox.
 932         - PCIe Slot6 11:00.xx: 82599-2p10GE Intel.
 933 4. Type-4: Purpose - (Arm Armada 8040 Processor) SUT for SW Data Plane Workload
 934    i.e. VPP, testpmd.
 935     - Quantity: 3
 936     - Physical connectivity:
 937         - Host management ports.
 938         - NIC ports connected into 2-node and 3-node topologies.
 939     - Main HW configuration:
 940         - Chassis: MACCHIATObin.
 941         - Processors: 1* Armada 8040 ~ 4* Arm Cortex-A72
 942         - RAM Memory: 1* 16GB DDR4
 943         - Disks: 1* 128GB(?) SATA SDD
 944     - NICs configuration:
 945         - pp2-2p10GE Marvell (on-chip Ethernet ports ; marvell plugin in VPP)
 946 5. Type-5: Purpose - (Intel Atom Processor) SUT for SW Data Plane Workload i.e.
 947    VPP, testpmd.
 948     - Quantity: TBD based on testbed allocation.
 949     - Physical connectivity:
 950         - Management: serial Port (usb) for console
 951         - NIC ports connected into 2-node.
 952     - Main HW configuration:
 953         - Chassis: Netgate XG-2758-1u
 954         - Processors: 1* Rangeley (Atom) C2758 2.4 GHz
 955         - RAM Memory: 16GB ECC
 956         - Disks: 150 GB
 957     - NICs configuration:
 958         - 2x 10Gb Intel 82599ES
 959         - 4x 1GB Intel I354
 960 6. Type-6: Purpose - (Intel Xeon Processor) SUT for VPP_Device functional tests.
 961     - Quantity: 1.
 962     - Physical connectivity:
 963         - IPMI and host management ports.
 964         - NIC ports connected into 2-node and 3-node topologies.
 965     - Main HW configuration:
 966         - Chassis: SuperMicro SYS-7049GP-TRT.
 967         - Motherboard: SuperMicro X11DPG-QT.
 968         - Processors: 2* Intel Platinum 8180 2.3 GHz.
 969         - RAM Memory: 16* 16GB DDR4-2666MHz.
 970         - Disks: 2* 1.6TB 6G SATA SSD.
 971     - NICs configuration:
 972         - Numa0: (x16, x16, x16 PCIe3.0 lanes)
 973             - PCIe Slot2 18:00.xx: x710-4p10GE Intel.
 974             - PCIe Slot4 3b:00.xx: x710-4p10GE Intel.
 975             - PCIe Slot9 5e:00.xx: empty.
 976         - Numa1: (x16, x16, x16 PCIe3.0 lanes)
 977             - PCIe Slot6 86:00.xx: empty.
 978             - PCIe Slot8 af:00.xx: empty.
 979             - PCIe Slot10 d8:00.xx: empty.
 980 7. Type-7: Purpose - Ixia PerfectStorm One Appliance TG for TCP/IP performance
 981    tests.
 982     - Quantity: 1.
 983     - Physical connectivity:
 984         - Host management interface: 10/100/1000-BaseT.
 985         - 8-port 10GE SFP+ integrated NIC.
 986     - Main HW configuration:
 987         - Chassis: PS10GE4NG.
 988         - Motherboard: SuperMicro X11DPG-QT.
 989         - Processors: Quad-Core, Intel Processor.
 990         - HW accelerators: FPGA offload.
 991         - RAM Memory: 64GB.
 992         - Disks: 1 * 1 TB, Enterprise Class, High MTBF.
 993         - Physical Interfaces: 4 * 10GE SFP+.
 994         - Operating System: Native IxOS.
 995     - Interface configuration:
 996         - Port-1: 10GE SFP+.
 997         - Port-2: 10GE SFP+.
 998         - Port-3: 10GE SFP+.
 999         - Port-4: 10GE SFP+.
1000 8. Type-8: Purpose - (Intel Xeon Processor) SUT for TCP/IP host stack tests.
1001     - Quantity: 1.
1002     - Physical connectivity:
1003         - IPMI and host management ports.
1004         - NIC ports.
1005     - Main HW configuration:
1006         - Chassis: SuperMicro SYS-7049GP-TRT.
1007         - Motherboard: SuperMicro X11DPG-QT.
1008         - Processors: 2* Intel Platinum 8180 2.3 GHz.
1009         - RAM Memory: 16* 16GB DDR4-2666MHz.
1010         - Disks: 2* 1.6TB 6G SATA SSD.
1011     - NICs configuration:
1012         - Numa0: (x16, x16, x16 PCIe3.0 lanes)
1013             - PCIe Slot2 18:00.xx: x710-4p10GE Intel.
1014             - PCIe Slot4 3b:00.xx: empty.
1015             - PCIe Slot9 5e:00.xx: empty.
1016         - Numa1: (x16, x16, x16 PCIe3.0 lanes)
1017             - PCIe Slot6 86:00.xx: empty.
1018             - PCIe Slot8 af:00.xx: empty.
1019             - PCIe Slot10 d8:00.xx: empty.
1020 9. Type-9: Purpose - (Marvell ThunderX2 Processor) SUT for VPP_Device functional
1021    tests.
1022     - Quantity: 1
1023     - Physical connectivity:
1024         - IPMI and host management ports.
1025         - NIC ports connected into 1-node topologies.
1026     - Main HW configuration:
1027         - Chassis: Gigabyte R181-T90 1U
1028         - Motheboard: MT91-FS1
1029         - Processors: 1* ThunderX2 ARMv8 CN9975 2.0 GHz
1030         - RAM Memory: 4* 32GB RDIMM
1031         - Disks: 1* 480GB SSD Micron, 1* 1000GB HDD Seagate_25
1032     - NICs configuration:
1033         - Numa0:
1034             - PCIe Slot1 05:00.xx: XL710-QDA2.
1035             - PCIe Slot3 08:00.xx: XL710-QDA2.
1036         - Numa1:
1037             - PCIe Slot6 85:00.xx: XL710-QDA2.
1038 10. Type-10: Purpose - (Intel Atom C3000 Processor) SUT for SW Data Plane
1039     Workload i.e. VPP, testpmd.
1040     - Quantity: 4
1041     - Physical connectivity:
1042         - IPMI and host management ports.
1043         - NIC ports.
1044     - Main HW configuration:
1045         - Chassis: SuperMicro SYS-E300-9A
1046         - Processors: 1* Intel(R) Atom(TM) CPU C3858 @ 2.00GHz
1047         - RAM Memory: 32GB ECC
1048         - Disks: 480 GB
1049     - NICs configuration:
1050         - 2x 10Gb Intel x553 fiber ports
1051         - 2x 10Gb Intel x553 copper ports
1052         - 4x 1GB Intel I350 ports
1053 ```
1054
1055 ### Xeon Skx Server BIOS Configuration
1056
1057 #### Boot Feature
1058
1059 ```
1060   |  Quiet Boot                                [Enabled]               |Boot option                  |
1061   |                                                                    |                             |
1062   |  Option ROM Messages                       [Force BIOS]            |                             |
1063   |  Bootup NumLock State                      [On]                    |                             |
1064   |  Wait For "F1" If Error                    [Enabled]               |                             |
1065   |  INT19 Trap Response                       [Immediate]             |                             |
1066   |  Re-try Boot                               [Disabled]              |                             |
1067   |  Install Windows 7 USB support             [Disabled]              |                             |
1068   |  Port 61h Bit-4 Emulation                  [Disabled]              |                             |
1069   |                                                                    |                             |
1070   |  Power Configuration                                               |                             |
1071   |  Watch Dog Function                        [Disabled]              |                             |
1072   |  Restore on AC Power Loss                  [Last State]            |                             |
1073   |  Power Button Function                     [Instant Off]           |                             |
1074   |  Throttle on Power Fail                    [Disabled]              |                             |
1075 ```
1076
1077 #### CPU Configuration
1078
1079 ```
1080   |  Processor Configuration                                           |Enables Hyper Threading      |
1081   |  --------------------------------------------------                |(Software Method to          |
1082   |  Processor BSP Revision                    50654 - SKX H0          |Enable/Disable Logical       |
1083   |  Processor Socket                          CPU1      |  CPU2       |Processor threads.           |
1084   |  Processor ID                              00050654* |  000506...  |                             |
1085   |  Processor Frequency                       2.500GHz  |  2.500GHz   |                             |
1086   |  Processor Max Ratio                            19H  |  19H        |                             |
1087   |  Processor Min Ratio                            0AH  |  0AH        |                             |
1088   |  Microcode Revision                        02000030                |                             |
1089   |  L1 Cache RAM                                  64KB  |      64KB   |                             |
1090   |  L2 Cache RAM                                1024KB  |    1024KB   |                             |
1091   |  L3 Cache RAM                               39424KB  |   39424KB   |                             |
1092   |  Processor 0 Version                                               |                             |
1093   |  Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz                      |                             |
1094   |  Processor 1 Version                                               |                             |
1095   |  Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz                      |                             |
1096   |                                                                    |                             |
1097   |  Hyper-Threading [ALL]                     [Enable]                |                             |
1098   |  Core Disable Bitmap(Hex)                  0                       |                             |
1099   |  Execute Disable Bit                       [Enable]                |                             |
1100   |  Intel Virtualization Technology           [Enable]                |                             |
1101   |  PPIN Control                              [Unlock/Enable]         |                             |
1102   |  Hardware Prefetcher                       [Enable]                |                             |
1103   |  Adjacent Cache Prefetch                   [Enable]                |                             |
1104   |  DCU Streamer Prefetcher                   [Enable]                |                             |
1105   |  DCU IP Prefetcher                         [Enable]                |                             |
1106   |  LLC Prefetch                              [Disable]               |                             |
1107   |  Extended APIC                             [Disable]               |                             |
1108   |  AES-NI                                    [Enable]                |                             |
1109   |> Advanced Power Management Configuration                           |                             |
1110 ```
1111
1112 ##### Advanced Power Management Configuration
1113
1114 ```
1115   |  Advanced Power Management Configuration                           |Switch CPU Power Management  |
1116   |  --------------------------------------------------                |profile                      |
1117   |  Power Technology                          [Custom]                |                             |
1118   |  Power Performance Tuning                  [BIOS Controls EPB]     |                             |
1119   |  ENERGY_PERF_BIAS_CFG mode                 [Maximum Performance]   |                             |
1120   |> CPU P State Control                                               |                             |
1121   |> Hardware PM State Control                                         |                             |
1122   |> CPU C State Control                                               |                             |
1123   |> Package C State Control                                           |                             |
1124   |> CPU T State Control                                               |                             |
1125 ```
1126
1127 ###### CPU P State Control
1128
1129 ```
1130   |  CPU P State Control                                               |Enable/Disable EIST          |
1131   |                                                                    |(P-States)                   |
1132   |  SpeedStep (Pstates)                       [Disable]               |                             |
1133   |  EIST PSD Function                         [HW_ALL]                |                             |
1134 ```
1135
1136 ###### Hardware PM State Control
1137
1138 ```
1139   |  Hardware PM State Control                                         |Disable: Hardware chooses a  |
1140   |                                                                    |P-state based on OS Request  |
1141   |  Hardware P-States                         [Disable]               |(Legacy P-States)            |
1142   |                                                                    |Native Mode:Hardware         |
1143   |                                                                    |chooses a P-state based on   |
1144   |                                                                    |OS guidance                  |
1145   |                                                                    |Out of Band Mode:Hardware    |
1146   |                                                                    |autonomously chooses a       |
1147   |                                                                    |P-state (no OS guidance)     |
1148 ```
1149
1150 ###### CPU C State Control
1151
1152 ```
1153   |  CPU C State Control                                               |Autonomous Core C-State      |
1154   |                                                                    |Control                      |
1155   |  Autonomous Core C-State                   [Disable]               |                             |
1156   |  CPU C6 report                             [Disable]               |                             |
1157   |  Enhanced Halt State (C1E)                 [Disable]               |                             |
1158 ```
1159
1160 ###### Package C State Control
1161
1162 ```
1163   |  Package C State Control                                           |Package C State limit        |
1164   |                                                                    |                             |
1165   |  Package C State                           [C0/C1 state]           |                             |
1166 ```
1167
1168 ###### CPU T State Control
1169
1170 ```
1171   |  CPU T State Control                                               |Enable/Disable Software      |
1172   |                                                                    |Controlled T-States          |
1173   |  Software Controlled T-States              [Disable]               |                             |
1174 ```
1175
1176 ##### Chipset Configuration
1177
1178 ```
1179   |  WARNING: Setting wrong values in below sections may cause         |North Bridge Parameters      |
1180   |           system to malfunction.                                   |                             |
1181   |> North Bridge                                                      |                             |
1182   |> South Bridge                                                      |                             |
1183 ```
1184
1185 ###### North Bridge
1186
1187 ```
1188   |> UPI Configuration                                                 |Displays and provides        |
1189   |> Memory Configuration                                              |option to change the UPI     |
1190   |> IIO Configuration                                                 |Settings                     |
1191 ```
1192
1193 ###### UPI Configuration
1194
1195 ```
1196   |  UPI Configuration                                                 |Choose Topology Precedence   |
1197   |  --------------------------------------------------                |to degrade features if       |
1198   |  Number of CPU                             2                       |system options are in        |
1199   |  Number of Active UPI Link                 3                       |conflict or choose Feature   |
1200   |  Current UPI Link Speed                    Fast                    |Precedence to degrade        |
1201   |  Current UPI Link Frequency                10.4 GT/s               |topology if system options   |
1202   |  UPI Global MMIO Low Base / Limit          90000000 / FBFFFFFF     |are in conflict.             |
1203   |  UPI Global MMIO High Base / Limit         0000000000000000 / ...  |                             |
1204   |  UPI Pci-e Configuration Base / Size       80000000 / 10000000     |                             |
1205   |  Degrade Precedence                        [Topology Precedence]   |                             |
1206   |  Link L0p Enable                           [Disable]               |                             |
1207   |  Link L1 Enable                            [Disable]               |                             |
1208   |  IO Directory Cache (IODC)                 [Auto]                  |                             |
1209   |  SNC                                       [Disable]               |                             |
1210   |  XPT Prefetch                              [Disable]               |                             |
1211   |  KTI Prefetch                              [Enable]                |                             |
1212   |  Local/Remote Threshold                    [Auto]                  |                             |
1213   |  Stale AtoS                                [Disable]               |                             |
1214   |  LLC dead line alloc                       [Enable]                |                             |
1215   |  Isoc Mode                                 [Auto]                  |                             |
1216 ```
1217
1218 ###### Memory Configuration
1219
1220 ```
1221   |                                                                    |POR - Enforces Plan Of       |
1222   |  --------------------------------------------------                |Record restrictions for      |
1223   |  Integrated Memory Controller (iMC)                                |DDR4 frequency and voltage   |
1224   |  --------------------------------------------------                |programming. Disable -       |
1225   |                                                                    |Disables this feature.       |
1226   |  Enforce POR                               [Disable]               |                             |
1227   |  Memory Frequency                          [2666]                  |                             |
1228   |  Data Scrambling for NVMDIMM               [Auto]                  |                             |
1229   |  Data Scrambling for DDR4                  [Auto]                  |                             |
1230   |  tCCD_L Relaxation                         [Auto]                  |                             |
1231   |  Memory tRWSR Relaxation                   [Enable]                |                             |
1232   |  2X REFRESH                                [Auto]                  |                             |
1233   |  Page Policy                               [Auto]                  |                             |
1234   |  IMC Interleaving                          [2-way Interleave]      |                             |
1235   |> Memory Topology                                                   |                             |
1236   |> Memory RAS Configuration                                          |                             |
1237 ```
1238
1239 ###### IIO Configuration
1240
1241 ```
1242   |  IIO Configuration                                                 |Expose IIO DFX devices and   |
1243   |  --------------------------------------------------                |other CPU devices like PMON  |
1244   |                                                                    |                             |
1245   |  EV DFX Features                           [Disable]               |                             |
1246   |> CPU1 Configuration                                                |                             |
1247   |> CPU2 Configuration                                                |                             |
1248   |> IOAT Configuration                                                |                             |
1249   |> Intel. VT for Directed I/O (VT-d)                                 |                             |
1250   |> Intel. VMD technology                                             |                             |
1251   |                                                                    |                             |
1252   |   IIO-PCIE Express Global Options                                  |                             |
1253   |  ========================================                          |                             |
1254   |  PCI-E Completion Timeout Disable          [No]                    |                             |
1255 ```
1256
1257 ###### CPU1 Configuration
1258
1259 ```
1260   |  IOU0 (IIO PCIe Br1)                       [Auto]                  |Selects PCIe port            |
1261   |  IOU1 (IIO PCIe Br2)                       [Auto]                  |Bifurcation for selected     |
1262   |  IOU2 (IIO PCIe Br3)                       [Auto]                  |slot(s)                      |
1263   |> CPU1 SLOT2 PCI-E 3.0 X16                                          |                             |
1264   |> CPU1 SLOT4 PCI-E 3.0 X16                                          |                             |
1265   |> CPU1 SLOT9 PCI-E 3.0 X16                                          |                             |
1266 ```
1267
1268 ###### CPU2 Configuration
1269
1270 ```
1271   |  IOU0 (IIO PCIe Br1)                       [Auto]                  |Selects PCIe port            |
1272   |  IOU1 (IIO PCIe Br2)                       [Auto]                  |Bifurcation for selected     |
1273   |  IOU2 (IIO PCIe Br3)                       [Auto]                  |slot(s)                      |
1274   |> CPU2 SLOT6 PCI-E 3.0 X16                                          |                             |
1275   |> CPU2 SLOT8 PCI-E 3.0 X16                                          |                             |
1276   |> CPU2 SLOT10 PCI-E 3.0 X16                                         |                             |
1277 ```
1278
1279 ##### South Bridge
1280
1281 ```
1282   |                                                                    |Enables Legacy USB support.  |
1283   |  USB Module Version                        17                      |AUTO option disables legacy  |
1284   |                                                                    |support if no USB devices    |
1285   |  USB Devices:                                                      |are connected. DISABLE       |
1286   |        1 Keyboard, 1 Mouse, 1 Hub                                  |option will keep USB         |
1287   |                                                                    |devices available only for   |
1288   |  Legacy USB Support                        [Enabled]               |EFI applications.            |
1289   |  XHCI Hand-off                             [Disabled]              |                             |
1290   |  Port 60/64 Emulation                      [Enabled]               |                             |
1291   |  PCIe PLL SSC                              [Disable]               |                             |
1292   |  Real USB Wake Up                          [Enabled]               |                             |
1293   |  Front USB Wake Up                         [Enabled]               |                             |
1294   |                                                                    |                             |
1295   |  Azalia                                    [Auto]                  |                             |
1296   |    Azalia PME Enable                       [Disabled]              |                             |
1297 ```
1298
1299 #### PCIe/PCI/PnP Configuration
1300
1301 ```
1302   |  PCI Bus Driver Version                    A5.01.12                |Enables or Disables 64bit    |
1303   |                                                                    |capable Devices to be        |
1304   |  PCI Devices Common Settings:                                      |Decoded in Above 4G Address  |
1305   |  Above 4G Decoding                         [Enabled]               |Space (Only if System        |
1306   |  SR-IOV Support                            [Enabled]               |Supports 64 bit PCI          |
1307   |  MMIO High Base                            [56T]                   |Decoding).                   |
1308   |  MMIO High Granularity Size                [256G]                  |                             |
1309   |  Maximum Read Request                      [Auto]                  |                             |
1310   |  MMCFG Base                                [2G]                    |                             |
1311   |  NVMe Firmware Source                      [Vendor Defined Fi...]  |                             |
1312   |  VGA Priority                              [Onboard]               |                             |
1313   |  CPU1 SLOT2 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
1314   |  CPU1 SLOT4 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
1315   |  CPU2 SLOT6 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
1316   |  CPU2 SLOT8 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
1317   |  CPU1 SLOT9 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
1318   |  CPU2 SLOT10 PCI-E 3.0 X16 OPROM           [Legacy]                |                             |
1319   |  CPU2 SLOT11 PCI-E 3.0 X4(IN X8) OPROM     [Legacy]                |                             |
1320   |  M.2 CONNECTOR OPROM                       [Legacy]                |                             |
1321   |  Onboard LAN1 Option ROM                   [Legacy]                |                             |
1322   |  Onboard LAN2 Option ROM                   [Disabled]              |                             |
1323   |  Onboard Video Option ROM                  [Legacy]                |                             |
1324   |> Network Stack Configuration                                       |                             |
1325 ```
1326
1327 #### ACPI Settings
1328
1329 ```
1330   |  ACPI Settings                                                     |Enable or Disable Non        |
1331   |                                                                    |uniform Memory Access        |
1332   |  NUMA                                      [Enabled]               |(NUMA).                      |
1333   |  WHEA Support                              [Enabled]               |                             |
1334   |  High Precision Event Timer                [Enabled]               |                             |
1335   |  ACPI Sleep State                          [S3 (Suspend to RAM)]   |                             |
1336 ```
1337
1338 #### DMIDECODE
1339
1340 ```
1341   # dmidecode 3.1
1342   Getting SMBIOS data from sysfs.
1343   SMBIOS 3.1.1 present.
1344   Table at 0x000E89C0.
1345
1346   Handle 0x0000, DMI type 0, 26 bytes
1347   BIOS Information
1348         Vendor: American Megatrends Inc.
1349         Version: 2.0
1350         Release Date: 11/29/2017
1351         Address: 0xF0000
1352         Runtime Size: 64 kB
1353         ROM Size: 64 MB
1354         Characteristics:
1355                 PCI is supported
1356                 BIOS is upgradeable
1357                 BIOS shadowing is allowed
1358                 Boot from CD is supported
1359                 Selectable boot is supported
1360                 BIOS ROM is socketed
1361                 EDD is supported
1362                 5.25"/1.2 MB floppy services are supported (int 13h)
1363                 3.5"/720 kB floppy services are supported (int 13h)
1364                 3.5"/2.88 MB floppy services are supported (int 13h)
1365                 Print screen service is supported (int 5h)
1366                 Serial services are supported (int 14h)
1367                 Printer services are supported (int 17h)
1368                 ACPI is supported
1369                 USB legacy is supported
1370                 BIOS boot specification is supported
1371                 Targeted content distribution is supported
1372                 UEFI is supported
1373         BIOS Revision: 5.12
1374
1375   Handle 0x0001, DMI type 1, 27 bytes
1376   System Information
1377         Manufacturer: Supermicro
1378         Product Name: SYS-7049GP-TRT
1379         Version: 0123456789
1380         Serial Number: S291427X8332242
1381         UUID: 00000000-0000-0000-0000-AC1F6B8A8DB6
1382         Wake-up Type: Power Switch
1383         SKU Number: To be filled by O.E.M.
1384         Family: To be filled by O.E.M.
1385
1386   Handle 0x0002, DMI type 2, 15 bytes
1387   Base Board Information
1388         Manufacturer: Supermicro
1389         Product Name: X11DPG-QT
1390         Version: 1.02
1391         Serial Number: VM183S014930
1392         Asset Tag: To be filled by O.E.M.
1393         Features:
1394                 Board is a hosting board
1395                 Board is replaceable
1396         Location In Chassis: To be filled by O.E.M.
1397         Chassis Handle: 0x0003
1398         Type: Motherboard
1399         Contained Object Handles: 0
1400
1401   Handle 0x0003, DMI type 3, 22 bytes
1402   Chassis Information
1403         Manufacturer: Supermicro
1404         Type: Other
1405         Lock: Not Present
1406         Version: 0123456789
1407         Serial Number: C7470KH06A20167
1408         Asset Tag: To be filled by O.E.M.
1409         Boot-up State: Safe
1410         Power Supply State: Safe
1411         Thermal State: Safe
1412         Security Status: None
1413         OEM Information: 0x00000000
1414
1415   Handle 0x0050, DMI type 4, 48 bytes
1416   Processor Information
1417         Socket Designation: CPU1
1418         Type: Central Processor
1419         Family: Xeon
1420         Manufacturer: Intel(R) Corporation
1421         ID: 54 06 05 00 FF FB EB BF
1422         Signature: Type 0, Family 6, Model 85, Stepping 4
1423         Flags:
1424                 FPU (Floating-point unit on-chip)
1425                 VME (Virtual mode extension)
1426                 DE (Debugging extension)
1427                 PSE (Page size extension)
1428                 TSC (Time stamp counter)
1429                 MSR (Model specific registers)
1430                 PAE (Physical address extension)
1431                 MCE (Machine check exception)
1432                 CX8 (CMPXCHG8 instruction supported)
1433                 APIC (On-chip APIC hardware supported)
1434                 SEP (Fast system call)
1435                 MTRR (Memory type range registers)
1436                 PGE (Page global enable)
1437                 MCA (Machine check architecture)
1438                 CMOV (Conditional move instruction supported)
1439                 PAT (Page attribute table)
1440                 PSE-36 (36-bit page size extension)
1441                 CLFSH (CLFLUSH instruction supported)
1442                 DS (Debug store)
1443                 ACPI (ACPI supported)
1444                 MMX (MMX technology supported)
1445                 FXSR (FXSAVE and FXSTOR instructions supported)
1446                 SSE (Streaming SIMD extensions)
1447                 SSE2 (Streaming SIMD extensions 2)
1448                 SS (Self-snoop)
1449                 HTT (Multi-threading)
1450                 TM (Thermal monitor supported)
1451                 PBE (Pending break enabled)
1452         Version: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
1453         Voltage: 1.6 V
1454         External Clock: 100 MHz
1455         Max Speed: 4000 MHz
1456         Current Speed: 2500 MHz
1457         Status: Populated, Enabled
1458         Upgrade: Other
1459         L1 Cache Handle: 0x004D
1460         L2 Cache Handle: 0x004E
1461         L3 Cache Handle: 0x004F
1462         Serial Number: Not Specified
1463         Asset Tag: UNKNOWN
1464         Part Number: Not Specified
1465         Core Count: 28
1466         Core Enabled: 28
1467         Thread Count: 56
1468         Characteristics:
1469                 64-bit capable
1470                 Multi-Core
1471                 Hardware Thread
1472                 Execute Protection
1473                 Enhanced Virtualization
1474                 Power/Performance Control
1475
1476
1477   Handle 0x0054, DMI type 4, 48 bytes
1478   Processor Information
1479         Socket Designation: CPU2
1480         Type: Central Processor
1481         Family: Xeon
1482         Manufacturer: Intel(R) Corporation
1483         ID: 54 06 05 00 FF FB EB BF
1484         Signature: Type 0, Family 6, Model 85, Stepping 4
1485         Flags:
1486                 FPU (Floating-point unit on-chip)
1487                 VME (Virtual mode extension)
1488                 DE (Debugging extension)
1489                 PSE (Page size extension)
1490                 TSC (Time stamp counter)
1491                 MSR (Model specific registers)
1492                 PAE (Physical address extension)
1493                 MCE (Machine check exception)
1494                 CX8 (CMPXCHG8 instruction supported)
1495                 APIC (On-chip APIC hardware supported)
1496                 SEP (Fast system call)
1497                 MTRR (Memory type range registers)
1498                 PGE (Page global enable)
1499                 MCA (Machine check architecture)
1500                 CMOV (Conditional move instruction supported)
1501                 PAT (Page attribute table)
1502                 PSE-36 (36-bit page size extension)
1503                 CLFSH (CLFLUSH instruction supported)
1504                 DS (Debug store)
1505                 ACPI (ACPI supported)
1506                 MMX (MMX technology supported)
1507                 FXSR (FXSAVE and FXSTOR instructions supported)
1508                 SSE (Streaming SIMD extensions)
1509                 SSE2 (Streaming SIMD extensions 2)
1510                 SS (Self-snoop)
1511                 HTT (Multi-threading)
1512                 TM (Thermal monitor supported)
1513                 PBE (Pending break enabled)
1514         Version: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
1515         Voltage: 1.6 V
1516         External Clock: 100 MHz
1517         Max Speed: 4000 MHz
1518         Current Speed: 2500 MHz
1519         Status: Populated, Enabled
1520         Upgrade: Other
1521         L1 Cache Handle: 0x0051
1522         L2 Cache Handle: 0x0052
1523         L3 Cache Handle: 0x0053
1524         Serial Number: Not Specified
1525         Asset Tag: UNKNOWN
1526         Part Number: Not Specified
1527         Core Count: 28
1528         Core Enabled: 28
1529         Thread Count: 56
1530         Characteristics:
1531                 64-bit capable
1532                 Multi-Core
1533                 Hardware Thread
1534                 Execute Protection
1535                 Enhanced Virtualization
1536                 Power/Performance Control
1537 ```
1538
1539 ### Xeon Skx Server Firmware Inventory
1540
1541 ```
1542 Host.           IPMI IP.      BIOS. CPLD.     Aptio SU.   CPU Microcode.  PCI Bus.   ME Operation FW.    X710 Firmware.            XXV710 Firmware.          i40e.
1543 s1-t11-sut1.    10.30.50.47.  2.1.  03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1544 s2-t12-sut1.    10.30.50.48.  2.1.  03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1545 s3-t21-sut1.    10.30.50.41.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1546 s4-t21-tg1.     10.30.50.42.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1547 s5-t22-sut1.    10.30.50.49.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1548 s6-t22-tg1.     10.30.50.50.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1549 s7-t23-sut1.    10.30.50.51.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1550 s8-t23-tg1.     10.30.50.52.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1551 s9-t24-sut1.    10.30.50.53.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1552 s10-t24-tg1.    10.30.50.54.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1553 s11-t31-sut1.   10.30.50.43.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1554 s12-t31-sut2.   10.30.50.44.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1555 s13-t31-tg1.    10.30.50.45.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1556 s14-t32-sut1.   10.30.50.55.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1557 s15-t32-sut2.   10.30.50.56.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1558 s16-t32-tg1.    10.30.50.57.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1559 s19-t33t34-tg1. 10.30.50.46.  2.0b. 03.B1.03. 2.19.1268.  02000043.       A5.01.12.  4.0.4.294.          6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
1560 ```