## FD.io CSIT testbeds - Xeon Skylake, Arm, Atom
-This is a low-level design implemented as an extensions of FD.io CSIT lab to accommodate the new Intel Xeon Skylake, Arm AArch64 and Atom devices. Content has been copied from [FD.io CSIT testbeds wiki page: Xeon_skx, Arm, Atom](https://wiki.fd.io/view/CSIT/Testbeds:_Xeon_Skx,_Arm,_Atom).
+
+This is a low-level design implemented as an extensions of FD.io CSIT lab to
+accommodate the new Intel Xeon Skylake, Arm AArch64 and Atom devices.
## Testbeds Overview
+
### Testbeds Type Breakdown
+
```
#. CSIT_tb. Purpose. SUT. TG. #tb. #SUTs. #TGs. #skx_node.
1. 1-node Xeon. func. skx. n/a. 2. 2. 0. 2.
```
### 1-Node Xeon Testbeds
-One 1-node Xeon testbed for VPP_Device tests is built using one SUT (Type-6 server), with NIC ports connected back-to-back.
+
+One 1-node Xeon testbed for VPP_Device tests is built using one SUT (Type-6
+server), with NIC ports connected back-to-back.
### 2-Node Xeon Testbeds
-Four 2-node Xeon testbeds (are expected to be built|are built), with each testbed using one SUTs (Type-1 server) and one TG (Type-2 server) connected back-to-back. NIC cards placement into slots and NIC ports connectivity is following the testbed specification included in next sections.
+
+Four 2-node Xeon testbeds (are expected to be built|are built), with each
+testbed using one SUTs (Type-1 server) and one TG (Type-2 server) connected
+back-to-back. NIC cards placement into slots and NIC ports connectivity is
+following the testbed specification included in next sections.
### 3-Node Xeon Testbeds
-Two 3-node Xeon testbeds (are expected to be built|are built), with each testbed using two SUTs (Type-1 server) and one TG (Type-2 server) connected in full-mesh triangle. NIC cards placement into slots and NIC ports connectivity is following the testbed specification included in next sections.
+
+Two 3-node Xeon testbeds (are expected to be built|are built), with each testbed
+using two SUTs (Type-1 server) and one TG (Type-2 server) connected in full-mesh
+triangle. NIC cards placement into slots and NIC ports connectivity is following
+the testbed specification included in next sections.
### 1-Node Arm Marvell ThunderX2 Testbed
-One 1-node ThunderX2 testbed for VPP_Device tests is expected to be built using one SUT (Type-9 server), with NIC ports connected back-to-back.
+
+One 1-node ThunderX2 testbed for VPP_Device tests is expected to be built using
+one SUT (Type-9 server), with NIC ports connected back-to-back.
### 3-Node Arm Huawei TaiShan Testbed
-One 3-node TaiShan testbed is built, with each testbed using two SUTs (Type-3 server) and one TG (Type-2 server) connected in full-mesh triangle. NIC cards placement into slots and NIC ports connectivity is following the testbed specification included in next sections.
+
+One 3-node TaiShan testbed is built, with each testbed using two SUTs (Type-3
+server) and one TG (Type-2 server) connected in full-mesh triangle. NIC cards
+placement into slots and NIC ports connectivity is following the testbed
+specification included in next sections.
### 3-Node Arm Marvell MACCHIATObin Testbed
-One 3-node MACCHIATObin testbeds is built, with each testbed using two SUTs (Type-4 server) and one TG (Type-2 server) connected in full-mesh triangle. Built-in NIC ports connectivity is following the testbed specification included in next sections.
+
+One 3-node MACCHIATObin testbeds is built, with each testbed using two SUTs
+(Type-4 server) and one TG (Type-2 server) connected in full-mesh triangle.
+Built-in NIC ports connectivity is following the testbed specification included
+in next sections.
### TCP/IP and L47 Testbeds
+
One 2-node Ixia PS One and Xeon server testbed, for TCP/IP host stack tests.
### Atom Testbeds
-One 3-node Atom (Netgate based) testbed is built consisting of three SUTs (Type-5 Netgate device.) NIC cards placement into slots and NIC ports connectivity is following the testbed specification included in the next section.
+One 3-node Atom (Netgate based) testbed is built consisting of three SUTs
+(Type-5 Netgate device.) NIC cards placement into slots and NIC ports
+connectivity is following the testbed specification included in the next
+section.
## Inventory
+
### Appliances
+
```
1. Ixia PerfectStorm One Appliance
- 1 * PS10GE4NG
```
### Arm Servers
+
```
1. Arm Cortex A-72 servers
- 1 * ThunderX2
| | VPP dev debug | Up, Manual VPP Device experiments, Full Skx TG too much for it - suggest to use LXC/DRC TG(!) | fdio-mcbin3 | 10.30.51.43 | 10.30.51.49, then connect to /dev/ttyUSB2 | 4 | 16GB | 2x10GbE SFP+ | Ubuntu 16.04.5
### Xeon and Atom Servers
+
```
1. Intel Xeon servers:
- 20 * SuperMicro SYS-7049GP-TRT with Xeon Skylake processors.
```
### Network Interface Cards
+
```
1. 10GE NICs
- 14 * Intel® Ethernet Converged Network Adapter X710-DA4
```
### Pluggables and Cables
+
Pluggables:
+
```
1. 10GE SFP+
- 16 * Intel E10GSFPSR Ethernet SFP+ SR Optics
```
Standalone cables:
+
```
1. 10GE
- None
```
### Other Network Cards
+
Any QATs?
## Installation Status
-Lab installation status is tracked by LF IT team in [FD.io Server Status](https://docs.google.com/document/d/16TdvGC73wuNQjkP355MTXRckv7yqnaxJkWwX7G7izEo/edit?ts=5b10411b#heading=h.dprb64shku8u).
+
+Lab installation status is tracked by LF IT team in
+[FD.io Server Status](https://docs.google.com/document/d/16TdvGC73wuNQjkP355MTXRckv7yqnaxJkWwX7G7izEo/edit?ts=5b10411b#heading=h.dprb64shku8u).
## Server/Device Management and Naming
+
### Server Management Requirements
+
Total of 20 SM SYS-7049GP-TRT servers are made available for FD.IO CSIT testbed.
-For management purposes, each server must have following two ports connected to the management network:
+For management purposes, each server must have following two ports connected to
+the management network:
+
```
- 1GE IPMI port
- IPMI - Intelligent Platform Management Interface.
- - Required for access to embedded server management with WebUI, CLI, SNMPv3, IPMIv2.0, for firmware (BIOS) and OS updates.
+ - Required for access to embedded server management with WebUI, CLI, SNMPv3,
+ IPMIv2.0, for firmware (BIOS) and OS updates.
- 1GE/10GE management port
- hostOS management port for general system management.
```
### Server and Port Naming Convention
-Following naming convention is used within this page to specify physical connectivity and wiring across defined CSIT testbeds:
+
+Following naming convention is used within this page to specify physical
+connectivity and wiring across defined CSIT testbeds:
+
```
- testbedname: testbedN.
- hostname:
```
### Server Management - Addressing
-Each server has a LOM (Lights-Out-Management e.g. SM IPMI) and a Management port, which are connected to two different VLANs.
+
+Each server has a LOM (Lights-Out-Management e.g. SM IPMI) and a Management
+port, which are connected to two different VLANs.
+
```
1. LOM (IPMI) VLAN:
- Subnet: 10.30.50.0/24
- DNS1: 199.204.44.24
- DNS2: 199.204.47.54
```
+
To access these hosts, an VPN connection is required.
### LOM (IPMI) VLAN IP Addresses
..
## Testbeds Specification - Target Build
+
### Server/Ports Naming, NIC Placement
+
#### 1-Node Xeon
-Each server in 1-node Xeon topology has its NIC cards placed, and NIC cards and ports indexed per following specification:
+
+Each server in 1-node Xeon topology has its NIC cards placed, and NIC cards and
+ports indexed per following specification:
+
```
- Server1 [Type-6]:
- testbedname: testbed11.
```
#### 2-Node Xeon
-Each server in 2-node Xeon topology has its NIC cards placed, and NIC cards and ports indexed per following specification:
+
+Each server in 2-node Xeon topology has its NIC cards placed, and NIC cards and
+ports indexed per following specification:
+
```
- Server3 [Type-1]:
- testbedname: testbed21.
```
#### 3-Node Xeon
-Each server in 3-node Xeon topology has its NIC cards placed, and NIC cards and ports indexed per following specification:
+
+Each server in 3-node Xeon topology has its NIC cards placed, and NIC cards and
+ports indexed per following specification:
+
```
- Server11 [Type-1]:
- testbedname: testbed31.
```
#### 1-Node Arm
+
```
- Server17 [Type-3]:
- testbedname: testbed13.
```
#### 3-Node Arm
+
Note: Server19 (TG) is shared between testbed33 & testbed34
+
```
- Server17 [Type-3]:
- testbedname: testbed33.
```
#### TCP/IP and L47
-Each server (appliance) in 2-node TCP/IP topology has its NIC cards placed, and NIC cards and ports indexed per following specification:
+
+Each server (appliance) in 2-node TCP/IP topology has its NIC cards placed, and
+NIC cards and ports indexed per following specification:
+
```
- Server25 [Type-8]:
- testbedname: testbed25.
```
#### 3-Node Atom
+
Note: There is no IPMI. Serial console is accessible via VIRL2 and VIRL3 USB.
+
```
- Server22 [Type-5]:
- testbedname: testbed35.
```
### Physical Connectivity within Testbeds
+
#### 1-Node Xeon
-Two 1-Node testbeds are constructed by connecting 2 Xeon servers using below specification:
+
+Two 1-Node testbeds are constructed by connecting 2 Xeon servers using below
+specification:
+
```
- testbed11:
- ring1 10GE-ports x710-4p10GE:
```
#### 2-Node Xeon
-Four 2-Node testbeds are constructed by connecting 8 Xeon servers using below specification:
+
+Four 2-Node testbeds are constructed by connecting 8 Xeon servers using below
+specification:
+
```
- testbed21:
- ring1 10GE-ports x710-4p10GE on SUT:
```
#### 3-Node Xeon
-Two 3-Node testbeds are constructed by connecting 6 Xeon servers using below specification:
+
+Two 3-Node testbeds are constructed by connecting 6 Xeon servers using below
+specification:
+
```
- testbed31:
- ring1 10GE-ports x710-4p10GE on SUTs:
#### 1-Node Arm
-One 1-Node testbed is constructed by connecting 1 Arm server using below specification:
+
+One 1-Node testbed is constructed by connecting 1 Arm server using below
+specification:
+
```
- testbed13:
- ring1 40GE-ports XL710-QDA2-2p40GE on SUTs:
- s27-t13-sut1-c6/p2 - s27-t13-sut1-c1/p1.
```
-
#### 3-Node Arm
-One 3-Node testbed is constructed by connecting 2 TaiShan servers and one Xeon server using below specification:
+
+One 3-Node testbed is constructed by connecting 2 TaiShan servers and one Xeon
+server using below specification:
+
```
- testbed33:
- ring1 10GE-ports 82599-2p10GE on SUTs:
- t33-sut1-c4/p1 - t33-sut2-c4/p2.
- t33-sut2-c4/p1 - t33t34-tg1-c4/p1.
```
-One 3-Node testbed is constructed by connecting 2 MACCHIATObin servers and one Xeon server using below specification:
+
+One 3-Node testbed is constructed by connecting 2 MACCHIATObin servers and one
+Xeon server using below specification:
+
```
- testbed34:
- ring1 10GE-ports Marvell on SUTs:
```
#### TCP/IP and L47
-One 2-Node TCP/IP testbed is constructed by connecting Ixia PSOne and 1 Xeon server using below specification:
+
+One 2-Node TCP/IP testbed is constructed by connecting Ixia PSOne and 1 Xeon
+server using below specification:
+
```
- testbed25:
- link1 10GE-port x710-4p10GE on SUT:
```
#### 3-Node Atom
+
..
## Server Specification
+
### Hardware Configuration
-The new FD.io CSIT-CPL lab (is expected to contain|contains) following hardware server configurations:
+
+The new FD.io CSIT-CPL lab (is expected to contain|contains) following hardware
+server configurations:
+
```
-1. Type-1: Purpose - (Intel Xeon Processor) SUT for SW Data Plane Workload i.e. VPP, testpmd.
+1. Type-1: Purpose - (Intel Xeon Processor) SUT for SW Data Plane Workload i.e.
+ VPP, testpmd.
- Quantity: TBD based on testbed allocation.
- Physical connectivity:
- IPMI and host management ports.
- PCIe Slot6 86:00.xx: empty.
- PCIe Slot8 af:00.xx: empty.
- PCIe Slot10 d8:00.xx: x710-4p10GE Intel.
-3. Type-3: Purpose - (Arm hip07-d05 Processor) SUT for SW Data Plane Workload i.e. VPP, testpmd.
+3. Type-3: Purpose - (Arm hip07-d05 Processor) SUT for SW Data Plane Workload
+ i.e. VPP, testpmd.
- Quantity: 2
- Physical connectivity:
- IPMI(?) and host management ports.
- NICs configuration:
- PCIe Slot4 e9:00.xx: connectx4-2p25GE Mellanox.
- PCIe Slot6 11:00.xx: 82599-2p10GE Intel.
-4. Type-4: Purpose - (Arm Armada 8040 Processor) SUT for SW Data Plane Workload i.e. VPP, testpmd.
+4. Type-4: Purpose - (Arm Armada 8040 Processor) SUT for SW Data Plane Workload
+ i.e. VPP, testpmd.
- Quantity: 3
- Physical connectivity:
- Host management ports.
- Disks: 1* 128GB(?) SATA SDD
- NICs configuration:
- pp2-2p10GE Marvell (on-chip Ethernet ports ; marvell plugin in VPP)
-5. Type-5: Purpose - (Intel Atom Processor) SUT for SW Data Plane Workload i.e. VPP, testpmd.
+5. Type-5: Purpose - (Intel Atom Processor) SUT for SW Data Plane Workload i.e.
+ VPP, testpmd.
- Quantity: TBD based on testbed allocation.
- Physical connectivity:
- Management: serial Port (usb) for console
- PCIe Slot6 86:00.xx: empty.
- PCIe Slot8 af:00.xx: empty.
- PCIe Slot10 d8:00.xx: empty.
-7. Type-7: Purpose - Ixia PerfectStorm One Appliance TG for TCP/IP performance tests.
+7. Type-7: Purpose - Ixia PerfectStorm One Appliance TG for TCP/IP performance
+ tests.
- Quantity: 1.
- Physical connectivity:
- Host management interface: 10/100/1000-BaseT.
- PCIe Slot6 86:00.xx: empty.
- PCIe Slot8 af:00.xx: empty.
- PCIe Slot10 d8:00.xx: empty.
-9. Type-9: Purpose - (Marvell ThunderX2 Processor) SUT for VPP_Device functional tests.
+9. Type-9: Purpose - (Marvell ThunderX2 Processor) SUT for VPP_Device functional
+ tests.
- Quantity: 1
- Physical connectivity:
- IPMI and host management ports.
- PCIe Slot3 08:00.xx: XL710-QDA2.
- Numa1:
- PCIe Slot6 85:00.xx: XL710-QDA2.
-10. Type-10: Purpose - (Intel Atom C3000 Processor) SUT for SW Data Plane Workload i.e. VPP, testpmd.
+10. Type-10: Purpose - (Intel Atom C3000 Processor) SUT for SW Data Plane
+ Workload i.e. VPP, testpmd.
- Quantity: 4
- Physical connectivity:
- IPMI and host management ports.
```
### Xeon Skx Server BIOS Configuration
+
#### Boot Feature
+
```
| Quiet Boot [Enabled] |Boot option |
| | |
```
#### CPU Configuration
+
```
| Processor Configuration |Enables Hyper Threading |
| -------------------------------------------------- |(Software Method to |
```
##### Advanced Power Management Configuration
+
```
| Advanced Power Management Configuration |Switch CPU Power Management |
| -------------------------------------------------- |profile |
```
###### CPU P State Control
+
```
| CPU P State Control |Enable/Disable EIST |
| |(P-States) |
```
###### Hardware PM State Control
+
```
| Hardware PM State Control |Disable: Hardware chooses a |
| |P-state based on OS Request |
```
###### CPU C State Control
+
```
| CPU C State Control |Autonomous Core C-State |
| |Control |
```
###### Package C State Control
+
```
| Package C State Control |Package C State limit |
| | |
```
###### CPU T State Control
+
```
| CPU T State Control |Enable/Disable Software |
| |Controlled T-States |
```
##### Chipset Configuration
+
```
| WARNING: Setting wrong values in below sections may cause |North Bridge Parameters |
| system to malfunction. | |
```
###### North Bridge
+
```
|> UPI Configuration |Displays and provides |
|> Memory Configuration |option to change the UPI |
```
###### UPI Configuration
+
```
| UPI Configuration |Choose Topology Precedence |
| -------------------------------------------------- |to degrade features if |
```
###### Memory Configuration
+
```
| |POR - Enforces Plan Of |
| -------------------------------------------------- |Record restrictions for |
```
###### IIO Configuration
+
```
| IIO Configuration |Expose IIO DFX devices and |
| -------------------------------------------------- |other CPU devices like PMON |
```
###### CPU1 Configuration
+
```
| IOU0 (IIO PCIe Br1) [Auto] |Selects PCIe port |
| IOU1 (IIO PCIe Br2) [Auto] |Bifurcation for selected |
```
###### CPU2 Configuration
+
```
| IOU0 (IIO PCIe Br1) [Auto] |Selects PCIe port |
| IOU1 (IIO PCIe Br2) [Auto] |Bifurcation for selected |
```
##### South Bridge
+
```
| |Enables Legacy USB support. |
| USB Module Version 17 |AUTO option disables legacy |
```
#### PCIe/PCI/PnP Configuration
+
```
| PCI Bus Driver Version A5.01.12 |Enables or Disables 64bit |
| |capable Devices to be |
```
#### ACPI Settings
+
```
| ACPI Settings |Enable or Disable Non |
| |uniform Memory Access |
```
#### DMIDECODE
+
```
# dmidecode 3.1
Getting SMBIOS data from sysfs.
```
### Xeon Skx Server Firmware Inventory
+
```
Host. IPMI IP. BIOS. CPLD. Aptio SU. CPU Microcode. PCI Bus. ME Operation FW. X710 Firmware. XXV710 Firmware. i40e.
s1-t11-sut1. 10.30.50.47. 2.1. 03.B1.03. 2.19.1268. 02000043. A5.01.12. 4.0.4.294. 6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k.
Code Review / csit.git / blobdiff