X-Git-Url: https://gerrit.fd.io/r/gitweb?a=blobdiff_plain;ds=sidebyside;f=docs%2Flab%2Ftestbed_specifications.md;h=afa36ff56c7be09621e85bae6a1498aadf3a1981;hb=5982882c045085b1656b14efc4e52bbb59c4b264;hp=b12d42d22f7c40e4f1a289f88ddbb383f43e8130;hpb=ddbfd9e68e81e77bd253c8b6cd81839ec3bd2715;p=csit.git diff --git a/docs/lab/testbed_specifications.md b/docs/lab/testbed_specifications.md index b12d42d22f..afa36ff56c 100644 --- a/docs/lab/testbed_specifications.md +++ b/docs/lab/testbed_specifications.md @@ -1,58 +1,57 @@ - - - - [FD.io CSIT Testbed Specifications](#fdio-csit-testbed-specifications) - - [Testbeds Overview](#testbeds-overview) - - [Summary List](#summary-list) - - [1-Node-Skylake Xeon Intel (1n-skx)](#1-node-skylake-xeon-intel-1n-skx) - - [1-Node-ThunderX2 Arm Marvell (1n-tx2)](#1-node-thunderx2-arm-marvell-1n-tx2) - - [2-Node-Skylake Xeon Intel (2n-skx)](#2-node-skylake-xeon-intel-2n-skx) - - [2-Node-Denverton Atom Intel (2n-dnv)](#2-node-denverton-atom-intel-2n-dnv) - - [2-Node-IxiaPS1L47 Ixia PSOne L47 (2n-ps1)](#2-node-ixiaps1l47-ixia-psone-l47-2n-ps1) - - [3-Node-Haswell Xeon Intel (3n-skx)](#3-node-haswell-xeon-intel-3n-skx) - - [3-Node-Skylake Xeon Intel (3n-skx)](#3-node-skylake-xeon-intel-3n-skx) - - [3-Node-TaiShan Arm Huawei (3n-tsh)](#3-node-taishan-arm-huawei-3n-tsh) - - [3-Node-MACCHIATObin Arm Marvell](#3-node-macchiatobin-arm-marvell) - - [3-Node-Rangeley Atom Testbeds](#3-node-rangeley-atom-testbeds) - - [Server Management](#server-management) - - [Requirements](#requirements) - - [Addressing](#addressing) - - [LOM (IPMI) VLAN IP Addresses](#lom-ipmi-vlan-ip-addresses) - - [Management VLAN IP Addresses](#management-vlan-ip-addresses) - - [Server Type Specification](#server-type-specification) - - [Server and Port Naming](#server-and-port-naming) - - [Testbeds Configuration](#testbeds-configuration) - - [Per Testbed Server Allocation and Naming](#per-testbed-server-allocation-and-naming) - - [1-Node-Skylake (1n-skx) PROD](#1-node-skylake-1n-skx-prod) - - [1-Node-Thunderx2 (1n-tx2) WIP](#1-node-thunderx2-1n-tx2-wip) - - [2-Node-Skylake (2n-skx) PROD](#2-node-skylake-2n-skx-prod) - - [2-Node-Denverton (2n-dnv) TODO](#2-node-denverton-2n-dnv-todo) - - [2-Node-IxiaPS1L47 (2n-ps1) VERIFY](#2-node-ixiaps1l47-2n-ps1-verify) - - [3-Node-Haswell (3n-hsw) PROD](#3-node-haswell-3n-hsw-prod) - - [3-Node-Skylake (3n-skx) PROD](#3-node-skylake-3n-skx-prod) - - [3-Node-Rangeley (3n-rng) VERIFY](#3-node-rangeley-3n-rng-verify) - - [3-Node-Taishan (3n-tsh) WIP](#3-node-taishan-3n-tsh-wip) - - [3-Node-Mcbin (3n-mcb) TODO](#3-node-mcbin-3n-mcb-todo) - - [Per Testbed Wiring](#per-testbed-wiring) - - [1-Node-Skylake (1n-skx) PROD](#1-node-skylake-1n-skx-prod) - - [1-Node-Thunderx2 (1n-tx2) WIP](#1-node-thunderx2-1n-tx2-wip) - - [2-Node-Skylake (2n-skx) PROD](#2-node-skylake-2n-skx-prod) - - [2-Node-Denverton (2n-dnv) TODO](#2-node-denverton-2n-dnv-todo) - - [2-Node-IxiaPS1L47 (2n-ps1) VERIFY](#2-node-ixiaps1l47-2n-ps1-verify) - - [3-Node-Haswell (3n-hsw) PROD](#3-node-haswell-3n-hsw-prod) - - [3-Node-Skylake (3n-skx) PROD](#3-node-skylake-3n-skx-prod) - - [3-Node-Rangeley (3n-rng) TODO](#3-node-rangeley-3n-rng-todo) - - [3-Node-Taishan (3n-tsh) WIP](#3-node-taishan-3n-tsh-wip) - - [3-Node-Mcbin (3n-mcb) WIP](#3-node-mcbin-3n-mcb-wip) - - [Inventory](#inventory) - - [Appliances](#appliances) - - [Arm Servers](#arm-servers) - - [Xeon and Atom Servers](#xeon-and-atom-servers) - - [Network Interface Cards](#network-interface-cards) - - [Pluggables and Cables](#pluggables-and-cables) - - [Other Parts](#other-parts) - - -## FD.io CSIT Testbed Specifications +# FD.io CSIT Testbed Specifications + +1. [Purpose](#purpose) +1. [Testbeds Overview](#testbeds-overview) + 1. [Summary List](#summary-list) + 1. [1-Node-Skylake Xeon Intel (1n-skx)](#1-node-skylake-xeon-intel-1n-skx) + 1. [1-Node-ThunderX2 Arm Marvell (1n-tx2)](#1-node-thunderx2-arm-marvell-1n-tx2) + 1. [2-Node-Skylake Xeon Intel (2n-skx)](#2-node-skylake-xeon-intel-2n-skx) + 1. [2-Node-Denverton Atom Intel (2n-dnv)](#2-node-denverton-atom-intel-2n-dnv) + 1. [2-Node-IxiaPS1L47 Ixia PSOne L47 (2n-ps1)](#2-node-ixiaps1l47-ixia-psone-l47-2n-ps1) + 1. [3-Node-Haswell Xeon Intel (3n-skx)](#3-node-haswell-xeon-intel-3n-skx) + 1. [3-Node-Skylake Xeon Intel (3n-skx)](#3-node-skylake-xeon-intel-3n-skx) + 1. [3-Node-TaiShan Arm Huawei (3n-tsh)](#3-node-taishan-arm-huawei-3n-tsh) + 1. [3-Node-MACCHIATObin Arm Marvell](#3-node-macchiatobin-arm-marvell) + 1. [3-Node-Rangeley Atom Testbeds](#3-node-rangeley-atom-testbeds) +1. [Server Management](#server-management) + 1. [Requirements](#requirements) + 1. [Addressing](#addressing) + 1. [LOM (IPMI) VLAN IP Addresses](#lom-ipmi-vlan-ip-addresses) + 1. [Management VLAN IP Addresses](#management-vlan-ip-addresses) +1. [Server Type Specification](#server-type-specification) + 1. [Server and Port Naming](#server-and-port-naming) +1. [Testbeds Configuration](#testbeds-configuration) + 1. [Per Testbed Server Allocation and Naming](#per-testbed-server-allocation-and-naming) + 1. [1-Node-Skylake (1n-skx) PROD](#1-node-skylake-1n-skx-prod) + 1. [1-Node-Thunderx2 (1n-tx2) WIP](#1-node-thunderx2-1n-tx2-wip) + 1. [2-Node-Skylake (2n-skx) PROD](#2-node-skylake-2n-skx-prod) + 1. [2-Node-Denverton (2n-dnv) TODO](#2-node-denverton-2n-dnv-todo) + 1. [2-Node-IxiaPS1L47 (2n-ps1) VERIFY](#2-node-ixiaps1l47-2n-ps1-verify) + 1. [3-Node-Haswell (3n-hsw) PROD](#3-node-haswell-3n-hsw-prod) + 1. [3-Node-Skylake (3n-skx) PROD](#3-node-skylake-3n-skx-prod) + 1. [3-Node-Rangeley (3n-rng) VERIFY](#3-node-rangeley-3n-rng-verify) + 1. [3-Node-Taishan (3n-tsh) WIP](#3-node-taishan-3n-tsh-wip) + 1. [3-Node-Mcbin (3n-mcb) TODO](#3-node-mcbin-3n-mcb-todo) + 1. [Per Testbed Wiring](#per-testbed-wiring) + 1. [1-Node-Skylake (1n-skx) PROD](#1-node-skylake-1n-skx-prod) + 1. [1-Node-Thunderx2 (1n-tx2) WIP](#1-node-thunderx2-1n-tx2-wip) + 1. [2-Node-Skylake (2n-skx) PROD](#2-node-skylake-2n-skx-prod) + 1. [2-Node-Denverton (2n-dnv) TODO](#2-node-denverton-2n-dnv-todo) + 1. [2-Node-IxiaPS1L47 (2n-ps1) VERIFY](#2-node-ixiaps1l47-2n-ps1-verify) + 1. [3-Node-Haswell (3n-hsw) PROD](#3-node-haswell-3n-hsw-prod) + 1. [3-Node-Skylake (3n-skx) PROD](#3-node-skylake-3n-skx-prod) + 1. [3-Node-Rangeley (3n-rng) TODO](#3-node-rangeley-3n-rng-todo) + 1. [3-Node-Taishan (3n-tsh) WIP](#3-node-taishan-3n-tsh-wip) + 1. [3-Node-Mcbin (3n-mcb) WIP](#3-node-mcbin-3n-mcb-wip) +1. [Inventory](#inventory) + 1. [Appliances](#appliances) + 1. [Arm Servers](#arm-servers) + 1. [Xeon and Atom Servers](#xeon-and-atom-servers) + 1. [Network Interface Cards](#network-interface-cards) + 1. [Pluggables and Cables](#pluggables-and-cables) + 1. [Other Parts](#other-parts) + +## Purpose This note includes specification of the physical testbed infrastructure hosted by LFN FD.io CSIT project. @@ -74,7 +73,8 @@ hosted by LFN FD.io CSIT project. 9. 3-Node-Taishan perf tsh skx 1 2 1 0 .5 0 0 0 0 2 0 10. 3-Node-Mcbin perf mcb skx 1 2 1 0 .5 0 0 0 0 0 2 11. 1-Node-VIRL func hsw --- 3 3 0 3 0 0 0 0 0 0 0 - Totals: 17 26 14 12 19 1 2 1 1 2 2 +12. 3-Node-Denverton perf dnv skx 1 2 1 0 .5 0 0 2 0 0 0 + Totals: 18 28 15 12 19 1 2 3 1 2 2 ``` ### 1-Node-Skylake Xeon Intel (1n-skx) @@ -107,7 +107,11 @@ Each 2-Node-IxiaPS1L47 testbed includes one SUT (Type-B1 server) and one TG (Ixia PSOne appliance) with 10GE interfaces connected back-to-back. Used for FD.io TCP/IP and HTTP performance tests. -### 3-Node-Haswell Xeon Intel (3n-skx) +### 3-Node-Haswell Xeon Intel (3n-hsw) + +Each 3-Node-Haswell testbed includes two SUTs (Type-A1 servers) and one +TG (Type-A2 server) connected in full-mesh triangle. +Used for FD.io performance tests. ### 3-Node-Skylake Xeon Intel (3n-skx) @@ -403,7 +407,7 @@ FD.io CSIT lab contains following server types: - Disks: 1* 4TB SATA HDD - NICs configuration: - PCIe Slot4 e9:00.xx: connectx4-2p25GE Mellanox. - - PCIe Slot6 11:00.xx: 82599-2p10GE Intel. + - PCIe Slot6 11:00.xx: x520-2p10GE Intel. 4. Type-B4: Purpose - MACCHIATObin Arm Marvell SUT for FD.io performance testing. - Quantity: 3 - Physical connectivity: @@ -713,10 +717,39 @@ connectivity and wiring across defined CSIT testbeds: - s10-t24-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE. ``` -#### 2-Node-Denverton (2n-dnv) TODO +#### 2-Node-Denverton (2n-dnv) WIP + +Note: ServerB28 (TG) is shared between testbed26 & testbed35 ``` -To be completed. +- ServerB28 [Type-B2]: + - testbednames: testbed26 and testbed35. + - hostname: s28-t26t35-tg1. + - IPMI IP: 10.30.55.10 + - Host IP: 10.32.8.10 + - portnames: + - s28-t26t35-tg1-c2/p1 - 10GE-port1 x710da2-2p10GE. + - s28-t26t35-tg1-c2/p2 - 10GE-port2 x710da2-2p10GE. + - s28-t26t35-tg1-c4/p1 - 10GE-port1 x550t2-2p10GE. + - s28-t26t35-tg1-c4/p2 - 10GE-port2 x550t2-2p10GE. + - s28-t26t35-tg1-c9/p1 - 10GE-port1 x550t2-2p10GE. + - s28-t26t35-tg1-c9/p2 - 10GE-port2 x550t2-2p10GE. + - s28-t26t35-tg1-c6/p1 - 10GE-port1 x710da2-2p10GE. + - s28-t26t35-tg1-c6/p2 - 10GE-port2 x710da2-2p10GE. + - s28-t26t35-tg1-c8/p1 - 10GE-port1 x550t2-2p10GE. + - s28-t26t35-tg1-c8/p2 - 10GE-port2 x550t2-2p10GE. + - s28-t26t35-tg1-c10/p1 - 10GE-port1 x550t2-2p10GE. + - s28-t26t35-tg1-c10/p2 - 10GE-port2 x550t2-2p10GE. +- ServerB29 [Type-B10]: + - testbednames: testbed26. + - hostname: s29-t26-sut1. + - IPMI IP: 10.30.55.11 + - Host IP: 10.32.8.11 + - portnames: + - s29-t26-sut1-p1 - 10GE-port1 x553 copper port. + - s29-t26-sut1-p2 - 10GE-port2 x553 copper port. + - s29-t26-sut1-p3 - 10GE-port3 x553 fiber port. + - s29-t26-sut1-p4 - 10GE-port4 x553 fiber port. ``` #### 2-Node-IxiaPS1L47 (2n-ps1) VERIFY @@ -988,8 +1021,6 @@ To be completed. - s16-t32-tg1-c9/p2 - FUTURE 100GE-port2 ConnectX5-2p100GE. ``` -Note: ServerB19 (TG) is shared between testbed33 & testbed34 - #### 3-Node-Rangeley (3n-rng) VERIFY Note: There is no IPMI. Serial console is accessible via VIRL2 and VIRL3 USB. @@ -1026,6 +1057,8 @@ Note: There is no IPMI. Serial console is accessible via VIRL2 and VIRL3 USB. #### 3-Node-Taishan (3n-tsh) WIP +Note: ServerB19 (TG) is shared between testbed33 & testbed34 + ``` - ServerB17 [Type-B3]: - testbedname: testbed33. @@ -1033,8 +1066,8 @@ Note: There is no IPMI. Serial console is accessible via VIRL2 and VIRL3 USB. - IPMI IP: 10.30.50.36 - Host IP: 10.30.51.36 - portnames: - - s17-t33-sut1-c6/p1 - 10GE-port1 82599-2p10GE. - - s17-t33-sut1-c6/p2 - 10GE-port2 82599-2p10GE. + - s17-t33-sut1-c6/p1 - 10GE-port1 x520-2p10GE. + - s17-t33-sut1-c6/p2 - 10GE-port2 x520-2p10GE. - s17-t33-sut1-c4/p1 - 25GE-port1 cx4-2p25GE. - s17-t33-sut1-c4/p2 - 25GE-port2 cx4-2p25GE. - ServerB18 [Type-B3]: @@ -1043,8 +1076,8 @@ Note: There is no IPMI. Serial console is accessible via VIRL2 and VIRL3 USB. - IPMI IP: 10.30.50.37 - Host IP: 10.30.51.37 - portnames: - - s18-t33-sut2-c6/p1 - 10GE-port1 82599-2p10GE. - - s18-t33-sut2-c6/p2 - 10GE-port2 82599-2p10GE. + - s18-t33-sut2-c6/p1 - 10GE-port1 x520-2p10GE. + - s18-t33-sut2-c6/p2 - 10GE-port2 x520-2p10GE. - s18-t33-sut2-c4/p1 - 25GE-port1 cx4-2p25GE. - s18-t33-sut2-c4/p2 - 25GE-port2 cx4-2p25GE. - ServerB19 [Type-B2]: @@ -1081,6 +1114,31 @@ Note: There is no IPMI. Serial console is accessible via VIRL2 and VIRL3 USB. - s21-t34-sut2-ca/p2 - 10GE-port2 Marvell. ``` +#### 3-Node-Denverton (3n-dnv) WIP + +``` +- ServerB30 [Type-B10]: + - testbednames: testbed35. + - hostname: s30-t35-sut1. + - IPMI IP: 10.30.55.12 + - Host IP: 10.32.8.12 + - portnames: + - s30-t35-sut1-p1 - 10GE-port1 x553 copper port. + - s30-t35-sut1-p2 - 10GE-port2 x553 copper port. + - s30-t35-sut1-p3 - 10GE-port3 x553 fiber port. + - s30-t35-sut1-p4 - 10GE-port4 x553 fiber port. +- ServerB31 [Type-B10]: + - testbednames: testbed35. + - hostname: s31-t35-sut2. + - IPMI IP: 10.30.55.13 + - Host IP: 10.32.8.13 + - portnames: + - s31-t35-sut2-p1 - 10GE-port1 x553 copper port. + - s31-t35-sut2-p2 - 10GE-port2 x553 copper port. + - s31-t35-sut2-p3 - 10GE-port3 x553 fiber port. + - s31-t35-sut2-p4 - 10GE-port4 x553 fiber port. +``` + #### 3-Node-Mcbin (3n-mcb) TODO ``` @@ -1193,10 +1251,16 @@ To be completed. - s10-t24-tg1-c10/p3 to s10-t24-tg1-c10/p4. ``` -#### 2-Node-Denverton (2n-dnv) TODO +#### 2-Node-Denverton (2n-dnv) WIP ``` -To be completed. +- testbed26: + - ring1 10GE-ports x553 copper port on SUT: + - s28-t26t35-tg1-c4/p1 to s29-t26-sut1-p1. + - s28-t26t35-tg1-c9/p1 to s29-t26-sut1-p2. + - ring2 10GE-ports x553 fiber port on SUT: + - s28-t26t35-tg1-c2/p1 to s29-t26-sut1-p3. + - s28-t26t35-tg1-c2/p2 to s29-t26-sut1-p4. ``` #### 2-Node-IxiaPS1L47 (2n-ps1) VERIFY @@ -1344,7 +1408,7 @@ To be completed. ``` - testbed33: - - ring1 10GE-ports 82599-2p10GE on SUTs: + - ring1 10GE-ports x520-2p10GE on SUTs: - t33t34-tg1-c2/p2 - t33-sut1-c6/p2. - t33-sut1-c6/p1 - t33-sut2-c6/p2. - t33-sut2-c6/p1 - t33t34-tg1-c2/p1. @@ -1364,6 +1428,20 @@ To be completed. - t34-sut2-ca/p2 - t33t34-tg1-c2/p4. ``` +#### 3-Node-Denverton (3n-dnv) WIP + +``` +- testbed35: + - ring1 10GE-ports x553 copper port on SUTs: + - s28-t26t35-tg1-c8/p1 to s30-t35-sut1-p2. + - s30-t35-sut1-p1 to s31-t35-sut2-p1. + - s28-t26t35-tg1-c10/p1 to s31-t35-sut2-p2. + - ring2 10GE-ports x553 fiber port on SUTs: + - s28-t26t35-tg1-c6/p1 to s30-t35-sut1-p4. + - s30-t35-sut1-p3 to s31-t35-sut2-p3. + - s28-t26t35-tg1-c6/p2 to s31-t35-sut2-p4. +``` + ## Inventory ### Appliances