7 This directory contains the *high-level* process to set up a hardware machine
8 as a CSIT testbed, either for use as a physical performance testbed host or as
11 Code in this directory is NOT executed as part of a regular CSIT test case
12 but is stored here for ad-hoc installation of HW, archiving and documentation
15 Setting up a hardware host
16 --------------------------
18 Documentation below is step by step tutorial and assumes an understanding of PXE
19 boot and Ansible and managing physical hardware via CIMC or IPMI.
21 This process is not specific for LF lab, but associated files and code, is based
22 on the assumption that it runs in LF environment. If run elsewhere, changes
23 will be required in following files:
25 #. Inventory directory: `ansible/inventories/sample_inventory/`
26 #. Inventory files: `ansible/inventories/sample_inventory/hosts`
27 #. Kickseed file: `pxe/ks.cfg`
28 #. DHCPD file: `pxe/dhcpd.conf`
29 #. Bootscreen file: `boot-screens_txt.cfg`
31 The process below assumes that there is a host used for bootstrapping (referred
32 to as "PXE bootstrap server" below).
34 Prepare the PXE bootstrap server when there is no http server AMD64
35 ```````````````````````````````````````````````````````````````````
37 #. Clone the csit repo:
41 git clone https://gerrit.fd.io/r/csit
42 cd csit/resources/tools/testbed-setup/pxe
44 #. Setup prerequisities (isc-dhcp-server tftpd-hpa nginx-light ansible):
48 sudo apt-get install isc-dhcp-server tftpd-hpa nginx-light ansible
54 sudo cp dhcpd.cfg /etc/dhcp/
55 sudo service isc-dhcp-server restart
58 #. Download Ubuntu 18.04 LTS - X86_64:
62 wget http://cdimage.ubuntu.com/ubuntu/releases/18.04/release/ubuntu-18.04-server-amd64.iso
63 sudo mount -o loop ubuntu-18.04-server-amd64.iso /mnt/cdrom/
64 sudo cp -r /mnt/cdrom/install/netboot/* /var/lib/tftpboot/
66 # Figure out root folder for NGINX webserver. The configuration is in one
67 # of the files in /etc/nginx/conf.d/, /etc/nginx/sites-enabled/ or in
68 # /etc/nginx/nginx.conf under section server/root. Save the path to
70 sudo mkdir -p ${WWW_ROOT}/download/ubuntu
71 sudo cp -r /mnt/cdrom/* ${WWW_ROOT}/download/ubuntu/
72 sudo cp /mnt/cdrom/ubuntu/isolinux/ldlinux.c32 /var/lib/tftpboot
73 sudo cp /mnt/cdrom/ubuntu/isolinux/libcom32.c32 /var/lib/tftpboot
74 sudo cp /mnt/cdrom/ubuntu/isolinux/libutil.c32 /var/lib/tftpboot
75 sudo cp /mnt/cdrom/ubuntu/isolinux/chain.c32 /var/lib/tftpboot
76 sudo umount /mnt/cdrom
78 #. Edit ks.cfg and replace IP address of PXE bootstrap server and subdir in
79 `/var/www` (in this case `/var/www/download`):
83 sudo cp ks.cfg ${WWW_ROOT}/download/ks.cfg
85 #. Edit boot-screens_txt.cfg and replace IP address of PXE bootstrap server and
86 subdir in `/var/www` (in this case `/var/www/download`):
90 sudo cp boot-screens_txt.cfg /var/lib/tftpboot/ubuntu-installer/amd64/boot-screens/txt.cfg
91 sudo cp syslinux.cfg /var/lib/tftpboot/ubuntu-installer/amd64/boot-screens/syslinux.cfg
93 New testbed host - manual preparation
94 `````````````````````````````````````
96 Set CIMC/IPMI address, username, password and hostname an BIOS.
101 Convenient way to re-stage host via script:
105 sudo ./bootstrap_setup_testbed.sh <linux_ip> <mgmt_ip> <username> <pass>
107 Optional: CIMC - From PXE boostrap server
108 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
110 #. Initialize args.ip: Power-Off, reset BIOS defaults, Enable console redir, get
115 ./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -i
117 #. Adjust BIOS settings:
121 ./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -s '<biosVfIntelHyperThreadingTech rn="Intel-HyperThreading-Tech" vpIntelHyperThreadingTech="disabled" />' -s '<biosVfEnhancedIntelSpeedStepTech rn="Enhanced-Intel-SpeedStep-Tech" vpEnhancedIntelSpeedStepTech="disabled" />' -s '<biosVfIntelTurboBoostTech rn="Intel-Turbo-Boost-Tech" vpIntelTurboBoostTech="disabled" />'
123 #. If RAID is not created in CIMC. Create RAID array. Reboot:
127 ./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d --wipe
128 ./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -r -rl 1 -rs <disk size> -rd '[1,2]'
130 #. Reboot server with boot from PXE (restart immediately):
134 ./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -pxe
136 #. Set the next boot from HDD (without restart). Execute while Ubuntu install
141 ./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -hdd
143 Optional: IPMI - From PXE boostrap server
144 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
146 #. Get MAC address of LAN0:
150 ipmitool -U ADMIN -H $HOST_ADDRESS raw 0x30 0x21 | tail -c 18
152 #. Reboot into PXE for next boot only:
156 ipmitool -I lanplus -H $HOST_ADDRESS -U ADMIN chassis bootdev pxe
157 ipmitool -I lanplus -H $HOST_ADDRESS -U ADMIN power reset
159 #. For live watching SOL (Serial-over-LAN console):
163 ipmitool -I lanplus -H $HOST_ADDRESS -U ADMIN sol activate
164 ipmitool -I lanplus -H $HOST_ADDRESS -U ADMIN sol deactivate
169 Prerequisities for running Ansible
170 ..................................
172 - Ansible can run on any machine that has direct SSH connectivity to target
173 machines that will be provisioned (does not need to be PXE server).
174 - User `testuser` with password `Csit1234` is created with home folder
175 initialized on all target machines that will be provisioned.
176 - SSH keys for no pass access are copied to all target machines that will be
177 provisioned: `ssh-copy-id x.x.x.x`.
178 - Inventory directory is created with same or similar content as
179 `inventories/lf_inventory` in `inventories/` directory (`sample_inventory`
181 - Group variables in `ansible/inventories/<inventory>/group_vars/all.yaml` are
182 adjusted per environment. Special attention to `proxy_env` variable.
183 - Host variables in `ansible/inventories/<inventory>/host_vars/x.x.x.x.yaml` are
189 Ansible is defining roles `TG` (Traffic Generator), `SUT` (System Under Test),
190 `VPP_DEVICE` (vpp_device host for functional testing).
192 Each Host has corresponding Ansible role mapped and is applied only if Host
193 with that role is present in inventory file. As a part of optimization the role
194 `common` contains Ansible tasks applied for all Hosts.
198 You may see `[WARNING]: Could not match supplied host pattern, ignoring:
199 <role>` in case you have not define hosts for that particular role.
201 Ansible structure is described below:
206 ├── inventories # Contains all inventories.
207 │ ├── sample_inventory # Sample, free for edits outside of LF.
208 │ │ ├── group_vars # Variables applied for all hosts.
210 │ │ ├── hosts # Inventory list with sample hosts.
211 │ │ └── host_vars # Variables applied for single host only.
212 │ │ └── 1.1.1.1.yaml # Sample host with IP 1.1.1.1
213 │ └── lf_inventory # Linux Foundation inventory.
218 ├── roles # CSIT roles.
219 │ ├── common # Role applied for all hosts.
220 │ ├── sut # Role applied for all SUTs only.
221 │ ├── tg # Role applied for all TGs only.
222 │ ├── tg_sut # Role applied for TGs and SUTs only.
223 │ └── vpp_device # Role applied for vpp_device only.
224 ├── site.yaml # Main playbook.
225 ├── sut.yaml # SUT playbook.
226 ├── tg.yaml # TG playbook.
227 ├── vault_pass # Main password for vualt.
228 ├── vault.yml # Ansible vualt storage.
229 └── vpp_device.yaml # vpp_device playbook.
234 #. Go to ansible directory: `cd csit/resources/tools/testbed-setup/ansible`
235 #. Run ansible on selected hosts:
236 `ansible-playbook --vault-id vault_pass --extra-vars '@vault.yml' --inventory <inventory_file> site.yaml --limit x.x.x.x`
240 In case you want to provision only particular role. You can use tags: `tg`,
246 Manually reboot hosts after Ansible provisioning succeeded.