Testbed Setup ============= Introduction ------------ This directory contains the *high-level* process to set up a hardware machine as a CSIT testbed, either for use as a physical performance testbed host or as a vpp_device host. Code in this directory is NOT executed as part of a regular CSIT test case but is stored here for ad-hoc installation of HW, archiving and documentation purposes. Setting up a hardware host -------------------------- Documentation below is step by step tutorial and assumes an understanding of PXE boot and Ansible and managing physical hardware via CIMC or IPMI. This process is not specific for LF lab, but associated files and code, is based on the assumption that it runs in LF environment. If run elsewhere, changes will be required in following files: #. Inventory directory: `ansible/inventories/sample_inventory/` #. Inventory files: `ansible/inventories/sample_inventory/hosts` #. Kickseed file: `pxe/ks.cfg` #. DHCPD file: `pxe/dhcpd.conf` #. Bootscreen file: `boot-screens_txt.cfg` The process below assumes that there is a host used for bootstrapping (referred to as "PXE bootstrap server" below). Prepare the PXE bootstrap server when there is no http server AMD64 ``````````````````````````````````````````````````````````````````` #. Clone the csit repo: .. code-block:: bash git clone https://gerrit.fd.io/r/csit cd csit/resources/tools/testbed-setup/pxe #. Setup prerequisities (isc-dhcp-server tftpd-hpa nginx-light ansible): .. code-block:: bash sudo apt-get install isc-dhcp-server tftpd-hpa nginx-light ansible #. Edit dhcpd.cfg: .. code-block:: bash sudo cp dhcpd.cfg /etc/dhcp/ sudo service isc-dhcp-server restart sudo mkdir /mnt/cdrom #. Download Ubuntu 18.04 LTS - X86_64: .. code-block:: bash wget http://cdimage.ubuntu.com/ubuntu/releases/18.04/release/ubuntu-18.04-server-amd64.iso sudo mount -o loop ubuntu-18.04-server-amd64.iso /mnt/cdrom/ sudo cp -r /mnt/cdrom/install/netboot/* /var/lib/tftpboot/ # Figure out root folder for NGINX webserver. The configuration is in one # of the files in /etc/nginx/conf.d/, /etc/nginx/sites-enabled/ or in # /etc/nginx/nginx.conf under section server/root. Save the path to # variable WWW_ROOT. sudo mkdir -p ${WWW_ROOT}/download/ubuntu sudo cp -r /mnt/cdrom/* ${WWW_ROOT}/download/ubuntu/ sudo cp /mnt/cdrom/ubuntu/isolinux/ldlinux.c32 /var/lib/tftpboot sudo cp /mnt/cdrom/ubuntu/isolinux/libcom32.c32 /var/lib/tftpboot sudo cp /mnt/cdrom/ubuntu/isolinux/libutil.c32 /var/lib/tftpboot sudo cp /mnt/cdrom/ubuntu/isolinux/chain.c32 /var/lib/tftpboot sudo umount /mnt/cdrom #. Edit ks.cfg and replace IP address of PXE bootstrap server and subdir in `/var/www` (in this case `/var/www/download`): .. code-block:: bash sudo cp ks.cfg ${WWW_ROOT}/download/ks.cfg #. Edit boot-screens_txt.cfg and replace IP address of PXE bootstrap server and subdir in `/var/www` (in this case `/var/www/download`): .. code-block:: bash sudo cp boot-screens_txt.cfg /var/lib/tftpboot/ubuntu-installer/amd64/boot-screens/txt.cfg sudo cp syslinux.cfg /var/lib/tftpboot/ubuntu-installer/amd64/boot-screens/syslinux.cfg New testbed host - manual preparation ````````````````````````````````````` Set CIMC/IPMI address, username, password and hostname an BIOS. Bootstrap the host `````````````````` Convenient way to re-stage host via script: .. code-block:: bash sudo ./bootstrap_setup_testbed.sh Optional: CIMC - From PXE boostrap server ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #. Initialize args.ip: Power-Off, reset BIOS defaults, Enable console redir, get LOM MAC addr: .. code-block:: bash ./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -i #. Adjust BIOS settings: .. code-block:: bash ./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -s '' -s '' -s '' #. If RAID is not created in CIMC. Create RAID array. Reboot: .. code-block:: bash ./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d --wipe ./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -r -rl 1 -rs -rd '[1,2]' #. Reboot server with boot from PXE (restart immediately): .. code-block:: bash ./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -pxe #. Set the next boot from HDD (without restart). Execute while Ubuntu install is running: .. code-block:: bash ./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -hdd Optional: IPMI - From PXE boostrap server ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #. Get MAC address of LAN0: .. code-block:: bash ipmitool -U ADMIN -H $HOST_ADDRESS raw 0x30 0x21 | tail -c 18 #. Reboot into PXE for next boot only: .. code-block:: bash ipmitool -I lanplus -H $HOST_ADDRESS -U ADMIN chassis bootdev pxe ipmitool -I lanplus -H $HOST_ADDRESS -U ADMIN power reset #. For live watching SOL (Serial-over-LAN console): .. code-block:: bash ipmitool -I lanplus -H $HOST_ADDRESS -U ADMIN sol activate ipmitool -I lanplus -H $HOST_ADDRESS -U ADMIN sol deactivate Ansible machine ~~~~~~~~~~~~~~~ Prerequisities for running Ansible .................................. - Ansible can run on any machine that has direct SSH connectivity to target machines that will be provisioned (does not need to be PXE server). - User `testuser` with password `Csit1234` is created with home folder initialized on all target machines that will be provisioned. - Inventory directory is created with same or similar content as `inventories/lf_inventory` in `inventories/` directory (`sample_inventory` can be used). - Group variables in `ansible/inventories//group_vars/all.yaml` are adjusted per environment. Special attention to `proxy_env` variable. - Host variables in `ansible/inventories//host_vars/x.x.x.x.yaml` are defined. Ansible structure ................. Ansible is defining roles `TG` (Traffic Generator), `SUT` (System Under Test), `VPP_DEVICE` (vpp_device host for functional testing). `COMMON` (Applicable for all servers in inventory). Each Host has corresponding Ansible role mapped and is applied only if Host with that role is present in inventory file. As a part of optimization the role `common` contains Ansible tasks applied for all Hosts. .. note:: You may see `[WARNING]: Could not match supplied host pattern, ignoring: ` in case you have not define hosts for that particular role. Ansible structure is described below: .. code-block:: bash . ├── inventories # Contains all inventories. │   ├── sample_inventory # Sample, free for edits outside of LF. │   │   ├── group_vars # Variables applied for all hosts. │   │   │   └── all.yaml │   │   ├── hosts # Inventory list with sample hosts. │   │   └── host_vars # Variables applied for single host only. │   │   └── 1.1.1.1.yaml # Sample host with IP 1.1.1.1 │   └── lf_inventory # Linux Foundation inventory. │   ├── group_vars │   │   └── all.yaml │   ├── hosts │   └── host_vars ├── roles # CSIT roles. │   ├── common # Role applied for all hosts. │   ├── sut # Role applied for all SUTs only. │   ├── tg # Role applied for all TGs only. │   ├── tg_sut # Role applied for TGs and SUTs only. │   └── vpp_device # Role applied for vpp_device only. ├── site.yaml # Main playbook. ├── sut.yaml # SUT playbook. ├── tg.yaml # TG playbook. ├── vault_pass # Main password for vualt. ├── vault.yml # Ansible vualt storage. └── vpp_device.yaml # vpp_device playbook. Tagging ....... Every task, handler, role, playbook is tagged with self-explanatory tags that could be used to limit which objects are applied to target systems. You can see which tags are applied to tasks, roles, and static imports by running `ansible-playbook` with the `--list-tasks` option. You can display all tags applied to the tasks with the `--list-tags` option. Running Ansible ............... #. Go to ansible directory: `cd csit/resources/tools/testbed-setup/ansible` #. Run ansible on selected hosts: `ansible-playbook --vault-password-file=vault_pass --extra-vars '@vault.yml' --inventory site.yaml --limit x.x.x.x` .. note:: In case you want to provision only particular role. You can use tags: `tg`, `sut`, `vpp_device`. Reboot hosts ------------ Manually reboot hosts after Ansible provisioning succeeded.