5 This directoctory contains the *high-level* process to set up a hardware
6 machine as a CSIT testbed, either for use as a physical testbed host or
9 Code in this directory is NOT executed as part of a regular CSIT test case
10 but is stored here merely for archiving and documentation purposes.
13 ## Setting up a hardware host
15 Documentation below is just bullet points and assumes and understanding
16 of PXE boot and ansible.
18 This process is specific for LF lab, and both examples given here as
19 well as associated code, are based on the assumption that they are run
20 in LF environment. If run elsewhere, changes will be required to IP addresses
23 The process below assumes that there is a host used for boostrapping (referred
24 to as "PXE boostrap server" below), and that the directory containig this README
25 is available on the PXE bootstrap server in ~testuser/host-setup.
27 ### Prepare the PXE bootstrap server when there is no http server
29 - `sudo apt-get install isc-dhcp-server tftpd-hpa nginx-light ansible`
30 - edit dhcpd.conf and place it to /etc/dhcp/
31 - `sudo cp dhcpd.cfg /etc/dhcp/`
32 - `sudo service isc-dhcp-server restart`
33 - `cd ~testuser/host-setup`
34 - `wget 'http://releases.ubuntu.com/16.04.2/ubuntu-16.04.2-server-amd64.iso'`
35 - `sudo mkdir /mnt/cdrom`
36 - `sudo mount -o loop ubuntu-16.04.2-server-amd64.iso /mnt/cdrom/`
37 - `sudo cp -r /mnt/cdrom/install/netboot/* /var/lib/tftpboot/`
38 - figure out where nginx will look for files on the filesystem when
39 responding to HTTP requests. The configuration is in one of the
40 files in /etc/nginx/conf.d/, /etc/nginx/sites-enabled/ or in
41 /etc/nginx/nginx.conf under section server/root. Save the path to NGINX_ROOT
42 - `sudo mkdir -p ${NGINX_ROOT}/download/ubuntu`
43 - `sudo cp -r /mnt/cdrom/* ${NGINX_ROOT}/download/ubuntu/`
44 - `sudo cp /mnt/cdrom/ubuntu/isolinux/ldlinux.c32 /var/lib/tftpboot`
45 - `sudo cp /mnt/cdrom/ubuntu/isolinux/libcom32.c32 /var/lib/tftpboot`
46 - `sudo cp /mnt/cdrom/ubuntu/isolinux/libutil.c32 /var/lib/tftpboot`
47 - `sudo cp /mnt/cdrom/ubuntu/isolinux/chain.c32 /var/lib/tftpboot`
48 - `sudo umount /mnt/cdrom`
49 - edit ks.cfg and replace IP address with that of your PXE bootstrap server
50 - `sudo cp ks.cfg ${NGINX_ROOT}/download/ks.cfg`
51 - edit boot-screens_txt.cfg and replace IP address with that of your PXE bootstrap server
52 - `sudo cp boot-screens_txt.cfg /var/lib/tftpboot/ubuntu-installer/amd64/boot-screens/txt.cfg`
53 - `sudo cp syslinux.cfg /var/lib/tftpboot/ubuntu-installer/amd64/boot-screens/syslinux.cfg`
55 ### PREFERED: Prepare the PXE bootstrap server when an http server is already configured
57 - `sudo apt-get install isc-dhcp-server tftpd-hpa ansible`
58 - edit dhcpd.conf and place it to /etc/dhcp/
59 - `sudo cp dhcpd.cfg /etc/dhcp/`
60 - `sudo service isc-dhcp-server restart`
61 - `cd ~testuser/host-setup`
62 - `wget 'http://releases.ubuntu.com/16.04.2/ubuntu-16.04.2-server-amd64.iso'`
63 - `sudo mkdir /mnt/cdrom`
64 - `sudo mount -o loop ubuntu-16.04.1-server-amd64.iso /mnt/cdrom/`
65 - `sudo cp -r /mnt/cdrom/install/netboot/* /var/lib/tftpboot/`
66 - `sudo mkdir /var/www/download/ubuntu`
67 - `sudo cp -r /mnt/cdrom/* /var/www/download/ubuntu/`
68 - `sudo cp /mnt/cdrom/ubuntu/isolinux/ldlinux.c32 /var/lib/tftpboot`
69 - `sudo cp /mnt/cdrom/ubuntu/isolinux/libcom32.c32 /var/lib/tftpboot`
70 - `sudo cp /mnt/cdrom/ubuntu/isolinux/libutil.c32 /var/lib/tftpboot`
71 - `sudo cp /mnt/cdrom/ubuntu/isolinux/chain.c32 /var/lib/tftpboot`
72 - `sudo umount /mnt/cdrom`
73 - edit ks.cfg and replace IP address with that of your PXE bootstrap server and subdir in /var/www (in this case /download)
74 - `sudo cp ks.cfg /var/www/download/ks.cfg`
75 - edit boot-screens_txt.cfg and replace IP address with that of your PXE bootstrap server and subdir in /var/www (in this case /download)
76 - `sudo cp boot-screens_txt.cfg /var/lib/tftpboot/ubuntu-installer/amd64/boot-screens/txt.cfg`
77 - `sudo cp syslinux.cfg /var/lib/tftpboot/ubuntu-installer/amd64/boot-screens/syslinux.cfg`
79 ### New testbed host - manual preparation
82 - set CIMC username, password and hostname
84 ### Bootstrap the host
86 From PXE boostrap server:
88 - `cd ~testuser/host-setup/cimc`
89 - Initialize args.ip: Power-Off, reset BIOS defaults, Enable console redir, get LOM MAC addr
90 - `./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -i`
91 - Adjust BIOS settings
92 - `./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -s '<biosVfIntelHyperThreadingTech rn="Intel-HyperThreading-Tech" vpIntelHyperThreadingTech="disabled" />' -s '<biosVfEnhancedIntelSpeedStepTech rn="Enhanced-Intel-SpeedStep-Tech" vpEnhancedIntelSpeedStepTech="disabled" />' -s '<biosVfIntelTurboBoostTech rn="Intel-Turbo-Boost-Tech" vpIntelTurboBoostTech="disabled" />'`
93 - add MAC address to DHCP (/etc/dhcp/dhcpd.conf)
94 - Reboot server with boot from PXE (restart immediately)
95 - `./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -pxe`
97 While Ubuntu install is running:
99 - create RAID array. Reboot if needed.
100 - `./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d --wipe`
101 - `./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -r -rl 1 -rs <disk size> -rd '[1,2]'`
102 Alternatively, create the RAID array manually.
104 - Set the next boot from HDD (without restart)
105 - `./cimc.py -u admin -p Cisco1234 $CIMC_ADDRESS -d -hdd`
107 When installation is finished:
109 - `ssh-copy-id 10.30.51.x`
110 - `cd ~testuser/host-setup/playbooks`
111 - edit /etc/ansible/hosts; add the hosts you are installing. *REMOVE ANY HOSTS YOU ARE NOT CURRENTLY INSTALLING*.
113 Example for physical testbed hosts:
116 10.30.51.24 hostname=t3-tg1 isolcpus="1-17,19-35" ansible_python_interpreter=/usr/bin/python2.7
119 10.30.51.25 hostname=t3-sut1 isolcpus="1-17,19-35" ansible_python_interpreter=/usr/bin/python2.7
120 10.30.51.26 hostname=t3-sut2 isolcpus="1-17,19-35" ansible_python_interpreter=/usr/bin/python2.7
123 Example for VIRL hosts -- use the "virl" tag and specify the flat network start and end addresses:
127 10.30.51.28 hostname=t4-virl1 virl_l2_start=10.30.52.2 virl_l2_end=10.30.52.253 virl_l2_network=10.30.52.0/24 virl_l2_ip=10.30.52.254 virl_public_port=eth0
130 - `ansible-playbook --ask-sudo-pass 01-host-setup.yaml`
131 - `ansible-playbook reboot.yaml`
133 For non-VIRL hosts, stop here.
136 ### VIRL installation
138 After the host has rebooted:
140 - `ansible-playbook 02-virl-bootstrap.yaml`
143 - `cd virl-bootstrap`
144 - `./virl-bootstrap-wrapper`
146 This command will error out when run the first time, as the VIRL host is not yet licensed.
148 Make sure we contact all three VIRL SALT masters:
150 - `for a in 1 2 4 ; do sudo salt-call --master us-${a}.virl.info test.ping ; done`
152 - Contact the VIRL team, provide the hostname and domain (linuxfoundation.org), and ask them
155 - After the key has been accepted, verify that connectivity with the SALT master is now OK:
157 `for a in 1 2 4 ; do sudo salt-call --master us-${a}.virl.info test.ping ; done`
159 - `./virl-bootstrap-wrapper`
162 After reboot, ssh to host again
163 - as VIRL user, NOT AS ROOT:
167 After reboot, ssh to host again
169 - `sudo salt-call state.sls virl.routervms.all`
170 - `sudo salt-call state.sls virl.vmm.vmmall`
172 Back on the PXE bootstrap server:
174 - obtain the current server disk image and place it into
175 `files/virl-server-image/` as `server.qcow2`
177 TO-DO: Need to find a place to store this image
179 - `ansible-playbook 03-virl-post-install.yaml`
181 - Run the following command ONLY ONCE. Otherwise it will create
182 duplicates of the VIRL disk image:
184 `ansible-playbook 04-disk-image.yaml`
186 The VIRL host should now be operational. Test, and when ready, create a ~jenkins-in/status file with the appropriate status.