3 # Copyright (c) 2016 Cisco and/or its affiliates.
4 # Licensed under the Apache License, Version 2.0 (the "License");
5 # you may not use this file except in compliance with the License.
6 # You may obtain a copy of the License at:
8 # http://www.apache.org/licenses/LICENSE-2.0
10 # Unless required by applicable law or agreed to in writing, software
11 # distributed under the License is distributed on an "AS IS" BASIS,
12 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 # See the License for the specific language governing permissions and
14 # limitations under the License.
16 # Purpose of this script is to build a VirtualBox or QCOW2 disk image
17 # for use with CSIT testing.
19 # As input parameters, the script takes:
21 # - A base Linux distribution (currently, only "ubuntu-14.04.4" is
23 # - A release timestamp in the format "YYYY-MM-DD" eg. "2016-05-21".
24 # This timestamp MUST reference to a list of packages (APT, PIP)
25 # that was previously generated by the "listmaker" VM and script.
26 # - A path to the nested VM image.
28 # The bullk of the work is done by packer,
29 # while this script does some of the pre- and post-processing.
32 # 1.) Determine if building a QCOW or VirtualBox image. Currently,
33 # we build a QCOW image if VIRL_* environment variables are
34 # present (and we therefore assume we are building for VIRL),
35 # or VirtualBox otherwise.
37 # 2.) Download packer, if not already installed.
39 # 3.) Download APT and PIP packages as required for the image.
40 # We're downloading them here, rather than letting the VM
41 # download them off the public internet, for two reasons:
42 # a.) This allows us to keep and cache packets between runs
43 # and download them only once,
44 # b.) This means only the build host needs a working proxy
45 # configuration and we do not need to worry about setting
46 # a proxy inside the VM.
48 # 4.) Link the nested VM image into the main VM's temp directory
50 # 5.) Create Version and Changelog files
52 # 6.) Run Packer. Packer, in turn, will:
53 # 6.1.) Download the Operating System ISO image,
54 # 6.2.) Start a VM using the selected hypervisor (VirtualBox or Qemu),
55 # 6.3.) Boot the VM using the ISO image and send initial keystrokes
56 # to initiate installation using a Preseed or Kickstart file,
57 # 6.4.) Drive the installation using until the point the VM is reachable
59 # 6.5.) Copy the temporary directory populated in steps 3-5 above to the VM,
60 # 6.6.) Run a script on the VM that performs post-installation:
61 # 6.6.1.) Install selected .deb packages
62 # 6.6.2.) Install PIP packages as per requirements.txt file
63 # 6.6.3.) Install nested VM image and VERSION/CHANGELOG files
64 # 6.7.) Run a script on VM that creates a Vagrant user (VirtualBox only)
65 # 6.8.) Run a script on VM that performs clean-up:
66 # 6.8.1.) Remove any temporary files created,
67 # 6.8.2.) Remove SSH host-keys
68 # 6.8.3.) Remove root user password
69 # 6.8.4.) Shut down the VM
70 # 6.9.) [TODO]: Upload the image to VIRL (QCOW only), -or-
71 # Convert the image into a Vagrant box (VirtualBox only), and
72 # [TODO/FIX]: Upload the Vagrant box to Atlas (VirtualBox only)
77 ### 0. Set constants and verify parameters.
80 BUILD_DIR="$(pwd)/build"
81 PACKER_DIR="${BUILD_DIR}/packer"
83 RPM_CACHE_DIR="${BUILD_DIR}/cache/rpm"
84 PIP_CACHE_DIR="${BUILD_DIR}/cache/pip"
86 PACKER_TEMPLATE="$1.json"
87 LISTS_DIR="$(dirname $0)/lists"
90 echo 'Syntax: $0 <Operating System> <Release> <Nested VM image>'
92 echo '<Operating System>: Base distro, eg. ubuntu-14.04.4'
93 echo '<Release>: Release timestamp, eg. 2016-05-21'
94 echo '<Nested VM image>: Path to nested VM image'
99 ## Parse command line options
110 ## Identify version by looking at topmost version statement in CHANGELOG
112 VERSION=$(cat $(dirname $0)/CHANGELOG | grep '^## ' | head -1 | \
113 sed -e 's/.*\[\(.*\)\].*/\1/')
114 if [ "${VERSION}" = "" ]
116 echo "Unable to determine build version from CHANGELOG file. Make sure"
117 echo "that there is an entry for the most recent version in CHANGELOG,"
118 echo "and that the entry is formated like"
120 echo "## [1.0] - 2016-05-20"
123 RELEASE="csit-${OS}_${RELDATE}_${VERSION}"
125 OUTPUT_DIR="${BUILD_DIR}/output/${RELEASE}"
126 LIST="${LISTS_DIR}/${OS}_${RELDATE}_${VERSION}"
128 if [ ! -d "${LIST}" ]
130 echo "${LIST} not found"
134 if [ ! -f $NESTED_IMG ]
136 echo "Nested image $NESTED_IMG not found"
141 ATLAS_RELDATE=${RELDATE//-}
142 ATLAS_VERSION="${ATLAS_RELDATE}.${VERSION}"
144 # Find an MD5 checksum utility
146 MD5UTIL=$(which md5sum) || MD5UTIL=$(which md5)
149 echo "No MD5 utility found."
150 echo "Please make sure you either have \"md5sum\" or \"md5\" installed."
155 ### 1. Determine build target.
157 if [ "$VIRL_USER" = "" ] || [ "$VIRL_PASSWORD" = "" ]
159 OUTPUT_PROVIDER="virtualbox"
161 OUTPUT_PROVIDER="qemu"
164 echo "Building version $VERSION for ${OUTPUT_PROVIDER}"
165 echo "Release ${RELEASE}"
166 echo "Using Nested VM image: ${NESTED_IMG}"
171 ### 2. Download and extract packer, if not already installed
173 packer_os=$(uname -s)
174 if [ "$packer_os" = "Darwin" ]
176 packer_url="https://releases.hashicorp.com/packer/0.10.1/packer_0.10.1_darwin_amd64.zip"
177 elif [ "$packer_os" = "Linux" ]
179 packer_url="https://releases.hashicorp.com/packer/0.10.1/packer_0.10.1_linux_amd64.zip"
183 wget -P ${PACKER_DIR} -N ${packer_url}
185 unzip -n ${PACKER_DIR}/packer*zip -d ${PACKER_DIR}
188 ### 3. Download RPM and PIP packages, and cache them
189 ### Verify downloaded RPM packages.
190 ### Link required packages into a temp directory for the VM image.
195 mkdir -p ${OUTPUT_DIR}/temp/rpm
196 mkdir -p ${RPM_CACHE_DIR}
198 RPM_FILE="${LIST}/rpm-packages.txt"
199 REPO_FILE="${LIST}/Centos-Vault.repo"
201 ### Copy rpm package list to cache dir because we are going to use yum on the image
203 echo cp $RPM_FILE ${RPM_CACHE_DIR}
204 cp $RPM_FILE ${RPM_CACHE_DIR}
205 if [ -e $REPO_FILE ] ; then
206 echo cp $REPO_FILE ${RPM_CACHE_DIR}
207 cp $REPO_FILE ${RPM_CACHE_DIR}
208 ln ${RPM_CACHE_DIR}/Centos-Vault.repo ${OUTPUT_DIR}/temp/rpm/Centos-Vault.repo
210 ln ${RPM_CACHE_DIR}/rpm-packages.txt ${OUTPUT_DIR}/temp/rpm/rpm-packages.txt
214 mkdir -p ${PIP_CACHE_DIR}
216 # Let PIP do the work of downloading and verifying packages
217 pip download -d ${PIP_CACHE_DIR} -r ${LIST}/pip-requirements.txt
219 # Link packages and requirements file into VM's temp directory
220 mkdir -p ${OUTPUT_DIR}/temp/pip
221 ln ${PIP_CACHE_DIR}/* ${OUTPUT_DIR}/temp/pip/
222 ln ${LIST}/pip-requirements.txt ${OUTPUT_DIR}/temp/requirements.txt
225 ### 4. Link the nested VM image
227 rm -fr ${OUTPUT_DIR}/temp/nested-vm
228 mkdir ${OUTPUT_DIR}/temp/nested-vm
229 ln $NESTED_IMG ${OUTPUT_DIR}/temp/nested-vm/
232 ### 5. Create Version and Changelog files
235 echo ${RELEASE} > ${OUTPUT_DIR}/temp/VERSION
236 ln CHANGELOG ${OUTPUT_DIR}/temp/CHANGELOG
242 export PACKER_LOG_PATH=${OUTPUT_DIR}/packer.log
243 ${PACKER_DIR}/packer build -var "release=${RELEASE}" \
244 -only ${RELEASE}-${OUTPUT_PROVIDER} \
245 -var "output_dir=${OUTPUT_DIR}/packer" \
246 -var "temp_dir=${OUTPUT_DIR}/temp" \
247 -var "atlas_version=${ATLAS_VERSION}" \
249 -machine-readable ${PACKER_TEMPLATE}
251 # TODO: Need to figure out "packer push" later. Currently keeps failing.
252 # ${PACKER_DIR}/packer push -name ckoester/test123 -var "release=${RELEASE}" \
253 # -var "output_dir=${OUTPUT_DIR}/packer" \
254 # -var "temp_dir=${OUTPUT_DIR}/temp" \
260 rm -fr ${OUTPUT_DIR}/temp