1 # Automated Building Of FD.io CI Executor Docker Images
3 This collection of bash scripts and libraries is used to automate the process
4 of building FD.io docker 'builder' images (aka Nomad executors). The goal is to
5 create a completely automated CI/CD pipeline. The bash code is designed to be
6 run in a regular Linux bash shell in order to bootstrap the CI/CD pipeline
7 as well as in a docker 'builder' image started by a ci-management jenkins job.
8 The Dockerfile is generated prior to executing 'docker build' based on the os
9 parameter specified. The project git repos are also copied into the docker
10 container and retained for optimization of git object retrieval by the Jenkins
11 jobs running the CI/CD tasks.
13 ## Image Builder Algorithm
15 The general algorithm to automate the generation of the docker images such that
16 the downloadable requirements for each project are pre-installed or cached in
17 the executor image is as follows:
19 1. Run the docker image builder on a host of the target architecture. Bootstrap
20 images will be built 'by hand' on target hosts until such a time when the
21 CI is capable of executing the docker image builder scripts inside docker
22 images running on Nomad instances via jenkins jobs.
24 2. For each OS package manager, there is a bash function which generates the
25 Dockerfile for the specified OS which uses said package manager. For example,
26 lib_apt.sh contains 'generate_apt_dockerfile()' which is executed for Ubuntu
27 and debian OS's. lib_yum.sh and lib_dnf.sh contain similar functions for yum
28 (centos-7) and dnf (centos-8).
30 3. The Dockerfiles contain the following sections:
31 - a. Environment setup and copying of project workspace git repos
32 - b. Installation of OS package pre-requisites
33 - c. Docker install and project requirements installation (more on this below)
34 - d. Working environment setup
37 4. The Project installation section (c.) above is where all of the packages
38 for each of the supported project branches are installed or cached to
39 save time and bandwidth when the CI jobs are run. Each project script
40 defines the branches supported for each OS and iterates over them from
41 oldest to newest using the dependency and requirements files or build
42 targets in each supported project branch.
44 5. `docker build` is run on the generated Dockerfile.
46 ## Bash Libraries (lib_*.sh)
48 The bash libraries are designed to be sourced both inside of the docker build
49 environment (e.g. from a script invoked in a Dockerfile RUN statement) as well
50 as in a normal Linux shell. These scripts create environment variables and
51 bash functions for use by the operational scripts.
53 - `lib_apt.sh`: Dockerfile generation functions for apt package manager.
55 - `lib_common.sh`: Common utility functions and environment variables
57 - `lib_csit.sh`: CSIT specific functions and environment variables
59 - `lib_dnf.sh`: Dockerfile generation functions for dnf package manager.
61 - `lib_vpp.sh`: VPP specific functions and environment variables
63 - `lib_yum.sh`: Dockerfile generation functions for yum package manager.
67 There are two types of bash scripts, those intended to be run solely inside
68 the docker build execution environment, the other run either inside or
71 ### Docker Build (dbld_*.sh) Scripts
73 These scripts run inside the 'docker build' environment are either per-project
74 scripts that install OS and python packages or scripts that install other docker
75 image runtime requirements.
77 Python packages are not retained because they are typically installed in virtual
78 environments. However installing the python packages in the Docker Build scripts
79 populates the pip/http caches. Therefore packages are installed from the cache
80 files during CI job execution instead of being downloaded from the Internet.
82 - `dbld_csit_find_ansible_packages.sh`: Script to find OS packages installed by
85 - `dbld_csit_install_packages.sh`: Install OS and python packages for CSIT
88 - `dbld_dump_build_logs.sh`: Find warnings/errors in the build logs and dump
89 the build_executor_docker_image.sh execution log.
91 - `dbld_install_docker.sh`: Install docker ce
93 - `dbld_lfit_requirements.sh`: Install requirements for LFIT global-jjb
96 - `dbld_vpp_install_packages.sh`: Install OS and python packages for VPP
99 ### Executor Docker Image Management Bash Scripts
101 These scripts are used to build executor docker images, inspect the results, and
102 manage the docker image tags in the Docker Hub fdiotools repositories.
104 - `build_executor_docker_image.sh`: Build script to create one or more executor
107 - `update_dockerhub_prod_tags.sh`: Inspect/promote/revert production docker tag
108 in the Docker Hub fdiotools repositories.
110 ## Running The Scripts
112 ### Bootstrapping The Builder Images
114 The following commands are useful to build the initial builder images:
116 `cd <ci-managment repository directory>`
118 `sudo ./docker/scripts/build_executor_docker_image.sh ubuntu-18.04 2>&1 | tee u1804-$(uname -m).log | grep -ve '^+'`
120 `sudo ./docker/scripts/build_executor_docker_image.sh centos-7 2>&1 | tee centos7-$(uname -m).log | grep -ve '^+'`
122 `sudo ./docker/scripts/build_executor_docker_image.sh -apr sandbox 2>&1 | tee all-sandbox-$(uname -m).log | grep -ve '^+'`
124 Note: The initial population of a Docker Hub repository is performed manually by
125 tagging and pushing the verified sandbox image as 'prod-<arch>' and
126 'prod-prev-<arch>' as the update_dockerhub_prod_tags.sh script assumes that
127 both labels exist in the repo. After the intial images have been pushed to the
128 Docker Hub respository, the update script is used to prevent inadvertently
129 applying the wrong tags to images in the repository.
131 ### Building in a Builder Image
133 By running the docker image with docker socket mounted in the container,
134 the docker build environment runs on the host's docker daemon. This
135 avoids the pitfalls encountered with Docker-In-Docker environments:
137 `sudo docker run -it -v /var/run/docker.sock:/var/run/docker.sock <docker-image>`
139 The environment in the docker shell contains all of the necessary
140 environment variable definitions so the docker scripts can be run
141 directly on the cli. Here is an example command that would be used in a CI job
142 which automates the generation and testing of a new ubuntu-18.04 docker image
143 and push it to Docker Hub fdiotools/builder-ubuntu1804:test-<arch>:
145 `build_executor_docker_image.sh -pr test ubuntu-18.04`
147 In the future, a fully automated CI/CD pipeline may be created for production
150 # Docker Image Script Workflow
152 This section describes the current workflow used for managing the CI/CD pipeline
153 for the Docker Images used by the FD.io CI Jobs.
155 Note: all operations that push images or image tags to Docker Hub require an
156 account with management privileges of the fdiotools repositories.
158 ## Update Production Docker Images
160 Note: Presently only the 'builder' class executor docker images are supported.
161 The others will be supported in the near future.
163 ### Build Docker Images and Push to Docker Hub with Sandbox CI Tag
165 For each hardware architecture, the build_executor_docker_image.sh script is
166 used to build all variants of the each executor class:
168 1. `git clone https://gerrit.fd.io/r/ci-management && cd ci-management`
170 2. `sudo ./docker/scripts/build_executor_docker_image.sh -p -r sandbox -a | tee builder-all-sandbox-$(uname -m).log | grep -ve '^+'``
172 3. `Inspect the build log for Errors and other build anomalies`
174 This step will take a very long time so best to do it overnight. There is not
175 currently an option to automatically run builds in parallel, so if optimizing
176 build times is important, then run the jobs in separate shells for each OS.
177 The aarch64 builds are particularly slow, thus may benefit from being run on
178 separate hosts in parallel.
180 Note: the 'prod' role is disallowed in the build script to prevent accidental
181 deployment of untested docker images to production.
183 ### Test Docker Images in the Jenkins Sandbox
185 In the future, this step will be automated using the role 'test' and associated
186 tags, but for now testing is a manual operation.
188 1. `git clone https://gerrit.fd.io/r/vpp ../vpp && source ../vpp/extras/bash/functions.sh`
190 2. Edit jjb/vpp/vpp.yam (or other project yaml file) and replace '-prod-' with '-sandbox-' for all of the docker image
192 3. `jjb-sandbox-env` # This bash function currently lives in ../vpp/extras/bash/functions.sh
193 - TODO: move it to ci-management repo.
195 4. For each job using one of the docker images:
197 a. `jjsb-update <job name(s)>` # bash function created by jjb-sandbox-env to
198 push job to the sandbox
200 b. manually run the job in https://jenkins.fd.io/sandbox
202 c. Inspect the console output of each job for unnecessary downloads & errors.
204 ### Promote Docker Images to Production
206 Once all of the docker images have been tested, promote each one to production:
208 `sudo ./docker/scripts/update_dockerhub_prod_tags.sh promote <image name>`
210 Note: this script currently requires human acceptance via the terminal to ensure
212 It pulls all tags from the Docker Hub repos, does an Inspect action (displaying
213 the current state of 'prod' & 'prod-prev' tags) and local Promotion action (i.e.
214 tags local images with 'prod-<arch>' and 'prod-prev-<arch>') with a required
215 confirmation to continue the promotion by pushing the tags to Docker Hub. If
216 'no' is specified, it restores the previous local tags so they match the state
217 of Docker Hub and does a new Inspect action for verification. If 'yes' is
218 specified, it prints out the command to use to restore the existing state of the
219 production tags on Docker Hub in case the script is terminated prior to
220 completion. If necessary, the restore command can be repeated multiple times
221 until it completes successfully since it promotes the 'prod-prev-<arch>' image,
222 then the 'prod-<arch>' image in succession.
224 ## Other Docker Hub Operations
226 ### Inspect Production Docker Image Tags
228 Inspect the current production docker image tags:
230 `sudo ./docker/scripts/update_dockerhub_prod_tags.sh inspect fdiotools/<class>-<os name>:prod-$(uname -m)`
232 ### Revert Production Docker Image To Previous Docker Image
234 Inspect the current production docker image tags:
236 `sudo ./docker/scripts/update_dockerhub_prod_tags.sh revert fdiotools/<class>-<os name>:prod-$(uname -m)`
238 ### Restoring Previous Production Image State
240 Assuming that the images still exist in the Docker Hub repository, any previous
241 state of the production image tags can be restored by executing the 'restore
242 command' as output by the build_executor_docker_image.sh script. This script
243 writes a copy of all of the terminal output to a log file in
244 /tmp/build_executor_docker_image.sh.<date>.log thus providing a history of the
245 restore commands. When the building of executor docker images is peformed by a
246 CI job, the logging can be removed since the job execution will be captured in
247 the Jenkins console output log.
249 ### Docker Image Garbage Collection
251 Presently, cleaning up the Docker Hub repositories of old images/tags is a
252 manual process using the Docker Hub WebUI. In the future, a garbage collection
253 script will be written to automate the process.
255 # DockerHub Repository & Docker Image Tag Nomenclature:
257 ## DockerHub Repositories
259 - fdiotools/builder-centos7
260 - fdiotools/builder-centos8
261 - fdiotools/builder-debian9
262 - fdiotools/builder-debian10
263 - fdiotools/builder-ubuntu1804
264 - fdiotools/builder-ubuntu2004
265 - fdiotools/csit-ubuntu1804
266 - fdiotools/csit_dut-ubuntu1804
267 - fdiotools/csit_shim-ubuntu1804
271 - prod-x86_64: Tag used to select the x86_64 production image by the associated
273 - prod-prev-x86_64: Tag of the previous x86_64 production image used to revert
274 a production image to the previous image used in production.
275 - prod-aarch64: Tag used to select the aarch64 production image by the
276 associated Jenkins-Nomad Label.
277 - prod-prev-aarch64 Tag of the previous aarch64 production image used to revert
278 a production image to the previous image used in production.
279 - sandbox-x86_64: Tag used to select the x86_64 sandbox image by the associated
281 - sandbox-aarch64: Tag used to select the aarch64 sandbox image by the
282 associated Jenkins-Nomad Label.
283 - test-x86_64: Tag used to select the x86_64 sandbox image by the associated
285 - test-aarch64: Tag used to select the aarch64 sandbox image by the associated
288 # Jenkins-Nomad Label Definitions
290 <class>-<os>-<role>-<arch> (e.g. builder-ubuntu1804-prod-x86_64)
315 ## Jenkins Nomad Plugin Node Labels
317 ### Common Attributes of All Jenkins Nomad Plugin Nodes
320 - Idle Termination Time: 10
322 - Usage: Only build jobs with label expressions matching this node
328 ### Production (prod) Jenkins Nomad Plugin Nodes
330 #### Node 'builder-ubuntu1804-prod-x86_64'
331 - Labels: builder-ubuntu1804-prod-x86_64
332 - Job Prefix: builder-ubuntu1804-prod-x86_64
333 - Image: fdiotools/builder-ubuntu1804:prod-x86_64
336 - ${attr.cpu.arch}: amd64
337 - ${node.class}: builder
339 #### Node 'builder-ubuntu1804-prod-aarch64'
340 - Labels: builder-ubuntu1804-prod-aarch64
341 - Job Prefix: builder-ubuntu1804-prod-aarch64
342 - Image: fdiotools/builder-ubuntu1804:prod-aarch64
345 - ${attr.cpu.arch}: arm64
346 - ${node.class}: builder
348 #### Node 'builder-centos7-prod-x86_64'
349 - Labels: builder-centos7-prod-x86_64
350 - Job Prefix: builder-centos7-prod-x86_64
351 - Image: fdiotools/builder-centos7:prod-x86_64
354 - ${attr.cpu.arch}: amd64
355 - ${node.class}: builder
357 #### Node 'builder-centos7-prod-aarch64'
358 - Labels: builder-centos7-prod-aarch64
359 - Job Prefix: builder-centos7-prod-aarch64
360 - Image: fdiotools/builder-centos7:prod-aarch64
363 - ${attr.cpu.arch}: arm64
364 - ${node.class}: builder
366 #### Node 'builder-ubuntu2004-prod-x86_64'
367 - Labels: builder-ubuntu2004-prod-x86_64
368 - Job Prefix: builder-ubuntu2004-prod-x86_64
369 - Image: fdiotools/builder-ubuntu2004:prod-x86_64
372 - ${attr.cpu.arch}: amd64
373 - ${node.class}: builder
375 #### Node 'builder-ubuntu2004-prod-aarch64'
376 - Labels: builder-ubuntu2004-prod-aarch64
377 - Job Prefix: builder-ubuntu2004-prod-aarch64
378 - Image: fdiotools/builder-ubuntu2004:prod-aarch64
381 - ${attr.cpu.arch}: arm64
382 - ${node.class}: builder
384 #### Node 'builder-centos8-prod-x86_64'
385 - Labels: builder-centos8-prod-x86_64
386 - Job Prefix: builder-centos7-prod-x86_64
387 - Image: fdiotools/builder-centos8:prod-x86_64
390 - ${attr.cpu.arch}: amd64
391 - ${node.class}: builder
393 #### Node 'builder-centos8-prod-aarch64'
394 - Labels: builder-centos8-prod-aarch64
395 - Job Prefix: builder-centos8-prod-aarch64
396 - Image: fdiotools/builder-centos8:prod-aarch64
399 - ${attr.cpu.arch}: arm64
400 - ${node.class}: builder
402 #### Node 'builder-debian9-prod-x86_64'
403 - Labels: builder-debian9-prod-x86_64
404 - Job Prefix: builder-debian9-prod-x86_64
405 - Image: fdiotools/builder-debian9:prod-x86_64
408 - ${attr.cpu.arch}: amd64
409 - ${node.class}: builder
411 #### Node 'builder-debian9-prod-aarch64'
412 - Labels: builder-debian9-prod-aarch64
413 - Job Prefix: builder-debian9-prod-aarch64
414 - Image: fdiotools/builder-debian9:prod-aarch64
417 - ${attr.cpu.arch}: arm64
418 - ${node.class}: builder
420 #### Node 'builder-debian10-prod-x86_64'
421 - Labels: builder-debian10-prod-x86_64
422 - Job Prefix: builder-debian10-prod-x86_64
423 - Image: fdiotools/builder-debian10:prod-x86_64
426 - ${attr.cpu.arch}: amd64
427 - ${node.class}: builder
429 #### Node 'builder-debian10-prod-aarch64'
430 - Labels: builder-debian10-prod-aarch64
431 - Job Prefix: builder-debian10-prod-aarch64
432 - Image: fdiotools/builder-debian10:prod-aarch64
435 - ${attr.cpu.arch}: arm64
436 - ${node.class}: builder
438 #### Node 'csit_dut-ubuntu1804-prod-x86_64'
439 - Labels: csit_dut-ubuntu1804-prod-x86_64
440 - Job Prefix: csit_dut-ubuntu1804-prod-x86_64
441 - Image: fdiotools/csit_dut-ubuntu1804:prod-x86_64
444 - ${attr.cpu.arch}: amd64
445 - ${node.class}: csit
447 #### Node 'csit_dut-ubuntu1804-prod-aarch64'
448 - Labels: csit_dut-ubuntu1804-prod-aarch64
449 - Job Prefix: csit_dut-ubuntu1804-prod-aarch64
450 - Image: fdiotools/csit_dut-ubuntu1804:prod-aarch64
453 - ${attr.cpu.arch}: arm64
454 - ${node.class}: csitarm
456 #### Node 'csit_shim-ubuntu1804-prod-x86_64'
457 - Labels: csit_shim-ubuntu1804-prod-x86_64
458 - Job Prefix: csit_shim-ubuntu1804-prod-x86_64
459 - Image: fdiotools/csit_shim-ubuntu1804:prod-x86_64
462 - ${attr.cpu.arch}: amd64
463 - ${node.class}: csit
465 #### Node 'csit_shim-ubuntu1804-prod-aarch64'
466 - Labels: csit_shim-ubuntu1804-prod-aarch64
467 - Job Prefix: csit_shim-ubuntu1804-prod-aarch64
468 - Image: fdiotools/csit_shim-ubuntu1804:prod-aarch64
471 - ${attr.cpu.arch}: arm64
472 - ${node.class}: csitarm
474 ### Sandbox (sandbox) Jenkins Nomad Plugin Nodes
476 #### Node 'builder-ubuntu1804-sandbox-x86_64'
477 - Labels: builder-ubuntu1804-sandbox-x86_64
478 - Job Prefix: builder-ubuntu1804-sandbox-x86_64
479 - Image: fdiotools/builder-ubuntu1804:sandbox-x86_64
482 - ${attr.cpu.arch}: amd64
483 - ${node.class}: builder
485 #### Node 'builder-ubuntu1804-sandbox-aarch64'
486 - Labels: builder-ubuntu1804-sandbox-aarch64
487 - Job Prefix: builder-ubuntu1804-sandbox-aarch64
488 - Image: fdiotools/builder-ubuntu1804:sandbox-aarch64
491 - ${attr.cpu.arch}: arm64
492 - ${node.class}: builder
494 #### Node 'builder-centos7-sandbox-x86_64'
495 - Labels: builder-centos7-sandbox-x86_64
496 - Job Prefix: builder-centos7-sandbox-x86_64
497 - Image: fdiotools/builder-centos7:sandbox-x86_64
500 - ${attr.cpu.arch}: amd64
501 - ${node.class}: builder
503 #### Node 'builder-centos7-sandbox-aarch64'
504 - Labels: builder-centos7-sandbox-aarch64
505 - Job Prefix: builder-centos7-sandbox-aarch64
506 - Image: fdiotools/builder-centos7:sandbox-aarch64
509 - ${attr.cpu.arch}: arm64
510 - ${node.class}: builder
512 #### Node 'builder-ubuntu2004-sandbox-x86_64'
513 - Labels: builder-ubuntu2004-sandbox-x86_64
514 - Job Prefix: builder-ubuntu2004-sandbox-x86_64
515 - Image: fdiotools/builder-ubuntu2004:sandbox-x86_64
518 - ${attr.cpu.arch}: amd64
519 - ${node.class}: builder
521 #### Node 'builder-ubuntu2004-sandbox-aarch64'
522 - Labels: builder-ubuntu2004-sandbox-aarch64
523 - Job Prefix: builder-ubuntu2004-sandbox-aarch64
524 - Image: fdiotools/builder-ubuntu2004:sandbox-aarch64
527 - ${attr.cpu.arch}: arm64
528 - ${node.class}: builder
530 #### Node 'builder-centos8-sandbox-x86_64'
531 - Labels: builder-centos8-sandbox-x86_64
532 - Job Prefix: builder-centos8-sandbox-x86_64
533 - Image: fdiotools/builder-centos8:sandbox-x86_64
536 - ${attr.cpu.arch}: amd64
537 - ${node.class}: builder
539 #### Node 'builder-centos8-sandbox-aarch64'
540 - Labels: builder-centos8-sandbox-aarch64
541 - Job Prefix: builder-centos8-sandbox-aarch64
542 - Image: fdiotools/builder-centos8:sandbox-aarch64
545 - ${attr.cpu.arch}: arm64
546 - ${node.class}: builder
548 #### Node 'builder-debian9-sandbox-x86_64'
549 - Labels: builder-debian9-sandbox-x86_64
550 - Job Prefix: builder-debian9-sandbox-x86_64
551 - Image: fdiotools/builder-debian9:sandbox-x86_64
554 - ${attr.cpu.arch}: amd64
555 - ${node.class}: builder
557 #### Node 'builder-debian9-sandbox-aarch64'
558 - Labels: builder-debian9-sandbox-aarch64
559 - Job Prefix: builder-debian9-sandbox-aarch64
560 - Image: fdiotools/builder-debian9:sandbox-aarch64
563 - ${attr.cpu.arch}: arm64
564 - ${node.class}: builder
566 #### Node 'builder-debian10-sandbox-x86_64'
567 - Labels: builder-debian10-sandbox-x86_64
568 - Job Prefix: builder-debian10-sandbox-x86_64
569 - Image: fdiotools/builder-debian10:sandbox-x86_64
572 - ${attr.cpu.arch}: amd64
573 - ${node.class}: builder
575 #### Node 'builder-debian10-sandbox-aarch64'
576 - Labels: builder-debian10-sandbox-aarch64
577 - Job Prefix: builder-debian10-sandbox-aarch64
578 - Image: fdiotools/builder-debian10:sandbox-aarch64
581 - ${attr.cpu.arch}: arm64
582 - ${node.class}: builder
584 #### Node 'csit_dut-ubuntu1804-sandbox-x86_64'
585 - Labels: csit_dut-ubuntu1804-sandbox-x86_64
586 - Job Prefix: csit_dut-ubuntu1804-sandbox-x86_64
587 - Image: fdiotools/csit_dut-ubuntu1804:sandbox-x86_64
590 - ${attr.cpu.arch}: amd64
591 - ${node.class}: csit
593 #### Node 'csit_dut-ubuntu1804-sandbox-aarch64'
594 - Labels: csit_dut-ubuntu1804-sandbox-aarch64
595 - Job Prefix: csit_dut-ubuntu1804-sandbox-aarch64
596 - Image: fdiotools/csit_dut-ubuntu1804:sandbox-aarch64
599 - ${attr.cpu.arch}: arm64
600 - ${node.class}: csitarm
602 #### Node 'csit_shim-ubuntu1804-sandbox-x86_64'
603 - Labels: csit_shim-ubuntu1804-sandbox-x86_64
604 - Job Prefix: csit_shim-ubuntu1804-sandbox-x86_64
605 - Image: fdiotools/csit_shim-ubuntu1804:sandbox-x86_64
608 - ${attr.cpu.arch}: amd64
609 - ${node.class}: csit
611 #### Node 'csit_shim-ubuntu1804-sandbox-aarch64'
612 - Labels: csit_shim-ubuntu1804-sandbox-aarch64
613 - Job Prefix: csit_shim-ubuntu1804-sandbox-aarch64
614 - Image: fdiotools/csit_shim-ubuntu1804:sandbox-aarch64
617 - ${attr.cpu.arch}: arm64
618 - ${node.class}: csitarm
620 ### Automated Testing (test) Jenkins Nomad Plugin Nodes
622 #### Node 'builder-ubuntu1804-test-x86_64'
623 - Labels: builder-ubuntu1804-test-x86_64
624 - Job Prefix: builder-ubuntu1804-test-x86_64
625 - Image: fdiotools/builder-ubuntu1804:test-x86_64
628 - ${attr.cpu.arch}: amd64
629 - ${node.class}: builder
631 #### Node 'builder-ubuntu1804-test-aarch64'
632 - Labels: builder-ubuntu1804-test-aarch64
633 - Job Prefix: builder-ubuntu1804-test-aarch64
634 - Image: fdiotools/builder-ubuntu1804:test-aarch64
637 - ${attr.cpu.arch}: arm64
638 - ${node.class}: builder
640 #### Node 'builder-centos7-test-x86_64'
641 - Labels: builder-centos7-test-x86_64
642 - Job Prefix: builder-centos7-test-x86_64
643 - Image: fdiotools/builder-centos7:test-x86_64
646 - ${attr.cpu.arch}: amd64
647 - ${node.class}: builder
649 #### Node 'builder-centos7-test-aarch64'
650 - Labels: builder-centos7-test-aarch64
651 - Job Prefix: builder-centos7-test-aarch64
652 - Image: fdiotools/builder-centos7:test-aarch64
655 - ${attr.cpu.arch}: arm64
656 - ${node.class}: builder
658 #### Node 'builder-ubuntu2004-test-x86_64'
659 - Labels: builder-ubuntu2004-test-x86_64
660 - Job Prefix: builder-ubuntu2004-test-x86_64
661 - Image: fdiotools/builder-ubuntu2004:test-x86_64
664 - ${attr.cpu.arch}: amd64
665 - ${node.class}: builder
667 #### Node 'builder-ubuntu2004-test-aarch64'
668 - Labels: builder-ubuntu2004-test-aarch64
669 - Job Prefix: builder-ubuntu2004-test-aarch64
670 - Image: fdiotools/builder-ubuntu2004:test-aarch64
673 - ${attr.cpu.arch}: arm64
674 - ${node.class}: builder
676 #### Node 'builder-centos8-test-x86_64'
677 - Labels: builder-centos8-test-x86_64
678 - Job Prefix: builder-centos8-test-x86_64
679 - Image: fdiotools/builder-centos8:test-x86_64
682 - ${attr.cpu.arch}: amd64
683 - ${node.class}: builder
685 #### Node 'builder-centos8-test-aarch64'
686 - Labels: builder-centos8-test-aarch64
687 - Job Prefix: builder-centos8-test-aarch64
688 - Image: fdiotools/builder-centos8:test-aarch64
691 - ${attr.cpu.arch}: arm64
692 - ${node.class}: builder
694 #### Node 'builder-debian9-test-x86_64'
695 - Labels: builder-debian9-test-x86_64
696 - Job Prefix: builder-debian9-test-x86_64
697 - Image: fdiotools/builder-debian9:test-x86_64
700 - ${attr.cpu.arch}: amd64
701 - ${node.class}: builder
703 #### Node 'builder-debian9-test-aarch64'
704 - Labels: builder-debian9-test-aarch64
705 - Job Prefix: builder-debian9-test-aarch64
706 - Image: fdiotools/builder-debian9:test-aarch64
709 - ${attr.cpu.arch}: arm64
710 - ${node.class}: builder
712 #### Node 'builder-debian10-test-x86_64'
713 - Labels: builder-debian10-test-x86_64
714 - Job Prefix: builder-debian10-test-x86_64
715 - Image: fdiotools/builder-debian10:test-x86_64
718 - ${attr.cpu.arch}: amd64
719 - ${node.class}: builder
721 #### Node 'builder-debian10-test-aarch64'
722 - Labels: builder-debian10-test-aarch64
723 - Job Prefix: builder-debian10-test-aarch64
724 - Image: fdiotools/builder-debian10:test-aarch64
727 - ${attr.cpu.arch}: arm64
728 - ${node.class}: builder
730 #### Node 'csit_dut-ubuntu1804-sandbox-x86_64'
731 - Labels: csit_dut-ubuntu1804-sandbox-x86_64
732 - Job Prefix: csit_dut-ubuntu1804-sandbox-x86_64
733 - Image: fdiotools/csit_dut-ubuntu1804:sandbox-x86_64
736 - ${attr.cpu.arch}: amd64
737 - ${node.class}: csit
739 #### Node 'csit_dut-ubuntu1804-test-aarch64'
740 - Labels: csit_dut-ubuntu1804-test-aarch64
741 - Job Prefix: csit_dut-ubuntu1804-test-aarch64
742 - Image: fdiotools/csit_dut-ubuntu1804:test-aarch64
745 - ${attr.cpu.arch}: arm64
746 - ${node.class}: csitarm
748 #### Node 'csit_shim-ubuntu1804-test-aarch64'
749 - Labels: csit_shim-ubuntu1804-test-aarch64
750 - Job Prefix: csit_shim-ubuntu1804-test-aarch64
751 - Image: fdiotools/csit_shim-ubuntu1804:test-aarch64
754 - ${attr.cpu.arch}: arm64
755 - ${node.class}: csitarm