X-Git-Url: https://gerrit.fd.io/r/gitweb?a=blobdiff_plain;f=docs%2Freport%2Fvpp_performance_tests%2Fdocumentation%2Fcontainers.rst;h=14817df64f4a7512f67be3855c4598f61b64dac8;hb=refs%2Fchanges%2F64%2F27564%2F9;hp=313b5b757540a5876275c17e2f46c78c5249686f;hpb=459d0af9c92134c5320e0f410b961c27f431f71f;p=csit.git diff --git a/docs/report/vpp_performance_tests/documentation/containers.rst b/docs/report/vpp_performance_tests/documentation/containers.rst index 313b5b7575..14817df64f 100644 --- a/docs/report/vpp_performance_tests/documentation/containers.rst +++ b/docs/report/vpp_performance_tests/documentation/containers.rst @@ -60,7 +60,7 @@ list of applicable security control mechanisms: escaping a container or cause harm to the host, [apparmor]_. - SELinux - Security Enhanced Linux is a Linux kernel security module that provides similar function to AppArmor, supporting access control - security policies including United States Department of Defense–style + security policies including United States Department of Defense-style mandatory access controls. Mandatory access controls allow an administrator of a system to define how applications and users can access different resources such as files, devices, networks and inter- @@ -111,8 +111,6 @@ Current CSIT testing framework integrates following Linux container orchestration mechanisms: - LXC/Docker for complete VPP container lifecycle control. -- Combination of Kubernetes (container orchestration), Docker (container - images) and Ligato (container networking). LXC ~~~ @@ -148,13 +146,13 @@ Docker Docker builds on top of Linux kernel containment features, and offers a high-level tool for wrapping the processes, maintaining and -executing them in containers [docker]_. Currently it using *runc* a CLI tool for -spawning and running containers according to the `OCI specification -`_ +executing them in containers [docker]_. Currently it is using *runc*, +a CLI tool for spawning and running containers according to the +`OCI specification `_. A Docker container image is a lightweight, stand-alone, executable -package of a piece of software that includes everything needed to run -it: code, runtime, system tools, system libraries, settings. +package that includes everything needed to run the container: +code, runtime, system tools, system libraries, settings. CSIT uses Docker to manage the maintenance and execution of containerized applications used in CSIT performance tests. @@ -163,31 +161,6 @@ containerized applications used in CSIT performance tests. configuration file controls the range of CPU cores the Docker image must run on. VPP thread pinning defined vpp startup.conf. -Kubernetes -~~~~~~~~~~ - -Kubernetes [k8sdoc]_, or K8s, is a production-grade container orchestration -platform for automating the deployment, scaling and operating -application containers. Kubernetes groups containers that make up an -application into logical units, pods, for easy management and discovery. -K8s pod definitions including compute resource allocation is provided in -.yaml files. - -CSIT uses K8s and its infrastructure components like etcd to control all -phases of container based virtualized network topologies. - -Ligato -~~~~~~ - -Ligato [ligato]_ is an open-source project developing a set of cloud-native -tools for orchestrating container networking. Ligato integrates with FD.io VPP -using goVPP [govpp]_ and vpp-agent [vppagent]_. - -**Known Issues** - -- Currently using a separate LF Jenkins job for building csit-centric - prod_vpp_agent docker images vs. dockerhub/ligato ones. - Implementation -------------- @@ -252,13 +225,13 @@ Current CSIT implementation is illustrated using UML Class diagram: | +----------+---------+ | | - +------+-------+ +------+-------+ - | LXC | | Docker | - +--------------+ +--------------+ - | (inherinted) | | (inherinted) | - +------+-------+ +------+-------+ - | | - +---------+---------+ + +------+------+ +------+------+ + | LXC | | Docker | + +-------------+ +-------------+ + | (inherited) | | (inherited) | + +------+------+ +------+------+ + | | + +----------+---------+ | | constructs | @@ -350,146 +323,66 @@ variables and no methods except overriden ``__getattr__`` and ``__setattr__``. Instance variables are assigned to container dynamically during the ``construct_container(**kwargs)`` call and are passed down from the RF keyword. -Usage example: +There is no parameters check functionality. Passing the correct arguments +is a responsibility of the caller. + +Examples +~~~~~~~~ + +This section contains a high-level example of multiple initialization steps +via ContainerManager; taken from an actual CSIT code, +but with non-code lines (comments, Documentation) removed for brevity. + +: .. code-block:: robotframework - | Construct VNF containers on all DUTs - | | [Arguments] | ${technology} | ${image} | ${cpu_count}=${1} | ${count}=${1} - | | ... - | | ${group}= | Set Variable | VNF - | | ${skip_cpus}= | Evaluate | ${vpp_cpus}+${system_cpus} + | Start containers for test + | | [Arguments] | ${dut}=${None} | ${nf_chains}=${1} | ${nf_nodes}=${1} + | | ... | ${auto_scale}=${True} | ${pinning}=${True} + | | + | | Set Test Variable | @{container_groups} | @{EMPTY} + | | Set Test Variable | ${container_group} | CNF + | | Set Test Variable | ${nf_nodes} | | Import Library | resources.libraries.python.ContainerUtils.ContainerManager - | | ... | engine=${container_engine} | WITH NAME | ${group} - | | ${duts}= | Get Matches | ${nodes} | DUT* - | | :FOR | ${dut} | IN | @{duts} - | | | ${env}= | Create List | DEBIAN_FRONTEND=noninteractive - | | | ${mnt}= | Create List | /tmp:/mnt/host | /dev:/dev - | | | ${cpu_node}= | Get interfaces numa node | ${nodes['${dut}']} - | | | ... | ${dut1_if1} | ${dut1_if2} - | | | Run Keyword | ${group}.Construct containers - | | | ... | name=${dut}_${group} | node=${nodes['${dut}']} | mnt=${mnt} - | | | ... | image=${container_image} | cpu_count=${container_cpus} - | | | ... | cpu_skip=${skip_cpus} | cpuset_mems=${cpu_node} - | | | ... | cpu_shared=${False} | env=${env} | count=${container_count} - | | | ... | install_dkms=${container_install_dkms} - | | Append To List | ${container_groups} | ${group} - -Mandatory parameters to create standalone container are: ``node``, ``name``, -``image`` [imagevar]_, ``cpu_count``, ``cpu_skip``, ``cpuset_mems``, -``cpu_shared``. - -There is no parameters check functionality. Passing required arguments is in -coder responsibility. All the above parameters are required to calculate the -correct cpu placement. See documentation for the full reference. + | | ... | engine=${container_engine} | WITH NAME | ${container_group} + | | Construct chains of containers + | | ... | dut=${dut} | nf_chains=${nf_chains} | nf_nodes=${nf_nodes} + | | ... | auto_scale=${auto_scale} | pinning=${pinning} + | | Acquire all '${container_group}' containers + | | Create all '${container_group}' containers + | | Configure VPP in all '${container_group}' containers + | | Start VPP in all '${container_group}' containers + | | Append To List | ${container_groups} | ${container_group} + | | Save VPP PIDs Kubernetes ~~~~~~~~~~ -Kubernetes is implemented as separate library ``KubernetesUtils.py``, -with a class with the same name. This utility provides an API for L2 -Robot Keywords to control ``kubectl`` installed on each of DUTs. One -time initialization script, ``resources/libraries/bash/k8s_setup.sh`` -does reset/init kubectl, applies Calico v2.6.3 and initializes the -``csit`` namespace. CSIT namespace is required to not to interfere with -existing setups and it further simplifies apply/get/delete -Pod/ConfigMap operations on SUTs. +For the future use, Kubernetes [k8sdoc]_ is implemented as separate library +``KubernetesUtils.py``, with a class with the same name. This utility provides +an API for L2 Robot Keywords to control ``kubectl`` installed on each of DUTs. +One time initialization script, ``resources/libraries/bash/k8s_setup.sh`` +does reset/init kubectl, and initializes the ``csit`` namespace. CSIT +namespace is required to not to interfere with existing setups and it +further simplifies apply/get/delete Pod/ConfigMap operations on SUTs. Kubernetes utility is based on YAML templates to avoid crafting the huge YAML configuration files, what would lower the readability of code and -requires complicated algorithms. The templates can be found in -``resources/templates/kubernetes`` and can be leveraged in the future -for other separate tasks. +requires complicated algorithms. Two types of YAML templates are defined: - Static - do not change between deployments, that is infrastructure containers like Kafka, Calico, ETCD. -- Dynamic - per test suite/case topology YAML files e.g. SFC_controller, - VNF, VSWITCH. +- Dynamic - per test suite/case topology YAML files. Making own python wrapper library of ``kubectl`` instead of using the official Python package allows to control and deploy environment over the SSH library without the need of using isolated driver running on each of DUTs. -Ligato -~~~~~~ - -Ligato integration does require to compile the ``vpp-agent`` tool and build the -bundled Docker image. Compilation of ``vpp-agent`` depends on specific VPP. In -``ligato/vpp-agent`` repository there are well prepared scripts for building the -Docker image. Building docker image is possible via series of commands: - -:: - - git clone https://github.com/ligato/vpp-agent - cd vpp_agent/docker/dev_vpp_agent - sudo docker build -t dev_vpp_agent --build-arg AGENT_COMMIT=\ - --build-arg VPP_COMMIT= --no-cache . - sudo ./shrink.sh - cd ../prod_vpp_agent - sudo ./build.sh - sudo ./shrink.sh - -CSIT requires Docker image to include the desired VPP version (per patch -testing, nightly testing, on demand testing). - -The entire build process of building ``dev_vpp_agent`` image heavily depends -on internet connectivity and also takes a significant amount of time (~1-1.5h -based on internet bandwidth and allocated resources). The optimal solution would -be to build the image on jenkins slave, transfer the Docker image to DUTs and -execute separate suite of tests. - -To adress the amount of time required to build ``dev_vpp_agent`` image, we can -pull existing specific version of ```dev_vpp_agent``` and exctract the -```vpp-agent``` from it. - -We created separate sets of Jenkins jobs, that will be executing following: - -1. Clone latest CSIT and Ligato repositaries. -2. Pull specific version of ``dev_vpp_agent`` image from Dockerhub. -3. Extract VPP API (from ``.deb`` package) and copy into ``dev_vpp_agent`` - image -4. Rebuild vpp-agent and extract outside image. -5. Build ``prod_vpp_image`` Docker image from ``dev_vpp_agent`` image. -6. Transfer ``prod_vpp_agent`` image to DUTs. -7. Execute subset of performance tests designed for Ligato testing. - -:: - - +-----------------------------------------------+ - | ubuntu:16.04 <-----| Base image on Dockerhub - +------------------------^----------------------+ - | - | - +------------------------+----------------------+ - | ligato/dev_vpp_agent <------| Pull this image from - +------------------------^----------------------+ | Dockerhub ligato/dev_vpp_agent: - | - | Rebuild and extract agent.tar.gz from dev_vpp_agent - +------------------------+----------------------+ - | prod_vpp_agent <------| Build by passing own - +-----------------------------------------------+ | vpp.tar.gz (from nexus - | or built by JJB) and - | agent.tar.gz extracted - | from ligato/dev_vpp_agent - - -Approximate size of vnf-agent docker images: - -:: - - REPOSITORY TAG IMAGE ID CREATED SIZE - dev-vpp-agent latest 78c53bd57e2 6 weeks ago 9.79GB - prod_vpp_agent latest f68af5afe601 5 weeks ago 443MB - -In CSIT we need to create separate performance suite under -``tests/kubernetes/perf`` which contains modified Suite setup in comparison -to standard perf tests. This is due to reason that VPP will act as vswitch in -Docker image and not as standalone installed service. - Tested Topologies ~~~~~~~~~~~~~~~~~ @@ -509,19 +402,6 @@ Following container networking topologies are tested in |csit-release|: - eth-l2xcbase-eth-2memif-1docker. - eth-l2xcbase-eth-1memif-1docker -- Kubernetes/Ligato topologies: - - - eth-1drcl2bdbasemaclrn-eth-2memif-1drcl2xc-1paral - - eth-1drcl2bdbasemaclrn-eth-2memif-2drcl2xc-1horiz - - eth-1drcl2bdbasemaclrn-eth-2memif-4drcl2xc-1horiz - - eth-1drcl2bdbasemaclrn-eth-4memif-2drcl2xc-1chain - - eth-1drcl2bdbasemaclrn-eth-8memif-4drcl2xc-1chain - - eth-1drcl2xcbase-eth-2memif-1drcl2xc-1paral - - eth-1drcl2xcbase-eth-2memif-2drcl2xc-1horiz - - eth-1drcl2xcbase-eth-2memif-4drcl2xc-1horiz - - eth-1drcl2xcbase-eth-4memif-2drcl2xc-1chain - - eth-1drcl2xcbase-eth-8memif-4drcl2xc-1chain - References ~~~~~~~~~~ @@ -539,7 +419,3 @@ References .. [seccomp] `SECure COMPuting with filters `_. .. [docker] `Docker `_. .. [k8sdoc] `Kubernetes documentation `_. -.. [ligato] `Ligato `_. -.. [govpp] `FD.io goVPP project `_. -.. [vppagent] `Ligato vpp-agent `_. -.. [imagevar] Image parameter is required in initial commit version. There is plan to implement container build class to build Docker/LXC image.