-.. _containter_orchestration_in_csit:
+.. _container_orchestration_in_csit:
Container Orchestration in CSIT
===============================
:abbr:`LXC (Linux Containers)` combine kernel's cgroups and support for isolated
namespaces to provide an isolated environment for applications. Docker
does use LXC as one of its execution drivers, enabling image management
-and providing deployment services. More information in [lxc]_, [lxc-namespace]_
+and providing deployment services. More information in [lxc]_, [lxcnamespace]_
and [stgraber]_.
Linux containers can be of two kinds: privileged containers and
users can create containers and will appear in the container as the
root, but will appear as userid <non-zero> on the host. Unprivileged
containers are also better suited to supporting multi-tenancy operating
-environments. More information in [lxc-security]_ and [stgraber]_.
+environments. More information in [lxcsecurity]_ and [stgraber]_.
Privileged Containers
~~~~~~~~~~~~~~~~~~~~~
- Seccomp - secure computing mode, enables filtering of system calls,
[seccomp]_.
-More information in [lxc-security]_ and [lxc-sec-features]_.
+More information in [lxcsecurity]_ and [lxcsecfeatures]_.
**Linux Containers in CSIT**
2. Build - building a container image from scratch or another
container image via :command:`docker build <dockerfile/composefile>` or
customizing LXC templates in
- `https://github.com/lxc/lxc/tree/master/templates`_
+ `GitHub <https://github.com/lxc/lxc/tree/master/templates>`_.
3. (Re-)Create - creating a running instance of a container application
from anew, or re-creating one that failed. A.k.a. (re-)deploy via
orchestration mechanisms:
- LXC/Docker for complete VPP container lifecycle control.
-- Combination of Kubernetes (container orchestration), Docker (container
- images) and Ligato (container networking).
LXC
~~~
LXC is the well-known and heavily tested low-level Linux container
-runtime [lxc-source]_, that provides a userspace interface for the Linux kernel
+runtime [lxcsource]_, that provides a userspace interface for the Linux kernel
containment features. With a powerful API and simple tools, LXC enables
Linux users to easily create and manage system or application
containers. LXC uses following kernel features to contain processes:
of functionality is better supported in LXC 2.1 but can be done is current
version as well.
-**Open Questions**
-
-- CSIT code is currently using cgroup to pin lxc data plane thread to
- cpu cores after lxc container is created. In the future may find a
- more universal way to do it.
+- CSIT code is currently using cgroup to control the range of CPU cores the
+ LXC container runs on. VPP thread pinning is defined vpp startup.conf.
Docker
~~~~~~
configuration file controls the range of CPU cores the Docker image
must run on. VPP thread pinning defined vpp startup.conf.
-
-Kubernetes
-~~~~~~~~~~
-
-Kubernetes [k8s-doc]_, or K8s, is a production-grade container orchestration
-platform for automating the deployment, scaling and operating
-application containers. Kubernetes groups containers that make up an
-application into logical units, pods, for easy management and discovery.
-K8s pod definitions including compute resource allocation is provided in
-.yaml files.
-
-CSIT uses K8s and its infrastructure components like etcd to control all
-phases of container based virtualized network topologies.
-
-**Known Issues**
-
-- Unable to properly pin k8s pods and containers to cpu cores. This will be
- addressed in Kubernetes 1.8+ in alpha testing.
-
-**Open Questions**
-
-- Clarify the functions provided by Contiv and Calico in Ligato system?
-
-Ligato
-~~~~~~
-
-Ligato [ligato]_ is an open-source project developing a set of cloud-native
-tools for orchestrating container networking. Ligato integrates with FD.io VPP
-using goVPP [govpp]_ and vpp-agent [vpp-agent]_.
-
-**Known Issues**
-
-**Open Questions**
-
-- Currently using a separate LF Jenkins job for building csit-centric
- vpp_agent docker images vs. dockerhub/ligato ones.
-
Implementation
--------------
Sequentional diagram that illustrates the creation of a single container.
-.. mk: what "RF KW" is meant below?
-.. mk: the flow sequence should adhere to the lifecycle events listed earlier in this doc.
-
::
Legend:
| | [Arguments] | ${technology} | ${image} | ${cpu_count}=${1} | ${count}=${1}
| | ...
| | ${group}= | Set Variable | VNF
- | | ${guest_dir}= | Set Variable | /mnt/host
- | | ${host_dir}= | Set Variable | /tmp
| | ${skip_cpus}= | Evaluate | ${vpp_cpus}+${system_cpus}
| | Import Library | resources.libraries.python.ContainerUtils.ContainerManager
- | | ... | engine=${technology} | WITH NAME | ${group}
+ | | ... | engine=${container_engine} | WITH NAME | ${group}
| | ${duts}= | Get Matches | ${nodes} | DUT*
| | :FOR | ${dut} | IN | @{duts}
- | | | {env}= | Create List | LC_ALL="en_US.UTF-8"
- | | | ... | DEBIAN_FRONTEND=noninteractive | ETCDV3_ENDPOINTS=172.17.0.1:2379
+ | | | ${env}= | Create List | DEBIAN_FRONTEND=noninteractive
+ | | | ${mnt}= | Create List | /tmp:/mnt/host | /dev:/dev
| | | ${cpu_node}= | Get interfaces numa node | ${nodes['${dut}']}
| | | ... | ${dut1_if1} | ${dut1_if2}
| | | Run Keyword | ${group}.Construct containers
- | | | ... | name=${dut}_${group}
- | | | ... | node=${nodes['${dut}']}
- | | | ... | host_dir=${host_dir}
- | | | ... | guest_dir=${guest_dir}
- | | | ... | image=${image}
- | | | ... | cpu_count=${cpu_count}
- | | | ... | cpu_skip=${skip_cpus}
- | | | ... | smt_used=${False}
- | | | ... | cpuset_mems=${cpu_node}
- | | | ... | cpu_shared=${False}
- | | | ... | env=${env}
+ | | | ... | name=${dut}_${group} | node=${nodes['${dut}']} | mnt=${mnt}
+ | | | ... | image=${container_image} | cpu_count=${container_cpus}
+ | | | ... | cpu_skip=${skip_cpus} | cpuset_mems=${cpu_node}
+ | | | ... | cpu_shared=${False} | env=${env} | count=${container_count}
+ | | | ... | install_dkms=${container_install_dkms}
+ | | Append To List | ${container_groups} | ${group}
Mandatory parameters to create standalone container are: ``node``, ``name``,
-``image`` [image-var]_, ``cpu_count``, ``cpu_skip``, ``smt_used``,
-``cpuset_mems``, ``cpu_shared``.
+``image`` [imagevar]_, ``cpu_count``, ``cpu_skip``, ``cpuset_mems``,
+``cpu_shared``.
There is no parameters check functionality. Passing required arguments is in
coder responsibility. All the above parameters are required to calculate the
Kubernetes
~~~~~~~~~~
-Kubernetes is implemented as separate library ``KubernetesUtils.py``,
-with a class with the same name. This utility provides an API for L2
-Robot Keywords to control ``kubectl`` installed on each of DUTs. One
-time initialization script, ``resources/libraries/bash/k8s_setup.sh``
-does reset/init kubectl, applies Calico v2.4.1 and initializes the
-``csit`` namespace. CSIT namespace is required to not to interfere with
-existing setups and it further simplifies apply/get/delete
-Pod/ConfigMap operations on SUTs.
+For the future use, Kubernetes [k8sdoc]_ is implemented as separate library
+``KubernetesUtils.py``, with a class with the same name. This utility provides
+an API for L2 Robot Keywords to control ``kubectl`` installed on each of DUTs.
+One time initialization script, ``resources/libraries/bash/k8s_setup.sh``
+does reset/init kubectl, and initializes the ``csit`` namespace. CSIT
+namespace is required to not to interfere with existing setups and it
+further simplifies apply/get/delete Pod/ConfigMap operations on SUTs.
Kubernetes utility is based on YAML templates to avoid crafting the huge
YAML configuration files, what would lower the readability of code and
-requires complicated algorithms. The templates can be found in
-``resources/templates/kubernetes`` and can be leveraged in the future
-for other separate tasks.
+requires complicated algorithms.
Two types of YAML templates are defined:
- Static - do not change between deployments, that is infrastructure
containers like Kafka, Calico, ETCD.
-- Dynamic - per test suite/case topology YAML files e.g. SFC_controller,
- VNF, VSWITCH.
+- Dynamic - per test suite/case topology YAML files.
Making own python wrapper library of ``kubectl`` instead of using the
official Python package allows to control and deploy environment over
the SSH library without the need of using isolated driver running on
each of DUTs.
-Ligato
-~~~~~~
-
-Ligato integration does require to compile the ``vpp-agent`` tool and build the
-bundled Docker image. Compilation of ``vpp-agent`` depends on specific VPP. In
-``ligato/vpp-agent`` repository there are well prepared scripts for building the
-Docker image. Building docker image is possible via series of commands:
-
-::
-
- git clone https://github.com/ligato/vpp-agent
- cd vpp_agent/docker/dev_vpp_agent
- sudo docker build -t dev_vpp_agent --build-arg AGENT_COMMIT=<agent commit id>\
- --build-arg VPP_COMMIT=<vpp commit id> --no-cache .
- sudo ./shrink.sh
- cd ../prod_vpp_agent
- sudo ./build.sh
- sudo ./shrink.sh
-
-CSIT requires Docker image to include the desired VPP version (per patch
-testing, nightly testing, on demand testing).
-
-The entire build process of building ``dev_vpp_agent`` image heavily depends
-on internet connectivity and also takes a significant amount of time (~1-1.5h
-based on internet bandwidth and allocated resources). The optimal solution would
-be to build the image on jenkins slave, transfer the Docker image to DUTs and
-execute separate suite of tests.
-
-To adress the amount of time required to build ``dev_vpp_agent`` image, we can
-pull existing specific version of ```dev_vpp_agent``` and exctract the
-```vpp-agent``` from it.
-
-We created separate sets of Jenkins jobs, that will be executing following:
-
-1. Clone latest CSIT and Ligato repositaries.
-2. Pull specific version of ``dev_vpp_agent`` image from Dockerhub.
-3. Build ``prod_vpp_image`` Docker image from ``dev_vpp_agent`` image.
-4. Shrink image using ``docker/dev_vpp_agent/shrink.sh`` script.
-5. Transfer ``prod_vpp_agent_shrink`` image to DUTs.
-6. Execute subset of performance tests designed for Ligato testing.
-
-::
-
- +-----------------------------------------------+
- | ubuntu:16.04 <-----| Base image on Dockerhub
- +------------------------^----------------------+
- |
- |
- +------------------------+----------------------+
- | ligato/dev_vpp_agent <------| Pull this image from
- +------------------------^----------------------+ | Dockerhub ligato/dev_vpp_agent:<version>
- |
- | Extract agent.tar.gz from dev_vpp_agent
- +------------------------+----------------------+
- | prod_vpp_agent <------| Build by passing own
- +-----------------------------------------------+ | vpp.tar.gz (from nexus
- | or built by JJB) and
- | agent.tar.gz extracted
- | from ligato/dev_vpp_agent
-
-
-Approximate size of vnf-agent docker images:
-
-::
-
- REPOSITORY TAG IMAGE ID CREATED SIZE
- dev_vpp_agent latest 442771972e4a 8 hours ago 3.57 GB
- dev_vpp_agent_shrink latest bd2e76980236 8 hours ago 1.68 GB
- prod_vpp_agent latest e33a5551b504 2 days ago 404 MB
- prod_vpp_agent_shrink latest 446b271cce26 2 days ago 257 MB
-
-In CSIT we need to create separate performance suite under
-``tests/kubernetes/perf`` which contains modified Suite setup in comparison
-to standard perf tests. This is due to reason that VPP will act as vswitch in
-Docker image and not as standalone installed service.
-
Tested Topologies
~~~~~~~~~~~~~~~~~
containerized VPP switch forwarding packets between NF containers. Each
NF container runs their own instance of VPP in L2XC configuration.
-Following container networking topologies are tested in CSIT |release|:
+Following container networking topologies are tested in |csit-release|:
- LXC topologies:
- Docker topologies:
- eth-l2xcbase-eth-2memif-1docker.
-
-- Kubernetes/Ligato topologies:
-
- - eth-1drcl2xcbase-eth-2memif-1drcl2xc.
- - eth-1drcl2xcbase-eth-4memif-2drcl2xc.
- - eth-1drcl2bdbasemaclrn-eth-2memif-1drcl2xc.
- - eth-1drcl2bdbasemaclrn-eth-4memif-2drcl2xc.
-
+ - eth-l2xcbase-eth-1memif-1docker
References
-----------
+~~~~~~~~~~
.. [lxc] `Linux Containers <https://linuxcontainers.org/>`_
-.. [lxc-namespace] `Resource management: Linux kernel Namespaces and cgroups <https://www.cs.ucsb.edu/~rich/class/cs293b-cloud/papers/lxc-namespace.pdf>`_.
+.. [lxcnamespace] `Resource management: Linux kernel Namespaces and cgroups <https://www.cs.ucsb.edu/~rich/class/cs293b-cloud/papers/lxc-namespace.pdf>`_.
.. [stgraber] `LXC 1.0: Blog post series <https://stgraber.org/2013/12/20/lxc-1-0-blog-post-series/>`_.
-.. [lxc-security] `Linux Containers Security <https://linuxcontainers.org/lxc/security/>`_.
-.. [capabilities] `Linux manual - capabilities - overview of Linux capabilities http://man7.org/linux/man-pages/man7/capabilities.7.html`_.
+.. [lxcsecurity] `Linux Containers Security <https://linuxcontainers.org/lxc/security/>`_.
+.. [capabilities] `Linux manual - capabilities - overview of Linux capabilities <http://man7.org/linux/man-pages/man7/capabilities.7.html>`_.
.. [cgroup1] `Linux kernel documentation: cgroups <https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt>`_.
.. [cgroup2] `Linux kernel documentation: Control Group v2 <https://www.kernel.org/doc/Documentation/cgroup-v2.txt>`_.
.. [selinux] `SELinux Project Wiki <http://selinuxproject.org/page/Main_Page>`_.
-.. [lxc-sec-features] `LXC 1.0: Security features <https://stgraber.org/2014/01/01/lxc-1-0-security-features/>`_.
-.. [lxc-source] `Linux Containers source <https://github.com/lxc/lxc>`_.
+.. [lxcsecfeatures] `LXC 1.0: Security features <https://stgraber.org/2014/01/01/lxc-1-0-security-features/>`_.
+.. [lxcsource] `Linux Containers source <https://github.com/lxc/lxc>`_.
.. [apparmor] `Ubuntu AppArmor <https://wiki.ubuntu.com/AppArmor>`_.
.. [seccomp] `SECure COMPuting with filters <https://www.kernel.org/doc/Documentation/prctl/seccomp_filter.txt>`_.
.. [docker] `Docker <https://www.docker.com/what-docker>`_.
-.. [k8s-doc] `Kubernetes documentation <https://kubernetes.io/docs/home/>`_.
-.. [ligato] `Ligato <https://github.com/ligato>`_.
-.. [govpp] `FD.io goVPP project <https://wiki.fd.io/view/GoVPP>`_.
-.. [vpp-agent] `Ligato vpp-agent <https://github.com/ligato/vpp-agent>`_.
-.. [image-var] Image parameter is required in initial commit version. There is plan to implement container build class to build Docker/LXC image.
+.. [k8sdoc] `Kubernetes documentation <https://kubernetes.io/docs/home/>`_.
Code Review / csit.git / blobdiff