1 Host stack test framework
2 =========================
7 The goal of the Host stack test framework (**hs-test**) is to ease writing and running end-to-end tests for VPP's Host Stack.
8 End-to-end tests often want multiple VPP instances, network namespaces, different types of interfaces
9 and to execute external tools or commands. With such requirements the existing VPP test framework is not sufficient.
10 For this, ``Go`` was chosen as a high level language, allowing rapid development, with ``Docker`` and ``ip`` being the tools for creating required topology.
12 `Ginkgo`_ forms the base framework upon which the *hs-test* is built and run.
13 All tests are technically in a single suite because we are only using ``package main``. We simulate suite behavior by grouping tests by the topology they require.
14 This allows us to run those mentioned groups in parallel, but not individual tests in parallel.
17 Anatomy of a test case
18 ----------------------
22 * Install hs-test dependencies with ``make install-deps``
23 * Tests use *hs-test*'s own docker image, so building it before starting tests is a prerequisite. Run ``make build[-debug]`` to do so
24 * Docker has to be installed and Go has to be in path of both the running user and root
25 * Root privileges are required to run tests as it uses Linux ``ip`` command for configuring topology
27 **Action flow when running a test case**:
29 #. It starts with running ``make test``. Optional arguments are VERBOSE, PERSIST (topology configuration isn't cleaned up after test run, use ``make cleanup-hst`` to clean up),
30 TEST=<test-name> to run a specific test and PARALLEL=[n-cpus].
31 #. ``make list-tests`` (or ``make help``) shows all tests.
32 #. ``Ginkgo`` looks for a spec suite in the current directory and then compiles it to a .test binary.
33 #. The Ginkgo test framework runs each function that was registered manually using ``Register[SuiteName]Test()``. Each of these functions correspond to a suite.
34 #. Ginkgo's ``RunSpecs(t, "Suite description")`` function is the entry point and does the following:
36 #. Ginkgo compiles the spec, builds a spec tree
37 #. ``Describe`` container nodes in suite\_\*.go files are run (in series by default, or in parallel with the argument PARALLEL=[n-cpus])
38 #. Suite is initialized. The topology is loaded and configured in this step
39 #. Registered tests are run in generated ``It`` subject nodes
40 #. Execute tear-down functions, which currently consists of stopping running containers
41 and clean-up of test topology
46 This describes adding a new test case to an existing suite.
47 For adding a new suite, please see `Modifying the framework`_ below.
49 #. To write a new test case, create a file whose name ends with ``_test.go`` or pick one that already exists
50 #. Declare method whose name ends with ``Test`` and specifies its parameter as a pointer to the suite's struct (defined in ``infra/suite_*.go``)
51 #. Implement test behaviour inside the test method. This typically includes the following:
53 #. Import ``. "fd.io/hs-test/infra"``
54 #. Retrieve a running container in which to run some action. Method ``GetContainerByName``
55 from ``HstSuite`` struct serves this purpose
56 #. Interact with VPP through the ``VppInstance`` struct embedded in container. It provides ``Vppctl`` method to access debug CLI
57 #. Run arbitrary commands inside the containers with ``Exec`` method
58 #. Run other external tool with one of the preexisting functions in the ``infra/utils.go`` file.
59 For example, use ``wget`` with ``StartWget`` function
60 #. Use ``exechelper`` or just plain ``exec`` packages to run whatever else
61 #. Verify results of your tests using ``Assert`` methods provided by the test suite.
63 #. Create an ``init()`` function and register the test using ``Register[SuiteName]Tests(testCaseFunction)``
68 Assumed are two docker containers, each with its own VPP instance running. One VPP then pings the other.
69 This can be put in file ``extras/hs-test/my_test.go`` and run with command ``make test TEST=MyTest``.
76 . "fd.io/hs-test/infra"
80 RegisterMySuiteTest(MyTest)
83 func MyTest(s *MySuite) {
84 clientVpp := s.GetContainerByName("client-vpp").VppInstance
86 serverVethAddress := s.NetInterfaces["server-iface"].Ip4AddressString()
88 result := clientVpp.Vppctl("ping " + serverVethAddress)
90 s.AssertNotNil(result)
97 The framework allows us to filter test cases in a few different ways, using ``make test TEST=``:
102 * All of the above as long as they are ordered properly, e.g. ``make test TEST=VethsSuite.http_test.go.HeaderServerTest``
104 **Names are case sensitive!**
106 Names don't have to be complete, as long as they are last:
107 This is valid and will run all tests in every ``http`` file (if there is more than one):
109 * ``make test TEST=VethsSuite.http``
113 * ``make test TEST=Veths.http``
115 They can also be left out:
117 * ``make test TEST=http_test.go`` will run every test in ``http_test.go``
118 * ``make test TEST=Nginx`` will run everything that has 'Nginx' in its name - suites, files and tests.
119 * ``make test TEST=HeaderServerTest`` will only run the header server test
122 Modifying the framework
123 -----------------------
125 **Adding a test suite**
129 #. To add a new suite, create a new file in the ``infra/`` folder. Naming convention for the suite files is ``suite_[name].go``.
131 #. Make a ``struct``, in the suite file, with at least ``HstSuite`` struct as its member.
132 HstSuite provides functionality that can be shared for all suites, like starting containers
134 #. Create a new map that will contain a file name where a test is located and test functions with a pointer to the suite's struct: ``var myTests = map[string][]func(s *MySuite){}``
138 var myTests = map[string][]func(s *MySuite){}
140 type MySuite struct {
145 #. Then create a new function that will add tests to that map:
149 func RegisterMyTests(tests ...func(s *MySuite)) {
150 myTests[getTestFilename()] = tests
154 #. In suite file, implement ``SetupSuite`` method which Ginkgo runs once before starting any of the tests.
155 It's important here to call ``ConfigureNetworkTopology()`` method,
156 pass the topology name to the function in a form of file name of one of the *yaml* files in ``topo-network`` folder.
157 Without the extension. In this example, *myTopology* corresponds to file ``extras/hs-test/topo-network/myTopology.yaml``
158 This will ensure network topology, such as network interfaces and namespaces, will be created.
159 Another important method to call is ``LoadContainerTopology()`` which will load
160 containers and shared volumes used by the suite. This time the name passed to method corresponds
161 to file in ``extras/hs-test/topo-containers`` folder
165 func (s *MySuite) SetupSuite() {
166 s.HstSuite.SetupSuite()
168 // Add custom setup code here
170 s.ConfigureNetworkTopology("myTopology")
171 s.LoadContainerTopology("2peerVeth")
174 #. In suite file, implement ``SetupTest`` method which gets executed before each test. Starting containers and
175 configuring VPP is usually placed here
179 func (s *MySuite) SetupTest() {
180 s.HstSuite.setupTest()
185 #. In order for ``Ginkgo`` to run this suite, we need to create a ``Describe`` container node with setup nodes and an ``It`` subject node.
186 Place them at the end of the suite file
188 * Declare a suite struct variable before anything else
189 * To use ``BeforeAll()`` and ``AfterAll()``, the container has to be marked as ``Ordered``
190 * Because the container is now marked as Ordered, if a test fails, all the subsequent tests are skipped.
191 To override this behavior, decorate the container node with ``ContinueOnFailure``
195 var _ = Describe("MySuite", Ordered, ContinueOnFailure, func() {
210 for filename, tests := range myTests {
211 for _, test := range tests {
213 pc := reflect.ValueOf(test).Pointer()
214 funcValue := runtime.FuncForPC(pc)
215 testName := filename + "/" + strings.Split(funcValue.Name(), ".")[2]
216 It(testName, func(ctx SpecContext) {
217 s.Log(testName + ": BEGIN")
219 }, SpecTimeout(SuiteTimeout))
224 #. Notice the loop - it will generate multiple ``It`` nodes, each running a different test.
225 ``test := test`` is necessary, otherwise only the last test in a suite will run.
226 For a more detailed description, check Ginkgo's documentation: https://onsi.github.io/ginkgo/#dynamically-generating-specs\.
228 #. ``testName`` contains the test name in the following format: ``[name]_test.go/MyTest``.
230 #. To run certain tests solo, create a register function and a map that will only contain tests that have to run solo.
231 Add a ``Serial`` decorator to the container node and ``Label("SOLO")`` to the ``It`` subject node:
235 var _ = Describe("MySuiteSolo", Ordered, ContinueOnFailure, Serial, func() {
237 It(testName, Label("SOLO"), func(ctx SpecContext) {
238 s.Log(testName + ": BEGIN")
240 }, SpecTimeout(time.Minute*5))
243 #. Next step is to add test cases to the suite. For that, see section `Adding a test case`_ above
245 **Adding a topology element**
247 Topology configuration exists as ``yaml`` files in the ``extras/hs-test/topo-network`` and
248 ``extras/hs-test/topo-containers`` folders. Processing of a network topology file for a particular test suite
249 is started by the ``configureNetworkTopology`` method depending on which file's name is passed to it.
250 Specified file is loaded and converted into internal data structures which represent various elements of the topology.
251 After parsing the configuration, framework loops over the elements and configures them one by one on the host system.
253 These are currently supported types of network elements.
255 * ``netns`` - network namespace
256 * ``veth`` - veth network interface, optionally with target network namespace or IPv4 address
257 * ``bridge`` - ethernet bridge to connect created interfaces, optionally with target network namespace
258 * ``tap`` - tap network interface with IP address
260 Similarly, container topology is started by ``loadContainerTopology()``, configuration file is processed
261 so that test suite retains map of defined containers and uses that to start them at the beginning
262 of each test case and stop containers after the test finishes. Container configuration can specify
263 also volumes which allow to share data between containers or between host system and containers.
265 Supporting a new type of topology element requires adding code to recognize the new element type during loading.
266 And adding code to set up the element in the host system with some Linux tool, such as *ip*.
267 This should be implemented in ``netconfig.go`` for network and in ``container.go`` for containers and volumes.
269 **Communicating between containers**
271 When two VPP instances or other applications, each in its own Docker container,
272 want to communicate there are typically two ways this can be done within *hs-test*.
274 * Network interfaces. Containers are being created with ``-d --network host`` options,
275 so they are connected with interfaces created in host system
276 * Shared folders. Containers are being created with ``-v`` option to create shared `volumes`_ between host system and containers
277 or just between containers
279 Host system connects to VPP instances running in containers using a shared folder
280 where binary API socket is accessible by both sides.
282 **Adding an external tool**
284 If an external program should be executed as part of a test case, it might be useful to wrap its execution in its own function.
285 These types of functions are placed in the ``utils.go`` file. If the external program is not available by default in Docker image,
286 add its installation to ``extras/hs-test/Dockerfile.vpp`` in ``apt-get install`` command.
287 Alternatively copy the executable from host system to the Docker image, similarly how the VPP executables and libraries are being copied.
291 ``HstSuite`` provides several methods that can be called in tests for skipping it conditionally or unconditionally such as:
292 ``skip()``, ``SkipIfMultiWorker()``, ``SkipUnlessExtendedTestsBuilt()``. You can also use Ginkgo's ``Skip()``.
293 However the tests currently run under test suites which set up topology and containers before actual test is run. For the reason of saving
294 test run time it is not advisable to use aforementioned skip methods and instead, just don't register the test.
298 It is possible to debug VPP by attaching ``gdb`` before test execution by adding ``DEBUG=true`` like follows:
302 $ make test TEST=LDPreloadIperfVppTest DEBUG=true
304 run following command in different terminal:
305 docker exec -it server-vpp2456109 gdb -ex "attach $(docker exec server-vpp2456109 pidof vpp)"
306 Afterwards press CTRL+\ to continue
308 If a test consists of more VPP instances then this is done for each of them.
311 **Eternal dependencies**
313 * Linux tools ``ip``, ``brctl``
314 * Standalone programs ``wget``, ``iperf3`` - since these are downloaded when Docker image is made,
315 they are reasonably up-to-date automatically
316 * Programs in Docker images - ``envoyproxy/envoy-contrib`` and ``nginx``
317 * ``http_server`` - homegrown application that listens on specified port and sends a test file in response
318 * Non-standard Go libraries - see ``extras/hs-test/go.mod``
320 Generally, these will be updated on a per-need basis, for example when a bug is discovered
321 or a new version incompatibility issue occurs.
324 .. _ginkgo: https://onsi.github.io/ginkgo/
325 .. _volumes: https://docs.docker.com/storage/volumes/