hs-test: transition to ginkgo test framework

[vpp.git] / extras / hs-test / README.rst

diff --git a/extras/hs-test/README.rst b/extras/hs-test/README.rst
index 6db832b..1dc1039 100644 (file)
--- a/extras/hs-test/README.rst
+++ b/extras/hs-test/README.rst
@@ -9,7 +9,10 @@ End-to-end tests often want multiple VPP instances, network namespaces, differen
 and to execute external tools or commands. With such requirements the existing VPP test framework is not sufficient.
 For this, ``Go`` was chosen as a high level language, allowing rapid development, with ``Docker`` and ``ip`` being the tools for creating required topology.
 
 and to execute external tools or commands. With such requirements the existing VPP test framework is not sufficient.
 For this, ``Go`` was chosen as a high level language, allowing rapid development, with ``Docker`` and ``ip`` being the tools for creating required topology.
 

-Go's package `testing`_ together with `go test`_ command form the base framework upon which the *hs-test* is built and run.
+`Ginkgo`_ forms the base framework upon which the *hs-test* is built and run.
+All tests are technically in a single suite because we are only using ``package main``. We simulate suite behavior by grouping tests by the topology they require.
+This allows us to run those mentioned groups in parallel, but not individual tests in parallel.
+

 
 Anatomy of a test case
 ----------------------
 
 Anatomy of a test case
 ----------------------


@@ -24,15 +27,16 @@ Anatomy of a test case
 **Action flow when running a test case**:
 
 #. It starts with running ``make test``. Optional arguments are VERBOSE, PERSIST (topology configuration isn't cleaned up after test run),
 **Action flow when running a test case**:
 
 #. It starts with running ``make test``. Optional arguments are VERBOSE, PERSIST (topology configuration isn't cleaned up after test run),

-   and TEST=<test-name> to run specific test.
-#. ``make list-tests`` (or ``make help``) shows all test names.
-#. ``go test`` compiles package ``main`` along with any files with names matching the file pattern ``*_test.go``
-   and then runs the resulting test binaries
-#. The go test framework runs each function matching :ref:`naming convention<test-convention>`. Each of these corresponds to a `test suite`_
-#. Testify toolkit's ``suite.Run(t *testing.T, suite TestingSuite)`` function runs the suite and does the following:
-
+   TEST=<test-name> to run a specific test and PARALLEL=[n-cpus].
+#. ``make list-tests`` (or ``make help``) shows all tests. The current `list of tests`_ is at the bottom of this document.
+#. ``Ginkgo`` looks for a spec suite in the current directory and then compiles it to a .test binary
+#. The Ginkgo test framework runs each function that was registered manually using ``registerMySuiteTest(s *MySuite)``. Each of these functions correspond to a suite
+#. Ginkgo's ``RunSpecs(t, "Suite description")`` function is the entry point and does the following:
+
+  #. Ginkgo compiles the spec, builds a spec tree
+  #. ``Describe`` container nodes in suite\_\*_test.go files are run (in series by default, or in parallel with the argument PARALLEL=[n-cpus])

   #. Suite is initialized. The topology is loaded and configured in this step
   #. Suite is initialized. The topology is loaded and configured in this step

-  #. Test suite runs all the tests attached to it
+  #. Registered tests are run in generated ``It`` subject nodes

   #. Execute tear-down functions, which currently consists of stopping running containers
      and clean-up of test topology
 
   #. Execute tear-down functions, which currently consists of stopping running containers
      and clean-up of test topology
 


@@ -43,23 +47,25 @@ This describes adding a new test case to an existing suite.
 For adding a new suite, please see `Modifying the framework`_ below.
 
 #. To write a new test case, create a file whose name ends with ``_test.go`` or pick one that already exists
 For adding a new suite, please see `Modifying the framework`_ below.
 
 #. To write a new test case, create a file whose name ends with ``_test.go`` or pick one that already exists

-#. Declare method whose name starts with ``Test`` and specifies its receiver as a pointer to the suite's struct (defined in ``framework_test.go``)
+#. Declare method whose name ends with ``Test`` and specifies its parameter as a pointer to the suite's struct (defined in ``suite_*_test.go``)

 #. Implement test behaviour inside the test method. This typically includes the following:
 
 #. Implement test behaviour inside the test method. This typically includes the following:
 

-  #. Retrieve a running container in which to run some action. Method ``getContainerByName``
-     from ``HstSuite`` struct serves this purpose
-  #. Interact with VPP through the ``VppInstance`` struct embedded in container. It provides ``vppctl`` method to access debug CLI
-  #. Run arbitrary commands inside the containers with ``exec`` method
-  #. Run other external tool with one of the preexisting functions in the ``utils.go`` file.
-     For example, use ``wget`` with ``startWget`` function
-  #. Use ``exechelper`` or just plain ``exec`` packages to run whatever else
-  #. Verify results of your tests using ``assert`` methods provided by the test suite,
-     implemented by HstSuite struct
+   #. Retrieve a running container in which to run some action. Method ``getContainerByName``
+      from ``HstSuite`` struct serves this purpose
+   #. Interact with VPP through the ``VppInstance`` struct embedded in container. It provides ``vppctl`` method to access debug CLI
+   #. Run arbitrary commands inside the containers with ``exec`` method
+   #. Run other external tool with one of the preexisting functions in the ``utils.go`` file.
+      For example, use ``wget`` with ``startWget`` function
+   #. Use ``exechelper`` or just plain ``exec`` packages to run whatever else
+   #. Verify results of your tests using ``assert`` methods provided by the test suite, implemented by HstSuite struct or use ``Gomega`` assert functions.
+
+#. Create an ``init()`` function and register the test using ``register*SuiteTests(testCaseFunction)``
+

 
 **Example test case**
 
 Assumed are two docker containers, each with its own VPP instance running. One VPP then pings the other.
 
 **Example test case**
 
 Assumed are two docker containers, each with its own VPP instance running. One VPP then pings the other.

-This can be put in file ``extras/hs-test/my_test.go`` and run with command ``./test -run TestMySuite/TestMyCase``.
+This can be put in file ``extras/hs-test/my_test.go`` and run with command ``make test TEST=MyTest`` or ``ginkgo -v --trace --focus MyTest``.

 
 ::
 
 
 ::
 


@@ -69,7 +75,11 @@ This can be put in file ``extras/hs-test/my_test.go`` and run with command ``./t
                 "fmt"
         )
 
                 "fmt"
         )
 

-        func (s *MySuite) TestMyCase() {
+        func init(){
+                registerMySuiteTest(MyTest)
+        }
+
+        func MyTest(s *MySuite) {

                 clientVpp := s.getContainerByName("client-vpp").vppInstance
 
                 serverVethAddress := s.netInterfaces["server-iface"].AddressString()
                 clientVpp := s.getContainerByName("client-vpp").vppInstance
 
                 serverVethAddress := s.netInterfaces["server-iface"].AddressString()


@@ -86,8 +96,7 @@ Modifying the framework
 
 .. _test-convention:
 
 
 .. _test-convention:
 

-#. Adding a new suite takes place in ``framework_test.go`` and by creating a new file for the suite.
-   Naming convention for the suite files is ``suite_name_test.go`` where *name* will be replaced
+#. To add a new suite, create a new file. Naming convention for the suite files is ``suite_name_test.go`` where *name* will be replaced

    by the actual name
 
 #. Make a ``struct``, in the suite file, with at least ``HstSuite`` struct as its member.
    by the actual name
 
 #. Make a ``struct``, in the suite file, with at least ``HstSuite`` struct as its member.


@@ -99,7 +108,17 @@ Modifying the framework
                         HstSuite
                 }
 
                         HstSuite
                 }
 

-#. In suite file, implement ``SetupSuite`` method which testify runs once before starting any of the tests.
+#. Create a new slice that will contain test functions with a pointer to the suite's struct: ``var myTests = []func(s *MySuite){}``
+
+#. Then create a new function that will append test functions to that slice:
+
+        ::
+
+                func registerMySuiteTests(tests ...func(s *MySuite)) {
+                       nginxTests = append(myTests, tests...)
+                }
+
+#. In suite file, implement ``SetupSuite`` method which Ginkgo runs once before starting any of the tests.

    It's important here to call ``configureNetworkTopology`` method,
    pass the topology name to the function in a form of file name of one of the *yaml* files in ``topo-network`` folder.
    Without the extension. In this example, *myTopology* corresponds to file ``extras/hs-test/topo-network/myTopology.yaml``
    It's important here to call ``configureNetworkTopology`` method,
    pass the topology name to the function in a form of file name of one of the *yaml* files in ``topo-network`` folder.
    Without the extension. In this example, *myTopology* corresponds to file ``extras/hs-test/topo-network/myTopology.yaml``


@@ -111,6 +130,8 @@ Modifying the framework
         ::
 
                 func (s *MySuite) SetupSuite() {
         ::
 
                 func (s *MySuite) SetupSuite() {

+                        s.HstSuite.SetupSuite()
+

                         // Add custom setup code here
 
                         s.configureNetworkTopology("myTopology")
                         // Add custom setup code here
 
                         s.configureNetworkTopology("myTopology")


@@ -123,19 +144,62 @@ Modifying the framework
         ::
 
                 func (s *MySuite) SetupTest() {
         ::
 
                 func (s *MySuite) SetupTest() {

+                        s.HstSuite.setupTest()

                         s.SetupVolumes()
                         s.SetupContainers()
                 }
 
                         s.SetupVolumes()
                         s.SetupContainers()
                 }
 

-#. In order for ``go test`` to run this suite, we need to create a normal test function and pass our suite to ``suite.Run``.
-   These functions are placed at the end of ``framework_test.go``
+#. In order for ``Ginkgo`` to run this suite, we need to create a ``Describe`` container node with setup nodes and an ``It`` subject node.
+   Place them at the end of the suite file
+
+   * Declare a suite struct variable before anything else
+   * To use ``BeforeAll()`` and ``AfterAll()``, the container has to be marked as ``Ordered``
+   * Because the container is now marked as Ordered, if a test fails, all the subsequent tests are skipped.
+     To override this behavior, decorate the container node with ``ContinueOnFailure``

 
         ::
 
 
         ::
 

-                func TestMySuite(t *testing.T) {
-                        var m MySuite
-                        suite.Run(t, &m)
-                }
+                var _ = Describe("MySuite", Ordered, ContinueOnFailure, func() {
+               var s MySuite
+               BeforeAll(func() {
+                       s.SetupSuite()
+               })
+               BeforeEach(func() {
+                       s.SetupTest()
+               })
+               AfterAll(func() {
+                       s.TearDownSuite()
+               })
+               AfterEach(func() {
+                       s.TearDownTest()
+               })
+               for _, test := range mySuiteTests {
+                       test := test
+                       pc := reflect.ValueOf(test).Pointer()
+                       funcValue := runtime.FuncForPC(pc)
+                       It(strings.Split(funcValue.Name(), ".")[2], func(ctx SpecContext) {
+                               test(&s)
+                       }, SpecTimeout(time.Minute*5))
+               }
+                })
+
+#. Notice the loop - it will generate multiple ``It`` nodes, each running a different test.
+   ``test := test`` is necessary, otherwise only the last test in a suite will run.
+   For a more detailed description, check Ginkgo's documentation: https://onsi.github.io/ginkgo/#dynamically-generating-specs\.
+
+#. ``funcValue.Name()`` returns the full name of a function (e.g. ``fd.io/hs-test.MyTest``), however, we only need the test name (``MyTest``).
+
+#. To run certain tests solo, create a new slice that will only contain tests that have to run solo and a new register function.
+   Add a ``Serial`` decorator to the container node and ``Label("SOLO")`` to the ``It`` subject node:
+
+        ::
+
+                var _ = Describe("MySuiteSolo", Ordered, ContinueOnFailure, Serial, func() {
+                        ...
+                        It(strings.Split(funcValue.Name(), ".")[2], Label("SOLO"), func(ctx SpecContext) {
+                       test(&s)
+                       }, SpecTimeout(time.Minute*5))
+                })

 
 #. Next step is to add test cases to the suite. For that, see section `Adding a test case`_ above
 
 
 #. Next step is to add test cases to the suite. For that, see section `Adding a test case`_ above
 


@@ -186,14 +250,9 @@ Alternatively copy the executable from host system to the Docker image, similarl
 **Skipping tests**
 
 ``HstSuite`` provides several methods that can be called in tests for skipping it conditionally or unconditionally such as:
 **Skipping tests**
 
 ``HstSuite`` provides several methods that can be called in tests for skipping it conditionally or unconditionally such as:

-``skip()``, ``SkipIfMultiWorker()``, ``SkipUnlessExtendedTestsBuilt()``.
+``skip()``, ``SkipIfMultiWorker()``, ``SkipUnlessExtendedTestsBuilt()``. You can also use Ginkgo's ``Skip()``.

 However the tests currently run under test suites which set up topology and containers before actual test is run. For the reason of saving
 However the tests currently run under test suites which set up topology and containers before actual test is run. For the reason of saving

-test run time it is not advisable to use aforementioned skip methods and instead prefix test name with ``Skip``:
-
-::
-
-    func (s *MySuite) SkipTest(){
-
+test run time it is not advisable to use aforementioned skip methods and instead, just don't register the test.

 
 **Debugging a test**
 
 
 **Debugging a test**
 


@@ -201,11 +260,11 @@ It is possible to debug VPP by attaching ``gdb`` before test execution by adding
 
 ::
 
 
 ::
 

-    $ make test TEST=TestVeths/TestLDPreloadIperfVpp DEBUG=true
+    $ make test TEST=LDPreloadIperfVppTest DEBUG=true

     ...
     run following command in different terminal:
     ...
     run following command in different terminal:

-    docker exec -it server-vpp gdb -ex "attach $(docker exec server-vpp pidof vpp)"
-    Afterwards press CTRL+C to continue
+    docker exec -it server-vpp2456109 gdb -ex "attach $(docker exec server-vpp2456109 pidof vpp)"
+    Afterwards press CTRL+\ to continue

 
 If a test consists of more VPP instances then this is done for each of them.
 
 
 If a test consists of more VPP instances then this is done for each of them.
 


@@ -223,8 +282,38 @@ Generally, these will be updated on a per-need basis, for example when a bug is
 or a new version incompatibility issue occurs.
 
 
 or a new version incompatibility issue occurs.
 
 

-.. _testing: https://pkg.go.dev/testing
-.. _go test: https://pkg.go.dev/cmd/go#hdr-Test_packages
-.. _test suite: https://github.com/stretchr/testify#suite-package
+.. _ginkgo: https://onsi.github.io/ginkgo/

 .. _volumes: https://docs.docker.com/storage/volumes/
 
 .. _volumes: https://docs.docker.com/storage/volumes/
 

+**List of tests**
+
+.. _list of tests:
+
+Please update this list whenever you add a new test by pasting the output below.
+
+* NsSuite/HttpTpsTest
+* NsSuite/VppProxyHttpTcpTest
+* NsSuite/VppProxyHttpTlsTest
+* NsSuite/EnvoyProxyHttpTcpTest
+* NginxSuite/MirroringTest
+* VethsSuiteSolo TcpWithLossTest [SOLO]
+* NoTopoSuiteSolo HttpStaticPromTest [SOLO]
+* TapSuite/LinuxIperfTest
+* NoTopoSuite/NginxHttp3Test
+* NoTopoSuite/NginxAsServerTest
+* NoTopoSuite/NginxPerfCpsTest
+* NoTopoSuite/NginxPerfRpsTest
+* NoTopoSuite/NginxPerfWrkTest
+* VethsSuite/EchoBuiltinTest
+* VethsSuite/HttpCliTest
+* VethsSuite/LDPreloadIperfVppTest
+* VethsSuite/VppEchoQuicTest
+* VethsSuite/VppEchoTcpTest
+* VethsSuite/VppEchoUdpTest
+* VethsSuite/XEchoVclClientUdpTest
+* VethsSuite/XEchoVclClientTcpTest
+* VethsSuite/XEchoVclServerUdpTest
+* VethsSuite/XEchoVclServerTcpTest
+* VethsSuite/VclEchoTcpTest
+* VethsSuite/VclEchoUdpTest
+* VethsSuite/VclRetryAttachTest