+++ /dev/null
-.. _autogen_methodology:
-
-Suite Generation
-^^^^^^^^^^^^^^^^
-
-CSIT uses robot suite files to define tests.
-However, not all suite files available for Jenkins jobs
-(or manually started bootstrap scripts) are present in CSIT git repository.
-They are generated only when needed.
-
-Autogen Library
-~~~~~~~~~~~~~~~
-
-There is a code generation layer implemented as Python library called "autogen",
-called by various bash scripts.
-
-It generates the full extent of CSIT suites, using the ones in git as templates.
-
-Sources
-~~~~~~~
-
-The generated suites (and their contents) are affected by multiple information
-sources, listed below.
-
-Git Suites
-``````````
-
-The suites present in git repository act as templates for generating suites.
-One of autogen design principles is that any template suite should also act
-as a full suite (no placeholders).
-
-In practice, autogen always re-creates the template suite with exactly
-the same content, it is one of checks that autogen works correctly.
-
-Regenerate Script
-`````````````````
-
-Not all suites present in CSIT git repository act as template for autogen.
-The distinction is on per-directory level. Directories with
-regenerate_testcases.py script usually consider all suites as templates
-(unless possibly not included by the glob patten in the script).
-
-The script also specifies minimal frame size, indirectly, by specifying protocol
-(protocol "ip4" is the default, leading to 64B frame size).
-
-Constants
-`````````
-
-Values in Constants.py are taken into consideration when generating suites.
-The values are mostly related to different NIC models and NIC drivers.
-
-Python Code
-```````````
-
-Python code in resources/libraries/python/autogen contains several other
-information sources.
-
-Testcase Templates
-__________________
-
-The test case part of template suite is ignored, test case lines
-are created according to text templates in Testcase.py file.
-
-Testcase Argument Lists
-_______________________
-
-Each testcase template has different number of "arguments", e.g. values
-to put into various placeholders. Different test types need different
-lists of the argument values, the lists are in regenerate_glob method
-in Regenerator.py file.
-
-Iteration Over Values
-_____________________
-
-Python code detects the test type (usually by substrings of suite file name),
-then iterates over different quantities based on type.
-For example, only ndrpdr suite templates generate other types (mrr and soak).
-
-Hardcoded Exclusions
-____________________
-
-Some combinations of values are known not to work, so they are excluded.
-Examples: Density tests for too much CPUs; IMIX for ASTF.
-
-Non-Sources
-~~~~~~~~~~~
-
-Some information sources are available in CSIT repository,
-but do not affect the suites generated by autogen.
-
-Testbeds
-````````
-
-Overall, no information visible in topology yaml files is taken into account
-by autogen.
-
-Testbed Architecture
-____________________
-
-Historically, suite files are agnostic to testbed architecture, e.g. ICX or ALT.
-
-Testbed Size
-____________
-
-Historically, 2-node and 3-node suites have diferent names, and while
-most of the code is common, the differences are not always simple enough.
-Autogen treat 2-node and 3-node suites as independent templates.
-
-TRex suites are intended for a 1-node circuit of otherwise 2-node or 3-node
-testbeds, so they support all 3 robot tags.
-They are also detected and treated differently by autogen,
-mainly because they need different testcase arguments (no CPU count).
-Autogen does nothing specifically related to the fact they should run
-only in testbeds/NICs with TG-TG line available.
-
-Other Topology Info
-___________________
-
-Some bonding tests need two (parallel) links between DUTs.
-Autogen does not care, as suites are agnostic.
-Robot tag marks the difference, but the link presence is not explicitly checked.
-
-Job specs
-`````````
-
-Information in job spec files depend on generated suites (not the other way).
-Autogen should generate more suites, as job spec is limited by time budget.
-More suites should be available for manually triggered verify jobs,
-so autogen covers that.
-
-Bootstrap Scripts
-`````````````````
-
-Historically, bootstrap scripts perform some logic,
-perhaps adding exclusion options to Robot invocation
-(e.g. skipping testbed+NIC combinations for tests that need parallel links).
-
-Once again, the logic here relies on what autogen generates,
-autogen does not look into bootstrap scripts.
Code Review / csit.git / blobdiff