3 title: "Suite Generation"
9 CSIT uses robot suite files to define tests.
10 However, not all suite files available for Jenkins jobs
11 (or manually started bootstrap scripts) are present in CSIT git repository.
12 They are generated only when needed.
16 There is a code generation layer implemented as Python library called "autogen",
17 called by various bash scripts.
19 It generates the full extent of CSIT suites, using the ones in git as templates.
23 The generated suites (and their contents) are affected by multiple information
24 sources, listed below.
28 The suites present in git repository act as templates for generating suites.
29 One of autogen design principles is that any template suite should also act
30 as a full suite (no placeholders).
32 In practice, autogen always re-creates the template suite with exactly
33 the same content, it is one of checks that autogen works correctly.
37 Not all suites present in CSIT git repository act as template for autogen.
38 The distinction is on per-directory level. Directories with
39 regenerate_testcases.py script usually consider all suites as templates
40 (unless possibly not included by the glob patten in the script).
42 The script also specifies minimal frame size, indirectly, by specifying protocol
43 (protocol "ip4" is the default, leading to 64B frame size).
47 Values in Constants.py are taken into consideration when generating suites.
48 The values are mostly related to different NIC models and NIC drivers.
52 Python code in resources/libraries/python/autogen contains several other
55 #### Testcase Templates
57 The test case part of template suite is ignored, test case lines
58 are created according to text templates in Testcase.py file.
60 #### Testcase Argument Lists
62 Each testcase template has different number of "arguments", e.g. values
63 to put into various placeholders. Different test types need different
64 lists of the argument values, the lists are in regenerate_glob method
65 in Regenerator.py file.
67 #### Iteration Over Values
69 Python code detects the test type (usually by substrings of suite file name),
70 then iterates over different quantities based on type.
71 For example, only ndrpdr suite templates generate other types (mrr and soak).
73 #### Hardcoded Exclusions
75 Some combinations of values are known not to work, so they are excluded.
76 Examples: Density tests for too much CPUs; IMIX for ASTF.
80 Some information sources are available in CSIT repository,
81 but do not affect the suites generated by autogen.
85 Overall, no information visible in topology yaml files is taken into account
88 #### Testbed Architecture
90 Historically, suite files are agnostic to testbed architecture, e.g. ICX or ALT.
94 Historically, 2-node and 3-node suites have diferent names, and while
95 most of the code is common, the differences are not always simple enough.
96 Autogen treat 2-node and 3-node suites as independent templates.
98 TRex suites are intended for a 1-node circuit of otherwise 2-node or 3-node
99 testbeds, so they support all 3 robot tags.
100 They are also detected and treated differently by autogen,
101 mainly because they need different testcase arguments (no CPU count).
102 Autogen does nothing specifically related to the fact they should run
103 only in testbeds/NICs with TG-TG line available.
105 #### Other Topology Info
107 Some bonding tests need two (parallel) links between DUTs.
108 Autogen does not care, as suites are agnostic.
109 Robot tag marks the difference, but the link presence is not explicitly checked.
113 Information in job spec files depend on generated suites (not the other way).
114 Autogen should generate more suites, as job spec is limited by time budget.
115 More suites should be available for manually triggered verify jobs,
116 so autogen covers that.
118 ### Bootstrap Scripts
120 Historically, bootstrap scripts perform some logic,
121 perhaps adding exclusion options to Robot invocation
122 (e.g. skipping testbed+NIC combinations for tests that need parallel links).
124 Once again, the logic here relies on what autogen generates,
125 autogen does not look into bootstrap scripts.