X-Git-Url: https://gerrit.fd.io/r/gitweb?a=blobdiff_plain;f=docs%2Fcontent%2Fmethodology%2Fsuite_generation.md;fp=docs%2Fcontent%2Fmethodology%2Fsuite_generation.md;h=0000000000000000000000000000000000000000;hb=374954b9d648f503f6783325a1266457953a998d;hp=4fa9dee0ce141c56ec7adbaf913caee45f76bc5a;hpb=46eac7bb697e8261dba5b439a15f5a6125f31760;p=csit.git diff --git a/docs/content/methodology/suite_generation.md b/docs/content/methodology/suite_generation.md deleted file mode 100644 index 4fa9dee0ce..0000000000 --- a/docs/content/methodology/suite_generation.md +++ /dev/null @@ -1,124 +0,0 @@ ---- -title: "Suite Generation" -weight: 19 ---- - -# Suite Generation - -CSIT uses robot suite files to define tests. -However, not all suite files available for Jenkins jobs -(or manually started bootstrap scripts) are present in CSIT git repository. -They are generated only when needed. - -## Autogen Library - -There is a code generation layer implemented as Python library called "autogen", -called by various bash scripts. - -It generates the full extent of CSIT suites, using the ones in git as templates. - -## Sources - -The generated suites (and their contents) are affected by multiple information -sources, listed below. - -### Git Suites - -The suites present in git repository act as templates for generating suites. -One of autogen design principles is that any template suite should also act -as a full suite (no placeholders). - -In practice, autogen always re-creates the template suite with exactly -the same content, it is one of checks that autogen works correctly. - -### Regenerate Script - -Not all suites present in CSIT git repository act as template for autogen. -The distinction is on per-directory level. Directories with -regenerate_testcases.py script usually consider all suites as templates -(unless possibly not included by the glob patten in the script). - -The script also specifies minimal frame size, indirectly, by specifying protocol -(protocol "ip4" is the default, leading to 64B frame size). - -### Constants - -Values in Constants.py are taken into consideration when generating suites. -The values are mostly related to different NIC models and NIC drivers. - -### Python Code - -Python code in resources/libraries/python/autogen contains several other -information sources. - -#### Testcase Templates - -The test case part of template suite is ignored, test case lines -are created according to text templates in Testcase.py file. - -#### Testcase Argument Lists - -Each testcase template has different number of "arguments", e.g. values -to put into various placeholders. Different test types need different -lists of the argument values, the lists are in regenerate_glob method -in Regenerator.py file. - -#### Iteration Over Values - -Python code detects the test type (usually by substrings of suite file name), -then iterates over different quantities based on type. -For example, only ndrpdr suite templates generate other types (mrr and soak). - -#### Hardcoded Exclusions - -Some combinations of values are known not to work, so they are excluded. -Examples: Density tests for too much CPUs; IMIX for ASTF. - -## Non-Sources - -Some information sources are available in CSIT repository, -but do not affect the suites generated by autogen. - -### Testbeds - -Overall, no information visible in topology yaml files is taken into account -by autogen. - -#### Testbed Architecture - -Historically, suite files are agnostic to testbed architecture, e.g. ICX or ALT. - -#### Testbed Size - -Historically, 2-node and 3-node suites have diferent names, and while -most of the code is common, the differences are not always simple enough. -Autogen treat 2-node and 3-node suites as independent templates. - -TRex suites are intended for a 1-node circuit of otherwise 2-node or 3-node -testbeds, so they support all 3 robot tags. -They are also detected and treated differently by autogen, -mainly because they need different testcase arguments (no CPU count). -Autogen does nothing specifically related to the fact they should run -only in testbeds/NICs with TG-TG line available. - -#### Other Topology Info - -Some bonding tests need two (parallel) links between DUTs. -Autogen does not care, as suites are agnostic. -Robot tag marks the difference, but the link presence is not explicitly checked. - -### Job specs - -Information in job spec files depend on generated suites (not the other way). -Autogen should generate more suites, as job spec is limited by time budget. -More suites should be available for manually triggered verify jobs, -so autogen covers that. - -### Bootstrap Scripts - -Historically, bootstrap scripts perform some logic, -perhaps adding exclusion options to Robot invocation -(e.g. skipping testbed+NIC combinations for tests that need parallel links). - -Once again, the logic here relies on what autogen generates, -autogen does not look into bootstrap scripts.