3 title: "CSIT Test Code Guidelines"
6 # CSIT Test Code Guidelines
8 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
9 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED",
10 "MAY", and "OPTIONAL" in this document are to be interpreted as
11 described in [BCP 14](https://tools.ietf.org/html/bcp14),
12 [RFC2119](https://tools.ietf.org/html/rfc2119),
13 [RFC8174](https://tools.ietf.org/html/rfc8174)
14 when, and only when, they appear in all capitals, as shown here.
16 This document SHALL describe guidelines for writing reliable, maintainable,
17 reusable and readable code for CSIT.
19 # RobotFramework test case files and resource files
23 + Contributors SHOULD look at requirements.txt in root CSIT directory
24 for the currently used Robot Framework version.
25 Contributors SHOULD read
26 [Robot Framework User Guide](http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html)
29 + RobotFramework test case files and resource files
30 SHALL use special extension .robot
32 + Pipe and space separated file format (without trailing pipe
33 and without pipe aligning) SHALL be used.
34 Tabs are invisible characters, which are error prone.
35 4-spaces separation is prone to accidental double space
36 acting as a separator.
38 + Files SHALL be encoded in UTF-8 (the default Robot source file encoding).
39 Usage of non-ASCII characters SHOULD be avoided if possible.
41 [escape](http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#escaping)
44 + Line length SHALL be limited to 80 characters.
46 + There SHALL be licence text present at the beginning of each file.
48 + Copy-pasting of the code NOT RECOMMENDED practice, any code that could be
49 re-used SHOULD be put into a library (Robot resource, Python library, ...).
53 + It is RECOMMENDED to use data-driven test case definitions
54 anytime suite contains test cases similar in structure.
55 Typically, a suite SHOULD define a Template keyword, and test cases
56 SHOULD only specify tags and argument values
59 | Test Template | Local Template
63 | tc01-64B-1c-eth-l2patch-mrr
65 | | framesize=${64} | phy_cores=${1}
67 + Test case templates (or testcases) SHALL be written in Behavior-driven style
68 i.e. in readable English, so that even non-technical project stakeholders
74 | | ... | [Cfg] DUT runs L2 patch config with ${phy_cores} phy core(s).
75 | | ... | [Ver] Measure NDR and PDR values using MLRsearch algorithm.\
77 | | ... | *Arguments:*
78 | | ... | - frame_size - Framesize in Bytes in integer
79 | | ... | or string (IMIX_v4_1). Type: integer, string
80 | | ... | - phy_cores - Number of physical cores. Type: integer
81 | | ... | - rxq - Number of RX queues, default value: ${None}.
82 | | ... | Type: integer
84 | | [Arguments] | ${frame_size} | ${phy_cores} | ${rxq}=${None}
86 | | Set Test Variable | \${frame_size}
88 | | Given Add worker threads and rxqueues to all DUTs
89 | | ... | ${phy_cores} | ${rxq}
90 | | And Add PCI devices to all DUTs
91 | | Set Max Rate And Jumbo And Handle Multi Seg
92 | | And Apply startup configuration on all VPP DUTs
93 | | When Initialize L2 patch
94 | | Then Find NDR and PDR intervals using optimized search
96 + Every suite and test case template (or testcase)
97 SHALL contain short documentation.
98 Generated CSIT web pages display the documentation.
100 + You SHOULD NOT use hard-coded constants.
101 It is RECOMMENDED to use the variable table
102 (\*\*\*Variables\*\*\*) to define test case specific values.
103 You SHALL use the assignment sign = after the variable name
104 to make assigning variables slightly more explicit
107 | ${traffic_profile}= | trex-stl-2n-ethip4-ip4src254
109 + Common test case specific settings of the test environment SHALL be done
110 in Test Setup keyword defined in the Setting table.
112 + Run Keywords construction is RECOMMENDED if it is more readable
115 + Separate keyword is RECOMMENDED if the construction is less readable.
117 + Post-test cleaning and processing actions SHALL be done in Test Teardown
118 part of the Setting table (e.g. download statistics from VPP nodes).
119 This part is executed even if the test case has failed. On the other hand
120 it is possible to disable the tear-down from command line, thus leaving
121 the system in “broken” state for investigation.
123 + Every testcase SHALL be correctly tagged. List of defined tags is in
124 csit/docs/introduction/test_tag_documentation.rst
126 + Whenever possible, common tags SHALL be set using Force Tags
129 + User high-level keywords specific for the particular test suite
130 SHOULD be implemented in the Keywords table of suitable Robot resource file
131 to enable readability and code-reuse.
133 + Such keywords MAY be implemented in Keywords table of the suite instead,
134 if the contributor believes no other test will use such keywords.
135 But this is NOT RECOMMENDED in general, as keywords in Resources
136 are easier to maintain.
138 + All test case names (and suite names) SHALL conform
139 to current naming convention.
140 https://wiki.fd.io/view/CSIT/csit-test-naming
142 + Frequently, different suites use the same test case layout.
143 It is RECOMMENDED to use autogeneration scripts available,
144 possibly extending them if their current functionality is not sufficient.
148 + SHALL be used to implement higher-level keywords that are used in test cases
149 or other higher-level (or medium-level) keywords.
151 + Every keyword SHALL contain Documentation where the purpose and arguments
152 of the keyword are described. Also document types, return values,
153 and any specific assumptions the particular keyword relies on.
155 + A keyword usage example SHALL be the part of the Documentation.
156 The example SHALL use pipe and space separated format
157 (with escaped pipes and) with a trailing pipe.
159 + The reason was possbile usage of Robot's libdoc tool
160 to generate tests and resources documentation. In that case
161 example keyword usage would be rendered in table.
163 + Keyword name SHALL describe what the keyword does,
164 specifically and in a reasonable length (“short sentence”).
166 + Keyword names SHALL be short enough for call sites
167 to fit within line length limit.
169 + If a keyword argument has a most commonly used value, it is RECOMMENDED
170 to set it as default. This makes keyword code longer,
171 but suite code shorter, and readability (and maintainability)
172 of suites SHALL always more important.
174 + If there is intermediate data (created by one keyword, to be used
175 by another keyword) of singleton semantics (it is clear that the test case
176 can have at most one instance of such data, even if the instance
177 is complex, for example ${nodes}), it is RECOMMENDED to store it
178 in test variables. You SHALL document test variables read or written
179 by a keyword. This makes the test template code less verbose.
180 As soon as the data instance is not unique, you SHALL pass it around
181 via arguments and return values explicitly (this makes lower level keywords
182 more reusable and less bug prone).
184 + It is RECOMMENDED to pass arguments explicitly via [Arguments] line.
185 Setting test variables takes more space and is less explicit.
186 Using arguments embedded in keyword name makes them less visible,
187 and it makes it harder for the line containing the resulting long name
188 to fit into the maximum character limit, so you SHOULD NOT use them.
190 # Python library files
194 + SHALL be used to implement low-level keywords that are called from
195 resource files (of higher-level keywords) or from test cases.
197 + Higher-level keywords MAY be implemented in python library file too.
198 it is RECOMMENDED especially in the case that their implementation
199 in resource file would be too difficult or impossible,
200 e.g. complex data structures or functional programming.
202 + Every keyword, Python module, class, method, enum SHALL contain
203 docstring with the short description and used input parameters
204 and possible return value(s) or raised exceptions.
206 + The docstrings SHOULD conform to
207 [PEP 257](https://www.python.org/dev/peps/pep-0257/)
208 and other quality standards.
210 + CSIT contributions SHALL use a specific formatting for documenting
211 arguments, return values and similar.
213 + Keyword usage examples MAY be grouped and used
214 in the class/module documentation string, to provide better overview
215 of the usage and relationships between keywords.
217 + Keyword name SHALL describe what the keyword does,
218 specifically and in a reasonable length (“short sentence”).
219 See https://wiki.fd.io/view/CSIT/csit-test-naming
221 + Python implementation of a keyword is a function,
222 so its name in the python library should be lowercase_with_underscores.
223 Robot call sites should usename with first letter capitalized, and spaces.
227 + It is RECOMMENDED to use some standard development tool
228 (e.g. PyCharm Community Edition) and follow
229 [PEP-8](https://www.python.org/dev/peps/pep-0008/) recommendations.
231 + All python code (not only Robot libraries) SHALL adhere to PEP-8 standard.
232 This is reported by CSIT Jenkins verify job.
234 + Indentation: You SHALL NOT use tab for indents!
235 Indent is defined as four spaces.
237 + Line length: SHALL be limited to 80 characters.
239 + CSIT Python code assumes PYTHONPATH is set
240 to the root of cloned CSIT git repository, creating a tree of sub-packages.
241 You SHALL use that tree for importing, for example
243 from resources.libraries.python.ssh import exec_cmd_no_error
245 + Imports SHALL be grouped in the following order:
247 1. standard library imports,
248 2. related third party imports,
249 3. local application/library specific imports.
251 You SHALL put a blank line between each group of imports.
253 + You SHALL use two blank lines between top-level definitions,
254 one blank line between method definitions.
256 + You SHALL NOT execute any active code on library import.
258 + You SHALL NOT use global variables inside library files.
260 + You MAY define constants inside library files.
262 + It is NOT RECOMMENDED to use hard-coded constants (e.g. numbers,
263 paths without any description). It is RECOMMENDED to use
264 configuration file(s), like /csit/resources/libraries/python/Constants.py,
265 with appropriate comments.
267 + The code SHALL log at the lowest possible level of implementation,
268 for debugging purposes. You SHALL use same style for similar events.
269 You SHALL keep logging as verbose as necessary.
271 + You SHALL use the most appropriate exception not general one (Exception)
272 if possible. You SHOULD create your own exception
273 if necessary and implement there logging, level debug.
275 + You MAY use RuntimeException for generally unexpected failures.
277 + It is RECOMMENDED to use RuntimeError also for
278 infrastructure failures, e.g. losing SSH connection to SUT.
280 + You MAY use EnvironmentError and its cublasses instead,
281 if the distinction is informative for callers.
283 + It is RECOMMENDED to use AssertionError when SUT is at fault.
285 + For each class (e.g. exception) it is RECOMMENDED to implement __repr__()
286 which SHALL return a string usable as a constructor call
287 (including repr()ed arguments).
288 When logging, you SHOULD log the repr form, unless the internal structure
289 of the object in question would likely result in too long output.
290 This is helpful for debugging.
292 + For composing and formatting strings, you SHOULD use .format()
293 with named arguments.
294 Example: "repr() of name: {name!r}".format(name=name)