1 # Copyright (c) 2021 Cisco and/or its affiliates.
2 # Licensed under the Apache License, Version 2.0 (the "License");
3 # you may not use this file except in compliance with the License.
4 # You may obtain a copy of the License at:
6 # http://www.apache.org/licenses/LICENSE-2.0
8 # Unless required by applicable law or agreed to in writing, software
9 # distributed under the License is distributed on an "AS IS" BASIS,
10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11 # See the License for the specific language governing permissions and
12 # limitations under the License.
16 $id: https://fd.io/FIXME/CSIT/UTI/test_case/info/1.0.0
17 $schema: https://json-schema.org/draft/2020-12/schema
19 Schema for info output of test case.
22 The main structure, without conditional relations between fields yet.
24 additionalProperties: false
28 A derived quantity, present only in info output files.
29 Difference between start_time and end_time, in seconds.
30 $ref: "#/$defs/types/nonnegative_number"
33 DUT type used, e.g. VPP or DPDK.
37 Version string appropriate to DUT type used.
41 UTC date and time in RFC 3339 format, specifying calendar time
42 just before test case ended (at the end of test case teardown).
43 $ref: "#/$defs/types/date_time"
46 Array of hosts this test interacted with.
47 This can be used for identifying testbed number.
48 Valid tests shoud interact with at least one DUT or TG.
49 The array is usually sorted, but that is not a requirement.
54 Host identifier, usually numeric IPv4 address.
58 No log items are implemented in the current version,
59 but the (empty) list is present to simplify logic
60 in multi-version importers.
61 $ref: "#/$defs/types/empty_array"
64 If passed is true, this value is empty.
65 Otherwise, value taken directly from TEST_MESSAGE
66 Robot variable, read at the end of test case
67 (in test teardown, before export and validation).
68 It contains information from the exception
69 that caused the failure, probably with additional
70 exceptions from teardown keywords.
74 Value set accordingly to TEST_STATUS Robot variable,
75 true if and only if the status is "PASS".
76 The status is read at the end of test case
77 (in test teardown, before export and validation).
83 Sub-schema common for all cases,
84 only result type identifier defined here.
88 Identifier of which result type case is applied.
94 Result type for unknown case.
95 This case represents a test with no specific result
96 (outside message), e.g. device test;
97 or a test with result not parsed into
98 this version of model yet, e.g. GSO test.
99 additionalProperties: false
104 Result type MRR case.
105 additionalProperties: false
111 The results refer to receive rates for multiple
112 MRR trials. For PPS, these are aggregate
113 (bidirectional) rates.
114 Currently, the tests are exporting
115 approximated receive rates.
116 That means the actual trial duration
117 is measured (as opposed to trusting traffic
118 generator to honor its target duration),
119 so the resulting values contain noise
120 from time measurement, and can be lower
121 than the real performance
122 (due to various time overheads).
123 Bandwidth values are supported, but currently
124 Robot does not export them.
125 $ref: "#/$defs/types/rate_list_with_bandwidth"
130 Result type NDRPDR case.
131 additionalProperties: false
137 The results refer to search for NDR
138 (Non Drop Rate). For PPS, this is aggregate
139 (bidirectional) rate.
140 Each bound was used as the target load value
141 in a full-duration trial measurement.
142 The accepted loss ratio for NDR is exact zero.
143 Note that packets the Traffic Generator
144 did not send are also counted as lost packets.
145 $ref: "#/$defs/macros/lower_and_upper_rate"
148 The results refer to search for PDR
149 (Partial Drop Rate). For PPS, this is aggregate
150 (bidirectional) rate.
151 Each bound was used as the target load value
152 in a full-duration trial measurement.
153 The accepted loss ratio for PDR is 0.5%.
154 Note that packets the Traffic Generator
155 did not send are also counted as lost packets.
156 $ref: "#/$defs/macros/lower_and_upper_rate"
159 Object with results related to latency part
160 of NDRPDR test, for forward traffic direction.
161 It is the direction used in unidirectional
163 ASTF profiles and IMIX STL profiles
164 do not support latency information,
165 so for those tests this object is missing.
166 It is also missing if Traffic Generator
167 fails to return valid latency results
168 for any other reasons,
169 e.g. latency rate is too high for CPU/NIC used.
170 $ref: "#/$defs/macros/latency_for_loads"
173 Object with results related to latency part
174 of NDRPDR test, for reverse traffic diration.
175 This object is not present
176 when unidirectional traffic profiles are used.
177 ASTF profiles and IMIX STL profiles
178 do not support latency information,
179 so for those tests this object is missing.
180 It is also missing if Traffic Generator
181 fails to return valid latency results
182 for any other reasons,
183 e.g. latency rate is too high for CPU/NIC used.
184 $ref: "#/$defs/macros/latency_for_loads"
190 Result type SOAK case.
191 additionalProperties: false
197 The results refer to bayesian estimate
198 of critical rate corresponding to
199 average loss ratio of 10^-7.
200 For PPS, this is aggregate (bidirectional) rate.
201 The bounds are computed from
202 trial measurement results,
203 but are not equal to any target load used.
204 Note that packets the Traffic Generator
205 did not send are also counted as lost packets.
206 $ref: "#/$defs/macros/lower_and_upper_rate"
212 UTC date and time in RFC 3339 format, specifying calendar time
213 just after test case started (at the start of test setup).
214 $ref: "#/$defs/types/date_time"
217 The list of strings comes directly
218 from Robot variable TEST_TAGS.
219 The content should include both static and dynamic tags
220 at the end of test case (teardown).
226 Value taken directly from TEST_DOCUMENTATION Robot variable.
227 The content is what you see in suite file
228 at test case definition, which is usually empty
229 as CSIT uses data driven test cases.
233 A derived quantity, present only in info output files.
234 It is the most complete and unique identifier for a test case.
235 This property has a value, of the following form:
236 {suite_name}.{test_name}
237 Here, suite name comes from SUITE_NAME robot variable,
238 test name comes from TEST_NAME robot variable,
239 but both are converted to lower case,
240 and spaces are replaced by underscores.
245 A derived quantity, present only in info output files.
246 This property has a value, of the following form:
247 {nic_short_name}-{frame_size}-{threads_and_cores}-{suite_part}
248 Here, suite part is very similar to suite tag,
249 but additionally may contain a prefix describing NIC driver used
250 (if it is not the default one, drv_vfio_pci for VPP tests).
251 Any space is replaced by underscore and letters are lower case.
256 A derived quantity, present only in info output files.
257 This property has a value very similar to suite tag,
258 but additionally may contain a prefix describing NIC driver used
259 (if it is not the default one, drv_vfio_pci for VPP tests).
260 Any space is replaced by underscore and letters are lower case.
265 A derived quantity, present only in info output files.
266 Test type identifier, PAL uses it to group similar tests,
267 e.g. for comparison tables.
268 Ideally, this information should be parseable from test name,
269 but the current naming scheme is not simple/consistent enough.
270 The current implementation queries the robot test tags.
271 The resulting value is frequently identical to result type,
272 but this schema version does not require any relation there,
273 as PAL may want to group tests differently.
286 CSIT model version (semver format)
287 the exporting code adhered to.
309 Subschema validating relation between status and message.
312 Subschema for passing tests, message has to be empty.
320 Subschema for failing tests, mesage cannot be empty.
328 Subschema validating relation between dut_type and dut_version.
331 Subschema for tests with no DUT, e.g. TRex self-test.
339 Subschema for DUT type VPP.
347 Subschema for DUT type DPDK.
377 Packets per second (pps) or connections per second (cps).
384 Unit of measurement for bandwidth values.
385 Currently a constant, but later versions of model
386 may allow more units.
391 Reusable composite type, value together with its
394 additionalProperties: false
398 Numeric value, context specified elsewhere.
399 The only assumption is that value is not negative.
400 $ref: "#/$defs/types/nonnegative_number"
403 Unit of measurement for the value.
404 Context and allowed values are specified elsewhere.
409 rate_without_bandwidth:
411 Reusable type, for various rate quantites.
413 - $ref: "#/$defs/types/value_with_unit"
417 Unless specified otherwise,
418 this is the aggregated rate
419 (sum of both traffic directions).
420 Depending on the usage, the value can express
421 intended load, offered load, receive rate,
422 and various approximations
423 or estimated bounds thereof.
426 A transaction rate unit the value is expressed in.
427 $ref: "#/$defs/types/rate_unit"
430 Reusable type, for various bandwidth quantites.
432 - $ref: "#/$defs/types/value_with_unit"
436 Bandwidth value computed
437 from the corresponding rate.
439 $ref: "#/$defs/types/bandwidth_unit"
442 Reusable composite type, joining primary rate
443 with optional derived bandwidth.
444 Not all test types currently compute bandwidth,
445 even if rate unit is pps.
447 additionalProperties: false
450 $ref: "#/$defs/types/rate_without_bandwidth"
452 $ref: "#/$defs/types/bandwidth"
455 value_list_with_unit_and_stats:
457 Reusable composite type, multiple values together with their
458 unit of measurement and derived statistics.
460 additionalProperties: false
464 List of values of the same unit, useful for MRR.
469 Numeric value, context specified elsewhere.
470 The only assumption is that the value is nonnegative.
471 $ref: "#/$defs/types/nonnegative_number"
474 A derived quantity, present only in info output files.
475 It is the arithmetic average of the values list.
476 $ref: "#/$defs/types/nonnegative_number"
479 A derived quantity, present only in info output files.
480 It is the standard deviation for the values list,
481 as computed by jumpavg library.
482 $ref: "#/$defs/types/nonnegative_number"
485 Unit of measurement for the values.
486 Context and allowed values are specified elsewhere.
493 rate_list_without_bandwidth:
495 Reusable composite type, multiple rate values.
497 - $ref: "#/$defs/types/value_list_with_unit_and_stats"
502 Unless specified otherwise,
503 this is the aggregated rate
504 (sum of both traffic directions).
505 Depending on the usage, the value can express
506 intended load, offered load, receive rate,
507 and various approximations or estimated bounds
510 $ref: "#/$defs/types/rate_unit"
513 Reusable composite type, multiple bandwidth values.
514 This is a derived entity, thus it only appears in info output,
515 and only if rate unit is pps.
517 - $ref: "#/$defs/types/value_list_with_unit_and_stats"
522 Unless specified otherwise,
523 this is the aggregated bandwidth
524 (sum of both traffic directions).
525 Depending on the usage, the value can express
526 intended load, offered load, receive rate,
527 and various approximations or estimated bounds
530 $ref: "#/$defs/types/bandwidth_unit"
531 rate_list_with_bandwidth:
533 Reusable composite type, joining primary rates
534 with optional derived bandwidths (and stats).
535 No test types currently computes the bandwidth part.
537 additionalProperties: false
540 $ref: "#/$defs/types/rate_list_without_bandwidth"
542 $ref: "#/$defs/types/bandwidth_list"
546 lower_and_upper_rate:
548 additionalProperties: false
552 The lower bound (or min_rate) for the estimate
553 of a particular searched value.
554 $ref: "#/$defs/types/rate_with_bandwidth"
557 The upper bound (or max_rate) for the estimate
558 of a particular searched value.
559 $ref: "#/$defs/types/rate_with_bandwidth"
565 additionalProperties: false
569 Rounded minimal latency time measured in this trial.
570 See unit property for the unit of measurement.
571 $ref: "#/$defs/types/nonnegative_integer"
574 Rounded maximal latency time measured in this trial.
575 See unit property for the unit of measurement.
576 Zero value is not allowed, as that is one of symptoms
577 of Traffic Generator failing to get proper latency.
578 $ref: "#/$defs/types/positive_integer"
581 Rounded average latency time measured in this trial.
582 See unit property for the unit of measurement.
583 $ref: "#/$defs/types/nonnegative_integer"
586 Base64-encoded compressed representation of HDRHistogram
587 of all latency sample times encountered
588 in this latency trial.
589 See unit property for the unit of measurement.
590 Note that some bins can be several units wide.
594 Unit of measurement for latency times.
595 Currently a constant, but later versions
596 of the model may allow more values.
608 additionalProperties: false
612 Object related to latency measurement performed
613 at minimal rate (currently 9000 pps per direction).
614 $ref: "#/$defs/macros/latency_numbers"
617 Object related to latency measurement performed
618 at 10% of PDR lower bound, if needed rounded up
619 to minimal rate (currently 9000 pps per direction).
620 $ref: "#/$defs/macros/latency_numbers"
623 Object related to latency measurement performed
624 at 50% of PDR lower bound, if needed rounded up
625 to minimal rate (currently 9000 pps per direction).
626 $ref: "#/$defs/macros/latency_numbers"
629 Object related to latency measurement performed
630 at 90% of PDR lower bound, if needed rounded up
631 to minimal rate (currently 9000 pps per direction).
632 $ref: "#/$defs/macros/latency_numbers"