1 # Copyright (c) 2022 Cisco and/or its affiliates.
2 # Licensed under the Apache License, Version 2.0 (the "License");
3 # you may not use this file except in compliance with the License.
4 # You may obtain a copy of the License at:
6 # http://www.apache.org/licenses/LICENSE-2.0
8 # Unless required by applicable law or agreed to in writing, software
9 # distributed under the License is distributed on an "AS IS" BASIS,
10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11 # See the License for the specific language governing permissions and
12 # limitations under the License.
16 $id: https://fd.io/FIXME/CSIT/UTI/test_case/info/1.2.0
17 $schema: https://json-schema.org/draft/2020-12/schema
19 Schema for output of test case.
22 The main structure, without conditional relations between fields yet.
24 additionalProperties: false
28 A derived quantity. Difference between start_time and end_time,
30 $ref: "#/$defs/types/nonnegative_number"
33 DUT type used, e.g. VPP or DPDK.
38 Version string appropriate to DUT type used.
43 UTC date and time in RFC 3339 format, specifying calendar time
44 just before test case ended (at the end of test case teardown).
45 $ref: "#/$defs/types/date_time"
48 Array of hosts this test interacted with.
49 This can be used for identifying testbed number.
50 Valid tests shoud interact with at least one DUT or TG.
51 The array is usually sorted, but that is not a requirement.
56 Host identifier, usually numeric IPv4 address.
60 Array of telemetry entries. Each entry represent one captured
70 If passed is true, this value is empty.
71 Otherwise, value taken directly from TEST_MESSAGE
72 Robot variable, read at the end of test case
73 (in test teardown, before export and validation).
74 It contains information from the exception
75 that caused the failure, probably with additional
76 exceptions from teardown keywords.
80 Value set accordingly to TEST_STATUS Robot variable,
81 true if and only if the status is "PASS".
82 The status is read at the end of test case
83 (in test teardown, before export and validation).
89 Sub-schema common for all cases,
90 only result type identifier defined here.
94 Identifier of which result type case is applied.
100 Result type for unknown case.
101 This case represents a test with no specific result
102 (outside message), e.g. device test;
103 or a test with result not parsed into
104 this version of model yet, e.g. GSO test.
105 additionalProperties: false
110 Result type MRR case.
111 additionalProperties: false
117 The results refer to receive rates for multiple
118 MRR trials. For PPS, these are aggregate
119 (bidirectional) rates.
120 Currently, the tests are exporting
121 approximated receive rates.
122 That means the actual trial duration
123 is measured (as opposed to trusting traffic
124 generator to honor its target duration),
125 so the resulting values contain noise
126 from time measurement, and can be lower
127 than the real performance
128 (due to various time overheads).
129 Bandwidth values are supported, but currently
130 Robot does not export them.
131 $ref: "#/$defs/types/rate_list_with_bandwidth"
136 Result type NDRPDR case.
137 additionalProperties: false
143 The results refer to search for NDR
144 (Non Drop Rate). For PPS, this is aggregate
145 (bidirectional) rate.
146 Each bound was used as the target load value
147 in a full-duration trial measurement.
148 The accepted loss ratio for NDR is exact zero.
149 Note that packets the Traffic Generator
150 did not send are also counted as lost packets.
151 $ref: "#/$defs/macros/lower_and_upper_rate"
154 The results refer to search for PDR
155 (Partial Drop Rate). For PPS, this is aggregate
156 (bidirectional) rate.
157 Each bound was used as the target load value
158 in a full-duration trial measurement.
159 The accepted loss ratio for PDR is 0.5%.
160 Note that packets the Traffic Generator
161 did not send are also counted as lost packets.
162 $ref: "#/$defs/macros/lower_and_upper_rate"
165 Object with results related to latency part
166 of NDRPDR test, for forward traffic direction.
167 It is the direction used in unidirectional
169 ASTF profiles and IMIX STL profiles
170 do not support latency information,
171 so for those tests this object is missing.
172 It is also missing if Traffic Generator
173 fails to return valid latency results
174 for any other reasons,
175 e.g. latency rate is too high for CPU/NIC used.
176 $ref: "#/$defs/macros/latency_for_loads"
179 Object with results related to latency part
180 of NDRPDR test, for reverse traffic diration.
181 This object is not present
182 when unidirectional traffic profiles are used.
183 ASTF profiles and IMIX STL profiles
184 do not support latency information,
185 so for those tests this object is missing.
186 It is also missing if Traffic Generator
187 fails to return valid latency results
188 for any other reasons,
189 e.g. latency rate is too high for CPU/NIC used.
190 $ref: "#/$defs/macros/latency_for_loads"
196 Result type SOAK case.
197 additionalProperties: false
203 The results refer to bayesian estimate
204 of critical rate corresponding to
205 average loss ratio of 10^-7.
206 For PPS, this is aggregate (bidirectional) rate.
207 The bounds are computed from
208 trial measurement results,
209 but are not equal to any target load used.
210 Note that packets the Traffic Generator
211 did not send are also counted as lost packets.
212 $ref: "#/$defs/macros/lower_and_upper_rate"
217 Result type RECONF case.
218 additionalProperties: false
224 Load used when reconfiguring, found as NDR lower
225 bound. This is an aggregate (bidirectional)
226 rate. Note that packets which the Traffic
227 Generator did not send, are also counted as lost
229 $ref: "#/$defs/types/rate_with_bandwidth"
232 Number of packets lost during reconfiguration,
233 with the time that equals packet loss divided by
235 $ref: "#/$defs/types/packet_with_time"
241 UTC date and time in RFC 3339 format, specifying calendar time
242 just after test case started (at the start of test setup).
243 $ref: "#/$defs/types/date_time"
246 The list of strings comes directly
247 from Robot variable TEST_TAGS.
248 The content should include both static and dynamic tags
249 at the end of test case (teardown).
255 Value taken directly from TEST_DOCUMENTATION Robot variable.
256 The content is what you see in suite file
257 at test case definition, which is usually empty
258 as CSIT uses data driven test cases.
263 It is the most complete and unique identifier for a test case.
264 This property has a value, of the following form:
265 {suite_name}.{test_name}
266 Here, suite name comes from SUITE_NAME robot variable,
267 test name comes from TEST_NAME robot variable,
268 but both are converted to lower case,
269 and spaces are replaced by underscores.
275 This property has a value, of the following form:
276 {nic_short_name}-{frame_size}-{threads_and_cores}-{suite_part}
277 Here, suite part is very similar to suite tag,
278 but additionally may contain a prefix describing NIC driver used
279 (if it is not the default one, drv_vfio_pci for VPP tests).
280 Any space is replaced by underscore and letters are lower case.
286 This property has a value very similar to suite tag,
287 but additionally may contain a prefix describing NIC driver used
288 (if it is not the default one, drv_vfio_pci for VPP tests).
289 Any space is replaced by underscore and letters are lower case.
295 Test type identifier, PAL uses it to group similar tests,
296 e.g. for comparison tables.
297 Ideally, this information should be parseable from test name,
298 but the current naming scheme is not simple/consistent enough.
299 The current implementation queries the robot test tags.
300 The resulting value is frequently identical to result type,
301 but this schema version does not require any relation there,
302 as PAL may want to group tests differently.
315 TG type used, e.g. TREX.
320 Version string appropriate to TG type used.
325 CSIT model version (semver format)
326 the exporting code adhered to.
350 Subschema validating relation between status and message.
353 Subschema for passing tests, message has to be empty.
361 Subschema for failing tests, mesage cannot be empty.
391 Packets per second (pps) or connections per second (cps).
398 Unit of measurement for bandwidth values.
399 Currently a constant, but later versions of model
400 may allow more units.
405 Type, for counting packets.
407 - $ref: "#/$defs/types/value_with_unit"
411 A number of packets of interest.
414 Unit suitable for displaying packet counts.
419 Reusable type, for various time quantites.
421 - $ref: "#/$defs/types/value_with_unit"
425 Unless specified otherwise, this is a duration
429 Only seconds are the unit supported for time
435 Reusable composite type, value together with its
438 additionalProperties: false
442 Numeric value, context specified elsewhere.
443 The only assumption is that value is not negative.
444 $ref: "#/$defs/types/nonnegative_number"
447 Unit of measurement for the value.
448 Context and allowed values are specified elsewhere.
453 rate_without_bandwidth:
455 Reusable type, for various rate quantites.
457 - $ref: "#/$defs/types/value_with_unit"
461 Unless specified otherwise,
462 this is the aggregated rate
463 (sum of both traffic directions).
464 Depending on the usage, the value can express
465 intended load, offered load, receive rate,
466 and various approximations
467 or estimated bounds thereof.
470 A transaction rate unit the value is expressed in.
471 $ref: "#/$defs/types/rate_unit"
474 Reusable type, for various bandwidth quantites.
476 - $ref: "#/$defs/types/value_with_unit"
480 Bandwidth value computed
481 from the corresponding rate.
483 $ref: "#/$defs/types/bandwidth_unit"
486 Reusable composite type, joining primary rate
487 with optional derived bandwidth.
488 Not all test types currently compute bandwidth,
489 even if rate unit is pps.
491 additionalProperties: false
494 $ref: "#/$defs/types/rate_without_bandwidth"
496 $ref: "#/$defs/types/bandwidth"
501 Reusable composite type, joining packet count with the
504 additionalProperties: false
507 $ref: "#/$defs/types/count_packets"
509 $ref: "#/$defs/types/time_quantity"
513 value_list_with_unit_and_stats:
515 Reusable composite type, multiple values together with their
516 unit of measurement and derived statistics.
518 additionalProperties: false
522 List of values of the same unit, useful for MRR.
527 Numeric value, context specified elsewhere. The only
528 assumption is that the value is nonnegative.
529 $ref: "#/$defs/types/nonnegative_number"
532 A derived quantity. It is the arithmetic average of the
534 $ref: "#/$defs/types/nonnegative_number"
537 A derived quantity. It is the standard deviation for the
538 values list, as computed by jumpavg library.
539 $ref: "#/$defs/types/nonnegative_number"
542 Unit of measurement for the values.
543 Context and allowed values are specified elsewhere.
550 rate_list_without_bandwidth:
552 Reusable composite type, multiple rate values.
554 - $ref: "#/$defs/types/value_list_with_unit_and_stats"
559 Unless specified otherwise,
560 this is the aggregated rate
561 (sum of both traffic directions).
562 Depending on the usage, the value can express
563 intended load, offered load, receive rate,
564 and various approximations or estimated bounds
567 $ref: "#/$defs/types/rate_unit"
570 Reusable composite type, multiple bandwidth values. This is a
573 - $ref: "#/$defs/types/value_list_with_unit_and_stats"
578 Unless specified otherwise,
579 this is the aggregated bandwidth
580 (sum of both traffic directions).
581 Depending on the usage, the value can express
582 intended load, offered load, receive rate,
583 and various approximations or estimated bounds
586 $ref: "#/$defs/types/bandwidth_unit"
587 rate_list_with_bandwidth:
589 Reusable composite type, joining primary rates
590 with optional derived bandwidths (and stats).
591 No test types currently computes the bandwidth part.
593 additionalProperties: false
596 $ref: "#/$defs/types/rate_list_without_bandwidth"
598 $ref: "#/$defs/types/bandwidth_list"
602 lower_and_upper_rate:
604 additionalProperties: false
608 The lower bound (or min_rate) for the estimate
609 of a particular searched value.
610 $ref: "#/$defs/types/rate_with_bandwidth"
613 The upper bound (or max_rate) for the estimate
614 of a particular searched value.
615 $ref: "#/$defs/types/rate_with_bandwidth"
621 additionalProperties: false
625 Rounded minimal latency time measured in this trial.
626 See unit property for the unit of measurement.
627 $ref: "#/$defs/types/nonnegative_integer"
630 Rounded maximal latency time measured in this trial.
631 See unit property for the unit of measurement.
632 Zero value is not allowed, as that is one of symptoms
633 of Traffic Generator failing to get proper latency.
634 $ref: "#/$defs/types/positive_integer"
637 Rounded average latency time measured in this trial.
638 See unit property for the unit of measurement.
639 $ref: "#/$defs/types/nonnegative_integer"
642 Base64-encoded compressed representation of HDRHistogram
643 of all latency sample times encountered
644 in this latency trial.
645 See unit property for the unit of measurement.
646 Note that some bins can be several units wide.
650 Unit of measurement for latency times.
651 Currently a constant, but later versions
652 of the model may allow more values.
664 additionalProperties: false
668 Object related to latency measurement performed
669 at minimal rate (currently 9000 pps per direction).
670 $ref: "#/$defs/macros/latency_numbers"
673 Object related to latency measurement performed
674 at 10% of PDR lower bound, if needed rounded up
675 to minimal rate (currently 9000 pps per direction).
676 $ref: "#/$defs/macros/latency_numbers"
679 Object related to latency measurement performed
680 at 50% of PDR lower bound, if needed rounded up
681 to minimal rate (currently 9000 pps per direction).
682 $ref: "#/$defs/macros/latency_numbers"
685 Object related to latency measurement performed
686 at 90% of PDR lower bound, if needed rounded up
687 to minimal rate (currently 9000 pps per direction).
688 $ref: "#/$defs/macros/latency_numbers"