1 # Copyright (c) 2023 Cisco and/or its affiliates.
2 # Licensed under the Apache License, Version 2.0 (the "License");
3 # you may not use this file except in compliance with the License.
4 # You may obtain a copy of the License at:
6 # http://www.apache.org/licenses/LICENSE-2.0
8 # Unless required by applicable law or agreed to in writing, software
9 # distributed under the License is distributed on an "AS IS" BASIS,
10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11 # See the License for the specific language governing permissions and
12 # limitations under the License.
16 $id: https://fd.io/FIXME/CSIT/UTI/test_case/info/1.5.1
17 $schema: https://json-schema.org/draft/2020-12/schema
19 Schema for output of test case.
22 The main structure, without conditional relations between fields yet.
24 additionalProperties: false
28 A derived quantity. Difference between start_time and end_time,
30 $ref: "#/$defs/types/nonnegative_number"
33 DUT type used, e.g. VPP or DPDK.
38 Version string appropriate to DUT type used.
43 UTC date and time in RFC 3339 format, specifying calendar time
44 just before test case ended (at the end of test case teardown).
45 $ref: "#/$defs/types/date_time"
48 Array of hosts this test interacted with.
49 This can be used for identifying testbed number.
50 Valid tests shoud interact with at least one DUT or TG.
51 The array is usually sorted, but that is not a requirement.
56 Host identifier, usually numeric IPv4 address.
60 Array of telemetry entries. Each entry represent one captured
70 If passed is true, this value is empty.
71 Otherwise, value taken directly from TEST_MESSAGE
72 Robot variable, read at the end of test case
73 (in test teardown, before export and validation).
74 It contains information from the exception
75 that caused the failure, probably with additional
76 exceptions from teardown keywords.
80 Value set accordingly to TEST_STATUS Robot variable,
81 true if and only if the status is "PASS".
82 The status is read at the end of test case
83 (in test teardown, before export and validation).
89 Sub-schema common for all cases,
90 only result type identifier defined here.
94 Identifier of which result type case is applied.
100 Result type for unknown case.
101 This case represents a test with no specific result
102 (outside message), e.g. device test;
103 or a test with result not parsed into
104 this version of model yet, e.g. GSO test.
105 additionalProperties: false
110 Result type MRR case.
111 additionalProperties: false
117 The results refer to receive rates for multiple
118 MRR trials. For PPS, these are aggregate
119 (bidirectional) rates.
120 Currently, the tests are exporting
121 approximated receive rates.
122 That means the actual trial duration
123 is measured (as opposed to trusting traffic
124 generator to honor its target duration),
125 so the resulting values contain noise
126 from time measurement, and can be lower
127 than the real performance
128 (due to various time overheads).
129 Bandwidth values are supported, but currently
130 Robot does not export them.
131 $ref: "#/$defs/types/rate_list_with_bandwidth"
136 Result type NDRPDR case.
137 additionalProperties: false
143 The results refer to search for NDR
144 The accepted loss ratio for NDR is exact zero.
146 For PPS, lowerbound is aggregate
147 (bidirectional) conditional throughput
148 (median forwarding rate among long trials),
149 upperbound is missing.
150 Note that packets the Traffic Generator
151 did not send are also counted as lost packets.
152 $ref: "#/$defs/macros/lower_and_maybe_upper_rate"
155 The results refer to search for PDR.
156 The accepted loss ratio for PDR is 0.5%.
158 For PPS, lowerbound is aggregate
159 (bidirectional) conditional throughput
160 (median forwarding rate among long trials),
161 upperbound is missing.
162 Note that packets the Traffic Generator
163 did not send are also counted as lost packets.
164 $ref: "#/$defs/macros/lower_and_maybe_upper_rate"
167 Object with results related to latency part
168 of NDRPDR test, for forward traffic direction.
169 It is the direction used in unidirectional
171 ASTF profiles and IMIX STL profiles
172 do not support latency information,
173 so for those tests this object is missing.
174 It is also missing if Traffic Generator
175 fails to return valid latency results
176 for any other reasons,
177 e.g. latency rate is too high for CPU/NIC used.
178 $ref: "#/$defs/macros/latency_for_loads"
181 Object with results related to latency part
182 of NDRPDR test, for reverse traffic diration.
183 This object is not present
184 when unidirectional traffic profiles are used.
185 ASTF profiles and IMIX STL profiles
186 do not support latency information,
187 so for those tests this object is missing.
188 It is also missing if Traffic Generator
189 fails to return valid latency results
190 for any other reasons,
191 e.g. latency rate is too high for CPU/NIC used.
192 $ref: "#/$defs/macros/latency_for_loads"
198 Result type SOAK case.
199 additionalProperties: false
205 The results refer to bayesian estimate
206 of critical rate corresponding to
207 average loss ratio of 10^-7.
208 For PPS, this is aggregate (bidirectional) rate.
209 The bounds are computed from
210 trial measurement results,
211 but are not equal to any target load used.
212 Note that packets the Traffic Generator
213 did not send are also counted as lost packets.
214 $ref: "#/$defs/macros/lower_and_maybe_upper_rate"
219 Result type RECONF case.
220 additionalProperties: false
226 Load used when reconfiguring, found as NDR lower
227 bound. This is an aggregate (bidirectional)
228 rate. Note that packets which the Traffic
229 Generator did not send, are also counted as lost
231 $ref: "#/$defs/types/rate_with_bandwidth"
234 Number of packets lost during reconfiguration,
235 with the time that equals packet loss divided by
237 $ref: "#/$defs/types/packet_with_time"
243 Result type HOSTSTACK case.
244 additionalProperties: false
250 Goodput measured in bits per second.
251 $ref: "#/$defs/types/bandwidth"
254 Number of completed requests.
255 $ref: "#/$defs/types/count_requests"
258 Number of failed requests.
259 $ref: "#/$defs/types/count_requests"
262 Number of retransmits.
263 $ref: "#/$defs/types/count_packets"
266 Value and unit of latency.
267 $ref: "#/$defs/types/value_with_unit"
270 The relative time difference (in seconds)
271 between program start and end.
272 $ref: "#/$defs/types/time_quantity"
275 RPS or CPS rate, with corresponding unit, as
277 $ref: "#/$defs/types/rate_without_bandwidth"
283 UTC date and time in RFC 3339 format, specifying calendar time
284 just after test case started (at the start of test setup).
285 $ref: "#/$defs/types/date_time"
288 The list of strings comes directly
289 from Robot variable TEST_TAGS.
290 The content should include both static and dynamic tags
291 at the end of test case (teardown).
297 Value taken directly from TEST_DOCUMENTATION Robot variable.
298 The content is what you see in suite file
299 at test case definition, which is usually empty
300 as CSIT uses data driven test cases.
305 It is the most complete and unique identifier for a test case.
306 This property has a value, of the following form:
307 {suite_name}.{test_name}
308 Here, suite name comes from SUITE_NAME robot variable,
309 test name comes from TEST_NAME robot variable,
310 but both are converted to lower case,
311 and spaces are replaced by underscores.
317 This property has a value, of the following form:
318 {nic_short_name}-{frame_size}-{threads_and_cores}-{suite_part}
319 Here, suite part is very similar to suite tag,
320 but additionally may contain a prefix describing NIC driver used
321 (if it is not the default one, drv_vfio_pci for VPP tests).
322 Any space is replaced by underscore and letters are lower case.
328 This property has a value very similar to suite tag,
329 but additionally may contain a prefix describing NIC driver used
330 (if it is not the default one, drv_vfio_pci for VPP tests).
331 Any space is replaced by underscore and letters are lower case.
337 Test type identifier, PAL uses it to group similar tests,
338 e.g. for comparison tables.
339 Ideally, this information should be parseable from test name,
340 but the current naming scheme is not simple/consistent enough.
341 The current implementation queries the robot test tags.
342 The resulting value is frequently identical to result type,
343 but this schema version does not require any relation there,
344 as PAL may want to group tests differently.
356 TG type used, e.g. TREX.
361 Version string appropriate to TG type used.
366 CSIT model version (semver format)
367 the exporting code adhered to.
391 Subschema validating relation between status and message.
394 Subschema for passing tests, message has to be empty.
402 Subschema for failing tests, mesage cannot be empty.
432 Packets per second (pps),
433 connections per second (cps),
434 requests per second (rps),
435 bits per second (bps).
444 Unit of measurement for bandwidth values.
445 Currently a constant, but later versions of model
446 may allow more units.
451 Type, for counting packets.
453 - $ref: "#/$defs/types/value_with_unit"
457 A number of packets of interest.
460 Unit suitable for displaying packet counts.
465 Type, for counting requests.
467 - $ref: "#/$defs/types/value_with_unit"
471 A number of requests of interest.
474 Unit suitable for displaying request counts.
479 Reusable type, for various time quantites.
481 - $ref: "#/$defs/types/value_with_unit"
485 Unless specified otherwise, this is a duration
489 Only seconds are the unit supported for time
495 Reusable composite type, value together with its
498 additionalProperties: false
502 Numeric value, context specified elsewhere.
503 The only assumption is that value is not negative.
504 $ref: "#/$defs/types/nonnegative_number"
507 Unit of measurement for the value.
508 Context and allowed values are specified elsewhere.
513 rate_without_bandwidth:
515 Reusable type, for various rate quantites.
517 - $ref: "#/$defs/types/value_with_unit"
521 Unless specified otherwise,
522 this is the aggregated rate
523 (sum of both traffic directions).
524 Depending on the usage, the value can express
525 intended load, offered load, receive rate,
526 and various approximations
527 or estimated bounds thereof.
530 A transaction rate unit the value is expressed in.
531 $ref: "#/$defs/types/rate_unit"
534 Reusable type, for various bandwidth quantites.
536 - $ref: "#/$defs/types/value_with_unit"
540 Bandwidth value computed from the corresponding
543 $ref: "#/$defs/types/bandwidth_unit"
546 Reusable composite type, joining primary rate
547 with optional derived bandwidth.
548 Not all test types currently compute bandwidth,
549 even if rate unit is pps.
551 additionalProperties: false
554 $ref: "#/$defs/types/rate_without_bandwidth"
556 $ref: "#/$defs/types/bandwidth"
561 Reusable composite type, joining packet count with the
564 additionalProperties: false
567 $ref: "#/$defs/types/count_packets"
569 $ref: "#/$defs/types/time_quantity"
573 value_list_with_unit_and_stats:
575 Reusable composite type, multiple values together with their
576 unit of measurement and derived statistics.
578 additionalProperties: false
582 List of values of the same unit, useful for MRR.
587 Numeric value, context specified elsewhere. The only
588 assumption is that the value is nonnegative.
589 $ref: "#/$defs/types/nonnegative_number"
592 A derived quantity. It is the arithmetic average of the
594 $ref: "#/$defs/types/nonnegative_number"
597 A derived quantity. It is the standard deviation for the
598 values list, as computed by jumpavg library.
599 $ref: "#/$defs/types/nonnegative_number"
602 Unit of measurement for the values.
603 Context and allowed values are specified elsewhere.
610 rate_list_without_bandwidth:
612 Reusable composite type, multiple rate values.
614 - $ref: "#/$defs/types/value_list_with_unit_and_stats"
619 Unless specified otherwise,
620 this is the aggregated rate
621 (sum of both traffic directions).
622 Depending on the usage, the value can express
623 intended load, offered load, receive rate,
624 and various approximations or estimated bounds
627 $ref: "#/$defs/types/rate_unit"
630 Reusable composite type, multiple bandwidth values. This is a
633 - $ref: "#/$defs/types/value_list_with_unit_and_stats"
638 Unless specified otherwise,
639 this is the aggregated bandwidth
640 (sum of both traffic directions).
641 Depending on the usage, the value can express
642 intended load, offered load, receive rate,
643 and various approximations or estimated bounds
646 $ref: "#/$defs/types/bandwidth_unit"
647 rate_list_with_bandwidth:
649 Reusable composite type, joining primary rates
650 with optional derived bandwidths (and stats).
651 No test types currently computes the bandwidth part.
653 additionalProperties: false
656 $ref: "#/$defs/types/rate_list_without_bandwidth"
658 $ref: "#/$defs/types/bandwidth_list"
662 lower_and_maybe_upper_rate:
664 additionalProperties: false
668 The lower bound (or min_rate) for the estimate
669 of a particular searched value.
670 $ref: "#/$defs/types/rate_with_bandwidth"
673 The upper bound (or max_rate) for the estimate
674 of a particular searched value.
675 $ref: "#/$defs/types/rate_with_bandwidth"
680 additionalProperties: false
684 Rounded minimal latency time measured in this trial.
685 See unit property for the unit of measurement.
686 $ref: "#/$defs/types/nonnegative_integer"
689 Rounded maximal latency time measured in this trial.
690 See unit property for the unit of measurement.
691 Zero value is not allowed, as that is one of symptoms
692 of Traffic Generator failing to get proper latency.
693 $ref: "#/$defs/types/positive_integer"
696 Rounded average latency time measured in this trial.
697 See unit property for the unit of measurement.
698 $ref: "#/$defs/types/nonnegative_integer"
701 Base64-encoded compressed representation of HDRHistogram
702 of all latency sample times encountered
703 in this latency trial.
704 See unit property for the unit of measurement.
705 Note that some bins can be several units wide.
709 Unit of measurement for latency times.
710 Currently a constant, but later versions
711 of the model may allow more values.
723 additionalProperties: false
727 Object related to latency measurement performed
728 at minimal rate (currently 9000 pps per direction).
729 $ref: "#/$defs/macros/latency_numbers"
732 Object related to latency measurement performed
733 at 10% of PDR lower bound, if needed rounded up
734 to minimal rate (currently 9000 pps per direction).
735 $ref: "#/$defs/macros/latency_numbers"
738 Object related to latency measurement performed
739 at 50% of PDR lower bound, if needed rounded up
740 to minimal rate (currently 9000 pps per direction).
741 $ref: "#/$defs/macros/latency_numbers"
744 Object related to latency measurement performed
745 at 90% of PDR lower bound, if needed rounded up
746 to minimal rate (currently 9000 pps per direction).
747 $ref: "#/$defs/macros/latency_numbers"