1 # Copyright (c) 2023 Cisco and/or its affiliates.
2 # Licensed under the Apache License, Version 2.0 (the "License");
3 # you may not use this file except in compliance with the License.
4 # You may obtain a copy of the License at:
6 # http://www.apache.org/licenses/LICENSE-2.0
8 # Unless required by applicable law or agreed to in writing, software
9 # distributed under the License is distributed on an "AS IS" BASIS,
10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11 # See the License for the specific language governing permissions and
12 # limitations under the License.
16 $id: https://fd.io/FIXME/CSIT/UTI/test_case/info/1.4.0
17 $schema: https://json-schema.org/draft/2020-12/schema
19 Schema for output of test case.
22 The main structure, without conditional relations between fields yet.
24 additionalProperties: false
28 A derived quantity. Difference between start_time and end_time,
30 $ref: "#/$defs/types/nonnegative_number"
33 DUT type used, e.g. VPP or DPDK.
38 Version string appropriate to DUT type used.
43 UTC date and time in RFC 3339 format, specifying calendar time
44 just before test case ended (at the end of test case teardown).
45 $ref: "#/$defs/types/date_time"
48 Array of hosts this test interacted with.
49 This can be used for identifying testbed number.
50 Valid tests shoud interact with at least one DUT or TG.
51 The array is usually sorted, but that is not a requirement.
56 Host identifier, usually numeric IPv4 address.
60 Array of telemetry entries. Each entry represent one captured
70 If passed is true, this value is empty.
71 Otherwise, value taken directly from TEST_MESSAGE
72 Robot variable, read at the end of test case
73 (in test teardown, before export and validation).
74 It contains information from the exception
75 that caused the failure, probably with additional
76 exceptions from teardown keywords.
80 Value set accordingly to TEST_STATUS Robot variable,
81 true if and only if the status is "PASS".
82 The status is read at the end of test case
83 (in test teardown, before export and validation).
89 Sub-schema common for all cases,
90 only result type identifier defined here.
94 Identifier of which result type case is applied.
100 Result type for unknown case.
101 This case represents a test with no specific result
102 (outside message), e.g. device test;
103 or a test with result not parsed into
104 this version of model yet, e.g. GSO test.
105 additionalProperties: false
110 Result type MRR case.
111 additionalProperties: false
117 The results refer to receive rates for multiple
118 MRR trials. For PPS, these are aggregate
119 (bidirectional) rates.
120 Currently, the tests are exporting
121 approximated receive rates.
122 That means the actual trial duration
123 is measured (as opposed to trusting traffic
124 generator to honor its target duration),
125 so the resulting values contain noise
126 from time measurement, and can be lower
127 than the real performance
128 (due to various time overheads).
129 Bandwidth values are supported, but currently
130 Robot does not export them.
131 $ref: "#/$defs/types/rate_list_with_bandwidth"
136 Result type NDRPDR case.
137 additionalProperties: false
143 The results refer to search for NDR
144 (Non Drop Rate). For PPS, this is aggregate
145 (bidirectional) rate.
146 Each bound was used as the target load value
147 in a full-duration trial measurement.
148 The accepted loss ratio for NDR is exact zero.
149 Note that packets the Traffic Generator
150 did not send are also counted as lost packets.
151 $ref: "#/$defs/macros/lower_and_upper_rate"
154 The results refer to search for PDR
155 (Partial Drop Rate). For PPS, this is aggregate
156 (bidirectional) rate.
157 Each bound was used as the target load value
158 in a full-duration trial measurement.
159 The accepted loss ratio for PDR is 0.5%.
160 Note that packets the Traffic Generator
161 did not send are also counted as lost packets.
162 $ref: "#/$defs/macros/lower_and_upper_rate"
165 Object with results related to latency part
166 of NDRPDR test, for forward traffic direction.
167 It is the direction used in unidirectional
169 ASTF profiles and IMIX STL profiles
170 do not support latency information,
171 so for those tests this object is missing.
172 It is also missing if Traffic Generator
173 fails to return valid latency results
174 for any other reasons,
175 e.g. latency rate is too high for CPU/NIC used.
176 $ref: "#/$defs/macros/latency_for_loads"
179 Object with results related to latency part
180 of NDRPDR test, for reverse traffic diration.
181 This object is not present
182 when unidirectional traffic profiles are used.
183 ASTF profiles and IMIX STL profiles
184 do not support latency information,
185 so for those tests this object is missing.
186 It is also missing if Traffic Generator
187 fails to return valid latency results
188 for any other reasons,
189 e.g. latency rate is too high for CPU/NIC used.
190 $ref: "#/$defs/macros/latency_for_loads"
196 Result type SOAK case.
197 additionalProperties: false
203 The results refer to bayesian estimate
204 of critical rate corresponding to
205 average loss ratio of 10^-7.
206 For PPS, this is aggregate (bidirectional) rate.
207 The bounds are computed from
208 trial measurement results,
209 but are not equal to any target load used.
210 Note that packets the Traffic Generator
211 did not send are also counted as lost packets.
212 $ref: "#/$defs/macros/lower_and_upper_rate"
217 Result type RECONF case.
218 additionalProperties: false
224 Load used when reconfiguring, found as NDR lower
225 bound. This is an aggregate (bidirectional)
226 rate. Note that packets which the Traffic
227 Generator did not send, are also counted as lost
229 $ref: "#/$defs/types/rate_with_bandwidth"
232 Number of packets lost during reconfiguration,
233 with the time that equals packet loss divided by
235 $ref: "#/$defs/types/packet_with_time"
241 Result type HOSTSTACK case.
242 additionalProperties: false
248 Goodput measured in bits per second.
249 $ref: "#/$defs/types/bandwidth"
252 Number of completed requests.
253 $ref: "#/$defs/types/count_requests"
256 Number of failed requests.
257 $ref: "#/$defs/types/count_requests"
260 Number of retransmits.
261 $ref: "#/$defs/types/count_packets"
264 Value and unit of latency.
265 $ref: "#/$defs/types/value_with_unit"
268 The relative time difference (in seconds)
269 between program start and end.
270 $ref: "#/$defs/types/time_quantity"
273 RPS or CPS rate, with corresponding unit, as
275 $ref: "#/$defs/types/rate_without_bandwidth"
281 UTC date and time in RFC 3339 format, specifying calendar time
282 just after test case started (at the start of test setup).
283 $ref: "#/$defs/types/date_time"
286 The list of strings comes directly
287 from Robot variable TEST_TAGS.
288 The content should include both static and dynamic tags
289 at the end of test case (teardown).
295 Value taken directly from TEST_DOCUMENTATION Robot variable.
296 The content is what you see in suite file
297 at test case definition, which is usually empty
298 as CSIT uses data driven test cases.
303 It is the most complete and unique identifier for a test case.
304 This property has a value, of the following form:
305 {suite_name}.{test_name}
306 Here, suite name comes from SUITE_NAME robot variable,
307 test name comes from TEST_NAME robot variable,
308 but both are converted to lower case,
309 and spaces are replaced by underscores.
315 This property has a value, of the following form:
316 {nic_short_name}-{frame_size}-{threads_and_cores}-{suite_part}
317 Here, suite part is very similar to suite tag,
318 but additionally may contain a prefix describing NIC driver used
319 (if it is not the default one, drv_vfio_pci for VPP tests).
320 Any space is replaced by underscore and letters are lower case.
326 This property has a value very similar to suite tag,
327 but additionally may contain a prefix describing NIC driver used
328 (if it is not the default one, drv_vfio_pci for VPP tests).
329 Any space is replaced by underscore and letters are lower case.
335 Test type identifier, PAL uses it to group similar tests,
336 e.g. for comparison tables.
337 Ideally, this information should be parseable from test name,
338 but the current naming scheme is not simple/consistent enough.
339 The current implementation queries the robot test tags.
340 The resulting value is frequently identical to result type,
341 but this schema version does not require any relation there,
342 as PAL may want to group tests differently.
354 TG type used, e.g. TREX.
359 Version string appropriate to TG type used.
364 CSIT model version (semver format)
365 the exporting code adhered to.
389 Subschema validating relation between status and message.
392 Subschema for passing tests, message has to be empty.
400 Subschema for failing tests, mesage cannot be empty.
430 Packets per second (pps),
431 connections per second (cps),
432 requests per second (rps).
440 Unit of measurement for bandwidth values.
441 Currently a constant, but later versions of model
442 may allow more units.
447 Type, for counting packets.
449 - $ref: "#/$defs/types/value_with_unit"
453 A number of packets of interest.
456 Unit suitable for displaying packet counts.
461 Type, for counting requests.
463 - $ref: "#/$defs/types/value_with_unit"
467 A number of requests of interest.
470 Unit suitable for displaying request counts.
475 Reusable type, for various time quantites.
477 - $ref: "#/$defs/types/value_with_unit"
481 Unless specified otherwise, this is a duration
485 Only seconds are the unit supported for time
491 Reusable composite type, value together with its
494 additionalProperties: false
498 Numeric value, context specified elsewhere.
499 The only assumption is that value is not negative.
500 $ref: "#/$defs/types/nonnegative_number"
503 Unit of measurement for the value.
504 Context and allowed values are specified elsewhere.
509 rate_without_bandwidth:
511 Reusable type, for various rate quantites.
513 - $ref: "#/$defs/types/value_with_unit"
517 Unless specified otherwise,
518 this is the aggregated rate
519 (sum of both traffic directions).
520 Depending on the usage, the value can express
521 intended load, offered load, receive rate,
522 and various approximations
523 or estimated bounds thereof.
526 A transaction rate unit the value is expressed in.
527 $ref: "#/$defs/types/rate_unit"
530 Reusable type, for various bandwidth quantites.
532 - $ref: "#/$defs/types/value_with_unit"
536 Bandwidth value computed from the corresponding
539 $ref: "#/$defs/types/bandwidth_unit"
542 Reusable composite type, joining primary rate
543 with optional derived bandwidth.
544 Not all test types currently compute bandwidth,
545 even if rate unit is pps.
547 additionalProperties: false
550 $ref: "#/$defs/types/rate_without_bandwidth"
552 $ref: "#/$defs/types/bandwidth"
557 Reusable composite type, joining packet count with the
560 additionalProperties: false
563 $ref: "#/$defs/types/count_packets"
565 $ref: "#/$defs/types/time_quantity"
569 value_list_with_unit_and_stats:
571 Reusable composite type, multiple values together with their
572 unit of measurement and derived statistics.
574 additionalProperties: false
578 List of values of the same unit, useful for MRR.
583 Numeric value, context specified elsewhere. The only
584 assumption is that the value is nonnegative.
585 $ref: "#/$defs/types/nonnegative_number"
588 A derived quantity. It is the arithmetic average of the
590 $ref: "#/$defs/types/nonnegative_number"
593 A derived quantity. It is the standard deviation for the
594 values list, as computed by jumpavg library.
595 $ref: "#/$defs/types/nonnegative_number"
598 Unit of measurement for the values.
599 Context and allowed values are specified elsewhere.
606 rate_list_without_bandwidth:
608 Reusable composite type, multiple rate values.
610 - $ref: "#/$defs/types/value_list_with_unit_and_stats"
615 Unless specified otherwise,
616 this is the aggregated rate
617 (sum of both traffic directions).
618 Depending on the usage, the value can express
619 intended load, offered load, receive rate,
620 and various approximations or estimated bounds
623 $ref: "#/$defs/types/rate_unit"
626 Reusable composite type, multiple bandwidth values. This is a
629 - $ref: "#/$defs/types/value_list_with_unit_and_stats"
634 Unless specified otherwise,
635 this is the aggregated bandwidth
636 (sum of both traffic directions).
637 Depending on the usage, the value can express
638 intended load, offered load, receive rate,
639 and various approximations or estimated bounds
642 $ref: "#/$defs/types/bandwidth_unit"
643 rate_list_with_bandwidth:
645 Reusable composite type, joining primary rates
646 with optional derived bandwidths (and stats).
647 No test types currently computes the bandwidth part.
649 additionalProperties: false
652 $ref: "#/$defs/types/rate_list_without_bandwidth"
654 $ref: "#/$defs/types/bandwidth_list"
658 lower_and_upper_rate:
660 additionalProperties: false
664 The lower bound (or min_rate) for the estimate
665 of a particular searched value.
666 $ref: "#/$defs/types/rate_with_bandwidth"
669 The upper bound (or max_rate) for the estimate
670 of a particular searched value.
671 $ref: "#/$defs/types/rate_with_bandwidth"
677 additionalProperties: false
681 Rounded minimal latency time measured in this trial.
682 See unit property for the unit of measurement.
683 $ref: "#/$defs/types/nonnegative_integer"
686 Rounded maximal latency time measured in this trial.
687 See unit property for the unit of measurement.
688 Zero value is not allowed, as that is one of symptoms
689 of Traffic Generator failing to get proper latency.
690 $ref: "#/$defs/types/positive_integer"
693 Rounded average latency time measured in this trial.
694 See unit property for the unit of measurement.
695 $ref: "#/$defs/types/nonnegative_integer"
698 Base64-encoded compressed representation of HDRHistogram
699 of all latency sample times encountered
700 in this latency trial.
701 See unit property for the unit of measurement.
702 Note that some bins can be several units wide.
706 Unit of measurement for latency times.
707 Currently a constant, but later versions
708 of the model may allow more values.
720 additionalProperties: false
724 Object related to latency measurement performed
725 at minimal rate (currently 9000 pps per direction).
726 $ref: "#/$defs/macros/latency_numbers"
729 Object related to latency measurement performed
730 at 10% of PDR lower bound, if needed rounded up
731 to minimal rate (currently 9000 pps per direction).
732 $ref: "#/$defs/macros/latency_numbers"
735 Object related to latency measurement performed
736 at 50% of PDR lower bound, if needed rounded up
737 to minimal rate (currently 9000 pps per direction).
738 $ref: "#/$defs/macros/latency_numbers"
741 Object related to latency measurement performed
742 at 90% of PDR lower bound, if needed rounded up
743 to minimal rate (currently 9000 pps per direction).
744 $ref: "#/$defs/macros/latency_numbers"