1 # Copyright (c) 2023 Cisco and/or its affiliates.
2 # Licensed under the Apache License, Version 2.0 (the "License");
3 # you may not use this file except in compliance with the License.
4 # You may obtain a copy of the License at:
6 # http://www.apache.org/licenses/LICENSE-2.0
8 # Unless required by applicable law or agreed to in writing, software
9 # distributed under the License is distributed on an "AS IS" BASIS,
10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11 # See the License for the specific language governing permissions and
12 # limitations under the License.
16 $id: https://fd.io/FIXME/CSIT/UTI/test_case/info/1.3.0
17 $schema: https://json-schema.org/draft/2020-12/schema
19 Schema for output of test case.
22 The main structure, without conditional relations between fields yet.
24 additionalProperties: false
28 A derived quantity. Difference between start_time and end_time,
30 $ref: "#/$defs/types/nonnegative_number"
33 DUT type used, e.g. VPP or DPDK.
38 Version string appropriate to DUT type used.
43 UTC date and time in RFC 3339 format, specifying calendar time
44 just before test case ended (at the end of test case teardown).
45 $ref: "#/$defs/types/date_time"
48 Array of hosts this test interacted with.
49 This can be used for identifying testbed number.
50 Valid tests shoud interact with at least one DUT or TG.
51 The array is usually sorted, but that is not a requirement.
56 Host identifier, usually numeric IPv4 address.
60 Array of telemetry entries. Each entry represent one captured
70 If passed is true, this value is empty.
71 Otherwise, value taken directly from TEST_MESSAGE
72 Robot variable, read at the end of test case
73 (in test teardown, before export and validation).
74 It contains information from the exception
75 that caused the failure, probably with additional
76 exceptions from teardown keywords.
80 Value set accordingly to TEST_STATUS Robot variable,
81 true if and only if the status is "PASS".
82 The status is read at the end of test case
83 (in test teardown, before export and validation).
89 Sub-schema common for all cases,
90 only result type identifier defined here.
94 Identifier of which result type case is applied.
100 Result type for unknown case.
101 This case represents a test with no specific result
102 (outside message), e.g. device test;
103 or a test with result not parsed into
104 this version of model yet, e.g. GSO test.
105 additionalProperties: false
110 Result type MRR case.
111 additionalProperties: false
117 The results refer to receive rates for multiple
118 MRR trials. For PPS, these are aggregate
119 (bidirectional) rates.
120 Currently, the tests are exporting
121 approximated receive rates.
122 That means the actual trial duration
123 is measured (as opposed to trusting traffic
124 generator to honor its target duration),
125 so the resulting values contain noise
126 from time measurement, and can be lower
127 than the real performance
128 (due to various time overheads).
129 Bandwidth values are supported, but currently
130 Robot does not export them.
131 $ref: "#/$defs/types/rate_list_with_bandwidth"
136 Result type NDRPDR case.
137 additionalProperties: false
143 The results refer to search for NDR
144 (Non Drop Rate). For PPS, this is aggregate
145 (bidirectional) rate.
146 Each bound was used as the target load value
147 in a full-duration trial measurement.
148 The accepted loss ratio for NDR is exact zero.
149 Note that packets the Traffic Generator
150 did not send are also counted as lost packets.
151 $ref: "#/$defs/macros/lower_and_upper_rate"
154 The results refer to search for PDR
155 (Partial Drop Rate). For PPS, this is aggregate
156 (bidirectional) rate.
157 Each bound was used as the target load value
158 in a full-duration trial measurement.
159 The accepted loss ratio for PDR is 0.5%.
160 Note that packets the Traffic Generator
161 did not send are also counted as lost packets.
162 $ref: "#/$defs/macros/lower_and_upper_rate"
165 Object with results related to latency part
166 of NDRPDR test, for forward traffic direction.
167 It is the direction used in unidirectional
169 ASTF profiles and IMIX STL profiles
170 do not support latency information,
171 so for those tests this object is missing.
172 It is also missing if Traffic Generator
173 fails to return valid latency results
174 for any other reasons,
175 e.g. latency rate is too high for CPU/NIC used.
176 $ref: "#/$defs/macros/latency_for_loads"
179 Object with results related to latency part
180 of NDRPDR test, for reverse traffic diration.
181 This object is not present
182 when unidirectional traffic profiles are used.
183 ASTF profiles and IMIX STL profiles
184 do not support latency information,
185 so for those tests this object is missing.
186 It is also missing if Traffic Generator
187 fails to return valid latency results
188 for any other reasons,
189 e.g. latency rate is too high for CPU/NIC used.
190 $ref: "#/$defs/macros/latency_for_loads"
196 Result type SOAK case.
197 additionalProperties: false
203 The results refer to bayesian estimate
204 of critical rate corresponding to
205 average loss ratio of 10^-7.
206 For PPS, this is aggregate (bidirectional) rate.
207 The bounds are computed from
208 trial measurement results,
209 but are not equal to any target load used.
210 Note that packets the Traffic Generator
211 did not send are also counted as lost packets.
212 $ref: "#/$defs/macros/lower_and_upper_rate"
217 Result type RECONF case.
218 additionalProperties: false
224 Load used when reconfiguring, found as NDR lower
225 bound. This is an aggregate (bidirectional)
226 rate. Note that packets which the Traffic
227 Generator did not send, are also counted as lost
229 $ref: "#/$defs/types/rate_with_bandwidth"
232 Number of packets lost during reconfiguration,
233 with the time that equals packet loss divided by
235 $ref: "#/$defs/types/packet_with_time"
241 Result type HOSTSTACK case.
242 additionalProperties: false
248 Goodput measured in bits per second.
249 $ref: "#/$defs/types/bandwidth"
252 Number of completed requests.
253 $ref: "#/$defs/types/count_requests"
256 Number of failed requests.
257 $ref: "#/$defs/types/count_requests"
260 Number of retransmits.
261 $ref: "#/$defs/types/count_packets"
264 Value and unit of latency.
265 $ref: "#/$defs/types/value_with_unit"
268 The relative time difference (in seconds)
269 between program start and end.
270 $ref: "#/$defs/types/time_quantity"
273 RPS or CPS rate, with corresponding unit, as
275 $ref: "#/$defs/types/rate_without_bandwidth"
281 UTC date and time in RFC 3339 format, specifying calendar time
282 just after test case started (at the start of test setup).
283 $ref: "#/$defs/types/date_time"
286 The list of strings comes directly
287 from Robot variable TEST_TAGS.
288 The content should include both static and dynamic tags
289 at the end of test case (teardown).
295 Value taken directly from TEST_DOCUMENTATION Robot variable.
296 The content is what you see in suite file
297 at test case definition, which is usually empty
298 as CSIT uses data driven test cases.
303 It is the most complete and unique identifier for a test case.
304 This property has a value, of the following form:
305 {suite_name}.{test_name}
306 Here, suite name comes from SUITE_NAME robot variable,
307 test name comes from TEST_NAME robot variable,
308 but both are converted to lower case,
309 and spaces are replaced by underscores.
315 This property has a value, of the following form:
316 {nic_short_name}-{frame_size}-{threads_and_cores}-{suite_part}
317 Here, suite part is very similar to suite tag,
318 but additionally may contain a prefix describing NIC driver used
319 (if it is not the default one, drv_vfio_pci for VPP tests).
320 Any space is replaced by underscore and letters are lower case.
326 This property has a value very similar to suite tag,
327 but additionally may contain a prefix describing NIC driver used
328 (if it is not the default one, drv_vfio_pci for VPP tests).
329 Any space is replaced by underscore and letters are lower case.
335 Test type identifier, PAL uses it to group similar tests,
336 e.g. for comparison tables.
337 Ideally, this information should be parseable from test name,
338 but the current naming scheme is not simple/consistent enough.
339 The current implementation queries the robot test tags.
340 The resulting value is frequently identical to result type,
341 but this schema version does not require any relation there,
342 as PAL may want to group tests differently.
355 TG type used, e.g. TREX.
360 Version string appropriate to TG type used.
365 CSIT model version (semver format)
366 the exporting code adhered to.
390 Subschema validating relation between status and message.
393 Subschema for passing tests, message has to be empty.
401 Subschema for failing tests, mesage cannot be empty.
431 Packets per second (pps),
432 connections per second (cps),
433 requests per second (rps).
441 Unit of measurement for bandwidth values.
442 Currently a constant, but later versions of model
443 may allow more units.
448 Type, for counting packets.
450 - $ref: "#/$defs/types/value_with_unit"
454 A number of packets of interest.
457 Unit suitable for displaying packet counts.
462 Type, for counting requests.
464 - $ref: "#/$defs/types/value_with_unit"
468 A number of requests of interest.
471 Unit suitable for displaying request counts.
476 Reusable type, for various time quantites.
478 - $ref: "#/$defs/types/value_with_unit"
482 Unless specified otherwise, this is a duration
486 Only seconds are the unit supported for time
492 Reusable composite type, value together with its
495 additionalProperties: false
499 Numeric value, context specified elsewhere.
500 The only assumption is that value is not negative.
501 $ref: "#/$defs/types/nonnegative_number"
504 Unit of measurement for the value.
505 Context and allowed values are specified elsewhere.
510 rate_without_bandwidth:
512 Reusable type, for various rate quantites.
514 - $ref: "#/$defs/types/value_with_unit"
518 Unless specified otherwise,
519 this is the aggregated rate
520 (sum of both traffic directions).
521 Depending on the usage, the value can express
522 intended load, offered load, receive rate,
523 and various approximations
524 or estimated bounds thereof.
527 A transaction rate unit the value is expressed in.
528 $ref: "#/$defs/types/rate_unit"
531 Reusable type, for various bandwidth quantites.
533 - $ref: "#/$defs/types/value_with_unit"
537 Bandwidth value computed from the corresponding
540 $ref: "#/$defs/types/bandwidth_unit"
543 Reusable composite type, joining primary rate
544 with optional derived bandwidth.
545 Not all test types currently compute bandwidth,
546 even if rate unit is pps.
548 additionalProperties: false
551 $ref: "#/$defs/types/rate_without_bandwidth"
553 $ref: "#/$defs/types/bandwidth"
558 Reusable composite type, joining packet count with the
561 additionalProperties: false
564 $ref: "#/$defs/types/count_packets"
566 $ref: "#/$defs/types/time_quantity"
570 value_list_with_unit_and_stats:
572 Reusable composite type, multiple values together with their
573 unit of measurement and derived statistics.
575 additionalProperties: false
579 List of values of the same unit, useful for MRR.
584 Numeric value, context specified elsewhere. The only
585 assumption is that the value is nonnegative.
586 $ref: "#/$defs/types/nonnegative_number"
589 A derived quantity. It is the arithmetic average of the
591 $ref: "#/$defs/types/nonnegative_number"
594 A derived quantity. It is the standard deviation for the
595 values list, as computed by jumpavg library.
596 $ref: "#/$defs/types/nonnegative_number"
599 Unit of measurement for the values.
600 Context and allowed values are specified elsewhere.
607 rate_list_without_bandwidth:
609 Reusable composite type, multiple rate values.
611 - $ref: "#/$defs/types/value_list_with_unit_and_stats"
616 Unless specified otherwise,
617 this is the aggregated rate
618 (sum of both traffic directions).
619 Depending on the usage, the value can express
620 intended load, offered load, receive rate,
621 and various approximations or estimated bounds
624 $ref: "#/$defs/types/rate_unit"
627 Reusable composite type, multiple bandwidth values. This is a
630 - $ref: "#/$defs/types/value_list_with_unit_and_stats"
635 Unless specified otherwise,
636 this is the aggregated bandwidth
637 (sum of both traffic directions).
638 Depending on the usage, the value can express
639 intended load, offered load, receive rate,
640 and various approximations or estimated bounds
643 $ref: "#/$defs/types/bandwidth_unit"
644 rate_list_with_bandwidth:
646 Reusable composite type, joining primary rates
647 with optional derived bandwidths (and stats).
648 No test types currently computes the bandwidth part.
650 additionalProperties: false
653 $ref: "#/$defs/types/rate_list_without_bandwidth"
655 $ref: "#/$defs/types/bandwidth_list"
659 lower_and_upper_rate:
661 additionalProperties: false
665 The lower bound (or min_rate) for the estimate
666 of a particular searched value.
667 $ref: "#/$defs/types/rate_with_bandwidth"
670 The upper bound (or max_rate) for the estimate
671 of a particular searched value.
672 $ref: "#/$defs/types/rate_with_bandwidth"
678 additionalProperties: false
682 Rounded minimal latency time measured in this trial.
683 See unit property for the unit of measurement.
684 $ref: "#/$defs/types/nonnegative_integer"
687 Rounded maximal latency time measured in this trial.
688 See unit property for the unit of measurement.
689 Zero value is not allowed, as that is one of symptoms
690 of Traffic Generator failing to get proper latency.
691 $ref: "#/$defs/types/positive_integer"
694 Rounded average latency time measured in this trial.
695 See unit property for the unit of measurement.
696 $ref: "#/$defs/types/nonnegative_integer"
699 Base64-encoded compressed representation of HDRHistogram
700 of all latency sample times encountered
701 in this latency trial.
702 See unit property for the unit of measurement.
703 Note that some bins can be several units wide.
707 Unit of measurement for latency times.
708 Currently a constant, but later versions
709 of the model may allow more values.
721 additionalProperties: false
725 Object related to latency measurement performed
726 at minimal rate (currently 9000 pps per direction).
727 $ref: "#/$defs/macros/latency_numbers"
730 Object related to latency measurement performed
731 at 10% of PDR lower bound, if needed rounded up
732 to minimal rate (currently 9000 pps per direction).
733 $ref: "#/$defs/macros/latency_numbers"
736 Object related to latency measurement performed
737 at 50% of PDR lower bound, if needed rounded up
738 to minimal rate (currently 9000 pps per direction).
739 $ref: "#/$defs/macros/latency_numbers"
742 Object related to latency measurement performed
743 at 90% of PDR lower bound, if needed rounded up
744 to minimal rate (currently 9000 pps per direction).
745 $ref: "#/$defs/macros/latency_numbers"