1 # Copyright (c) 2023 Cisco and/or its affiliates.
2 # Licensed under the Apache License, Version 2.0 (the "License");
3 # you may not use this file except in compliance with the License.
4 # You may obtain a copy of the License at:
6 # http://www.apache.org/licenses/LICENSE-2.0
8 # Unless required by applicable law or agreed to in writing, software
9 # distributed under the License is distributed on an "AS IS" BASIS,
10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11 # See the License for the specific language governing permissions and
12 # limitations under the License.
16 $id: https://fd.io/FIXME/CSIT/UTI/test_case/info/1.4.0
17 $schema: https://json-schema.org/draft/2020-12/schema
19 Schema for output of test case.
22 The main structure, without conditional relations between fields yet.
24 additionalProperties: false
28 A derived quantity. Difference between start_time and end_time,
30 $ref: "#/$defs/types/nonnegative_number"
33 DUT type used, e.g. VPP or DPDK.
38 Version string appropriate to DUT type used.
43 UTC date and time in RFC 3339 format, specifying calendar time
44 just before test case ended (at the end of test case teardown).
45 $ref: "#/$defs/types/date_time"
48 Array of hosts this test interacted with.
49 This can be used for identifying testbed number.
50 Valid tests shoud interact with at least one DUT or TG.
51 The array is usually sorted, but that is not a requirement.
56 Host identifier, usually numeric IPv4 address.
60 Array of telemetry entries. Each entry represent one captured
70 If passed is true, this value is empty.
71 Otherwise, value taken directly from TEST_MESSAGE
72 Robot variable, read at the end of test case
73 (in test teardown, before export and validation).
74 It contains information from the exception
75 that caused the failure, probably with additional
76 exceptions from teardown keywords.
80 Value set accordingly to TEST_STATUS Robot variable,
81 true if and only if the status is "PASS".
82 The status is read at the end of test case
83 (in test teardown, before export and validation).
89 Sub-schema common for all cases,
90 only result type identifier defined here.
94 Identifier of which result type case is applied.
100 Result type for unknown case.
101 This case represents a test with no specific result
102 (outside message), e.g. device test;
103 or a test with result not parsed into
104 this version of model yet, e.g. GSO test.
105 additionalProperties: false
110 Result type MRR case.
111 additionalProperties: false
117 The results refer to receive rates for multiple
118 MRR trials. For PPS, these are aggregate
119 (bidirectional) rates.
120 Currently, the tests are exporting
121 approximated receive rates.
122 That means the actual trial duration
123 is measured (as opposed to trusting traffic
124 generator to honor its target duration),
125 so the resulting values contain noise
126 from time measurement, and can be lower
127 than the real performance
128 (due to various time overheads).
129 Bandwidth values are supported, but currently
130 Robot does not export them.
131 $ref: "#/$defs/types/rate_list_with_bandwidth"
136 Result type NDRPDR case.
137 additionalProperties: false
143 The results refer to search for NDR
144 (Non Drop Rate). For PPS, this is aggregate
145 (bidirectional) rate.
146 Each bound was used as the target load value
147 in a full-duration trial measurement.
148 The accepted loss ratio for NDR is exact zero.
149 Note that packets the Traffic Generator
150 did not send are also counted as lost packets.
151 $ref: "#/$defs/macros/lower_and_upper_rate"
154 The results refer to search for PDR
155 (Partial Drop Rate). For PPS, this is aggregate
156 (bidirectional) rate.
157 Each bound was used as the target load value
158 in a full-duration trial measurement.
159 The accepted loss ratio for PDR is 0.5%.
160 Note that packets the Traffic Generator
161 did not send are also counted as lost packets.
162 $ref: "#/$defs/macros/lower_and_upper_rate"
165 Object with results related to latency part
166 of NDRPDR test, for forward traffic direction.
167 It is the direction used in unidirectional
169 ASTF profiles and IMIX STL profiles
170 do not support latency information,
171 so for those tests this object is missing.
172 It is also missing if Traffic Generator
173 fails to return valid latency results
174 for any other reasons,
175 e.g. latency rate is too high for CPU/NIC used.
176 $ref: "#/$defs/macros/latency_for_loads"
179 Object with results related to latency part
180 of NDRPDR test, for reverse traffic diration.
181 This object is not present
182 when unidirectional traffic profiles are used.
183 ASTF profiles and IMIX STL profiles
184 do not support latency information,
185 so for those tests this object is missing.
186 It is also missing if Traffic Generator
187 fails to return valid latency results
188 for any other reasons,
189 e.g. latency rate is too high for CPU/NIC used.
190 $ref: "#/$defs/macros/latency_for_loads"
196 Result type SOAK case.
197 additionalProperties: false
203 The results refer to bayesian estimate
204 of critical rate corresponding to
205 average loss ratio of 10^-7.
206 For PPS, this is aggregate (bidirectional) rate.
207 The bounds are computed from
208 trial measurement results,
209 but are not equal to any target load used.
210 Note that packets the Traffic Generator
211 did not send are also counted as lost packets.
212 $ref: "#/$defs/macros/lower_and_upper_rate"
217 Result type RECONF case.
218 additionalProperties: false
224 Load used when reconfiguring, found as NDR lower
225 bound. This is an aggregate (bidirectional)
226 rate. Note that packets which the Traffic
227 Generator did not send, are also counted as lost
229 $ref: "#/$defs/types/rate_with_bandwidth"
232 Number of packets lost during reconfiguration,
233 with the time that equals packet loss divided by
235 $ref: "#/$defs/types/packet_with_time"
241 Result type HOSTSTACK case.
242 additionalProperties: false
248 Goodput measured in bits per second.
249 $ref: "#/$defs/types/bandwidth"
252 Number of completed requests.
253 $ref: "#/$defs/types/count_requests"
256 Number of failed requests.
257 $ref: "#/$defs/types/count_requests"
260 Number of retransmits.
261 $ref: "#/$defs/types/count_packets"
264 Value and unit of latency.
265 $ref: "#/$defs/types/value_with_unit"
268 The relative time difference (in seconds)
269 between program start and end.
270 $ref: "#/$defs/types/time_quantity"
273 RPS or CPS rate, with corresponding unit, as
275 $ref: "#/$defs/types/rate_without_bandwidth"
281 UTC date and time in RFC 3339 format, specifying calendar time
282 just after test case started (at the start of test setup).
283 $ref: "#/$defs/types/date_time"
286 The list of strings comes directly
287 from Robot variable TEST_TAGS.
288 The content should include both static and dynamic tags
289 at the end of test case (teardown).
295 Value taken directly from TEST_DOCUMENTATION Robot variable.
296 The content is what you see in suite file
297 at test case definition, which is usually empty
298 as CSIT uses data driven test cases.
303 It is the most complete and unique identifier for a test case.
304 This property has a value, of the following form:
305 {suite_name}.{test_name}
306 Here, suite name comes from SUITE_NAME robot variable,
307 test name comes from TEST_NAME robot variable,
308 but both are converted to lower case,
309 and spaces are replaced by underscores.
315 This property has a value, of the following form:
316 {nic_short_name}-{frame_size}-{threads_and_cores}-{suite_part}
317 Here, suite part is very similar to suite tag,
318 but additionally may contain a prefix describing NIC driver used
319 (if it is not the default one, drv_vfio_pci for VPP tests).
320 Any space is replaced by underscore and letters are lower case.
326 This property has a value very similar to suite tag,
327 but additionally may contain a prefix describing NIC driver used
328 (if it is not the default one, drv_vfio_pci for VPP tests).
329 Any space is replaced by underscore and letters are lower case.
335 Test type identifier, PAL uses it to group similar tests,
336 e.g. for comparison tables.
337 Ideally, this information should be parseable from test name,
338 but the current naming scheme is not simple/consistent enough.
339 The current implementation queries the robot test tags.
340 The resulting value is frequently identical to result type,
341 but this schema version does not require any relation there,
342 as PAL may want to group tests differently.
354 TG type used, e.g. TREX.
359 Version string appropriate to TG type used.
364 CSIT model version (semver format)
365 the exporting code adhered to.
389 Subschema validating relation between status and message.
392 Subschema for passing tests, message has to be empty.
400 Subschema for failing tests, mesage cannot be empty.
430 Packets per second (pps),
431 connections per second (cps),
432 requests per second (rps),
433 bits per second (bps).
442 Unit of measurement for bandwidth values.
443 Currently a constant, but later versions of model
444 may allow more units.
449 Type, for counting packets.
451 - $ref: "#/$defs/types/value_with_unit"
455 A number of packets of interest.
458 Unit suitable for displaying packet counts.
463 Type, for counting requests.
465 - $ref: "#/$defs/types/value_with_unit"
469 A number of requests of interest.
472 Unit suitable for displaying request counts.
477 Reusable type, for various time quantites.
479 - $ref: "#/$defs/types/value_with_unit"
483 Unless specified otherwise, this is a duration
487 Only seconds are the unit supported for time
493 Reusable composite type, value together with its
496 additionalProperties: false
500 Numeric value, context specified elsewhere.
501 The only assumption is that value is not negative.
502 $ref: "#/$defs/types/nonnegative_number"
505 Unit of measurement for the value.
506 Context and allowed values are specified elsewhere.
511 rate_without_bandwidth:
513 Reusable type, for various rate quantites.
515 - $ref: "#/$defs/types/value_with_unit"
519 Unless specified otherwise,
520 this is the aggregated rate
521 (sum of both traffic directions).
522 Depending on the usage, the value can express
523 intended load, offered load, receive rate,
524 and various approximations
525 or estimated bounds thereof.
528 A transaction rate unit the value is expressed in.
529 $ref: "#/$defs/types/rate_unit"
532 Reusable type, for various bandwidth quantites.
534 - $ref: "#/$defs/types/value_with_unit"
538 Bandwidth value computed from the corresponding
541 $ref: "#/$defs/types/bandwidth_unit"
544 Reusable composite type, joining primary rate
545 with optional derived bandwidth.
546 Not all test types currently compute bandwidth,
547 even if rate unit is pps.
549 additionalProperties: false
552 $ref: "#/$defs/types/rate_without_bandwidth"
554 $ref: "#/$defs/types/bandwidth"
559 Reusable composite type, joining packet count with the
562 additionalProperties: false
565 $ref: "#/$defs/types/count_packets"
567 $ref: "#/$defs/types/time_quantity"
571 value_list_with_unit_and_stats:
573 Reusable composite type, multiple values together with their
574 unit of measurement and derived statistics.
576 additionalProperties: false
580 List of values of the same unit, useful for MRR.
585 Numeric value, context specified elsewhere. The only
586 assumption is that the value is nonnegative.
587 $ref: "#/$defs/types/nonnegative_number"
590 A derived quantity. It is the arithmetic average of the
592 $ref: "#/$defs/types/nonnegative_number"
595 A derived quantity. It is the standard deviation for the
596 values list, as computed by jumpavg library.
597 $ref: "#/$defs/types/nonnegative_number"
600 Unit of measurement for the values.
601 Context and allowed values are specified elsewhere.
608 rate_list_without_bandwidth:
610 Reusable composite type, multiple rate values.
612 - $ref: "#/$defs/types/value_list_with_unit_and_stats"
617 Unless specified otherwise,
618 this is the aggregated rate
619 (sum of both traffic directions).
620 Depending on the usage, the value can express
621 intended load, offered load, receive rate,
622 and various approximations or estimated bounds
625 $ref: "#/$defs/types/rate_unit"
628 Reusable composite type, multiple bandwidth values. This is a
631 - $ref: "#/$defs/types/value_list_with_unit_and_stats"
636 Unless specified otherwise,
637 this is the aggregated bandwidth
638 (sum of both traffic directions).
639 Depending on the usage, the value can express
640 intended load, offered load, receive rate,
641 and various approximations or estimated bounds
644 $ref: "#/$defs/types/bandwidth_unit"
645 rate_list_with_bandwidth:
647 Reusable composite type, joining primary rates
648 with optional derived bandwidths (and stats).
649 No test types currently computes the bandwidth part.
651 additionalProperties: false
654 $ref: "#/$defs/types/rate_list_without_bandwidth"
656 $ref: "#/$defs/types/bandwidth_list"
660 lower_and_upper_rate:
662 additionalProperties: false
666 The lower bound (or min_rate) for the estimate
667 of a particular searched value.
668 $ref: "#/$defs/types/rate_with_bandwidth"
671 The upper bound (or max_rate) for the estimate
672 of a particular searched value.
673 $ref: "#/$defs/types/rate_with_bandwidth"
679 additionalProperties: false
683 Rounded minimal latency time measured in this trial.
684 See unit property for the unit of measurement.
685 $ref: "#/$defs/types/nonnegative_integer"
688 Rounded maximal latency time measured in this trial.
689 See unit property for the unit of measurement.
690 Zero value is not allowed, as that is one of symptoms
691 of Traffic Generator failing to get proper latency.
692 $ref: "#/$defs/types/positive_integer"
695 Rounded average latency time measured in this trial.
696 See unit property for the unit of measurement.
697 $ref: "#/$defs/types/nonnegative_integer"
700 Base64-encoded compressed representation of HDRHistogram
701 of all latency sample times encountered
702 in this latency trial.
703 See unit property for the unit of measurement.
704 Note that some bins can be several units wide.
708 Unit of measurement for latency times.
709 Currently a constant, but later versions
710 of the model may allow more values.
722 additionalProperties: false
726 Object related to latency measurement performed
727 at minimal rate (currently 9000 pps per direction).
728 $ref: "#/$defs/macros/latency_numbers"
731 Object related to latency measurement performed
732 at 10% of PDR lower bound, if needed rounded up
733 to minimal rate (currently 9000 pps per direction).
734 $ref: "#/$defs/macros/latency_numbers"
737 Object related to latency measurement performed
738 at 50% of PDR lower bound, if needed rounded up
739 to minimal rate (currently 9000 pps per direction).
740 $ref: "#/$defs/macros/latency_numbers"
743 Object related to latency measurement performed
744 at 90% of PDR lower bound, if needed rounded up
745 to minimal rate (currently 9000 pps per direction).
746 $ref: "#/$defs/macros/latency_numbers"