1 # Copyright (c) 2021 Cisco and/or its affiliates.
2 # Licensed under the Apache License, Version 2.0 (the "License");
3 # you may not use this file except in compliance with the License.
4 # You may obtain a copy of the License at:
6 # http://www.apache.org/licenses/LICENSE-2.0
8 # Unless required by applicable law or agreed to in writing, software
9 # distributed under the License is distributed on an "AS IS" BASIS,
10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11 # See the License for the specific language governing permissions and
12 # limitations under the License.
15 | Documentation | Performance suite keywords - Handling of various values
16 | ... | to allow autodetection, computation and overriding by suite variables.
17 | Library | Collections
18 | Variables | ${CURDIR}/../../python/Constants.py
21 | ${extended_debug}= | ${EXTENDED_DEBUG}
26 | | ... | If Get Use Latency returns true, return false.
27 | | ... | Otherwise return value of \${disable_latency} variable,
28 | | ... | or \${False} if not defined.
30 | | ... | The return value controls whether latency trials in NDRPDR tests
31 | | ... | are executed. For example, ASTF tests do not support latency
32 | | ... | measurements yet, so executing the trials just wastes time.
33 | | ... | Return type: bool.
37 | | ... | \| \${disable_latency} = \| Get Disable Latency \|
39 | | ${use_latency} = | Get Use Latency
40 | | Return From Keyword If | ${use_latency} | ${False}
41 | | ${disable_latency} = | Get Variable Value | \${disable_latency} | ${False}
42 | | Return From Keyword | ${disable_latency}
46 | | ... | Return value of \${max_rate} variable,
47 | | ... | fail if it is not defined.
48 | | ... | Call this just before calling a Python keyword,
49 | | ... | as those have restricted access to Robot variables.
51 | | ... | The return value controls the maximal unidirectional packet rate.
52 | | ... | The value is also usable for minimal TPS value for ASTF tests.
53 | | ... | Return type: float.
57 | | ... | \| \${max_rate} = \| Get Max Rate \|
59 | | ${max_rate} = | Get Variable Value | \${max_rate} | ${0.0}
60 | | Return From Keyword If | ${max_rate} | ${max_rate}
61 | | Fail | \${max_rate} is not defined. Call Set Max Rate And Jumbo keyword.
65 | | ... | Return a hardcoded value.
66 | | ... | The return value controls the minimal unidirectional packet rate,
67 | | ... | to be used anywhere, including latency measurements at 0% load.
68 | | ... | The current value is the smallest one permitted
69 | | ... | by STL profiles with latency streams.
70 | | ... | Return type: float.
74 | | ... | \| \${min_rate_hard} = \| Get Min Rate Hard \|
76 | | Return From Keyword | ${9001.0}
80 | | ... | If ramp up rate is not defined, return the hard min value.
81 | | ... | If ramp up rate is defined (and larger than hard min), return that.
82 | | ... | The reason is, ramp up rate should already guarantee no loss.
84 | | ... | The return value controls the minimal unidirectional packet rate,
85 | | ... | to be used in various search algorithms.
86 | | ... | Latency measurements may want even lower loads, use hard min for that.
88 | | ... | The value is also usable for minimal TPS value for ASTF tests.
89 | | ... | Return type: float.
91 | | ... | Currently, undefined ramp up rate is reported as zero,
92 | | ... | so we return the maximum of ramp up rate and the hard min rate.
96 | | ... | \| \${min_rate_soft} = \| Get Min Rate Soft \|
98 | | ${min_rate_hard} = | Get Min Rate Hard
99 | | ${ramp_up_rate} = | Get Ramp Up Rate
100 | | ${min_rate_soft} = | Evaluate | max(${ramp_up_rate}, ${min_rate_hard})
101 | | Return From Keyword | ${min_rate_soft}
103 | Get Mrr Trial Duration
105 | | ... | Return value from Constants. This is an abstraction, useful in case
106 | | ... | we start allowing various other overrides or computations.
107 | | ... | Call this just before calling a Python keyword,
108 | | ... | as those have restricted access to Robot variables.
110 | | ... | The return value controls the duration of main trial measurement
111 | | ... | for MRR type tests.
112 | | ... | Return type: float.
116 | | ... | \| \${mrr_trial_duration} = \| Get Mrr Trial Duration \|
118 | | Return From Keyword | ${PERF_TRIAL_DURATION}
120 | Get Mrr Trial Multiplicity
122 | | ... | Return value from Constants. This is an abstraction, useful in case
123 | | ... | we start allowing various other overrides or computations.
124 | | ... | Call this just before calling a Python keyword,
125 | | ... | as those have restricted access to Robot variables.
127 | | ... | The return value controls the number of main trial measurement
128 | | ... | for (B)MRR type tests.
129 | | ... | Return type: integer.
133 | | ... | \| \${mrr_trial_multiplicity} = \| Get Mrr Trial Multiplicity \|
135 | | Return From Keyword | ${PERF_TRIAL_MULTIPLICITY}
137 | Get Packet Loss Ratio
139 | | ... | Return a hardcoded value. This is an abstraction, useful in case
140 | | ... | we start allowing various other overrides or computations.
141 | | ... | Call this just before calling a Python keyword,
142 | | ... | as those have restricted access to Robot variables.
144 | | ... | The return value controls the default packet loss ration for PDR
145 | | ... | in NDRPDR tests. Some other usages of MLRsearch (e.g. reconf tests)
146 | | ... | may use a different value.
147 | | ... | Return type: float.
151 | | ... | \| \${packet_loss_ratio} = \| Get Packet Loss Ratio \|
153 | | Return From Keyword | ${0.005}
155 | Get Packets Per Transaction Aggregated
157 | | ... | Return value of \${packets_per_transaction_aggregated};
158 | | ... | if not defined, assume traffic is symmetric and compute
159 | | ... | from unidirectional values.
161 | | ... | The return value is used when reporting PPS values from TPS found
162 | | ... | by some search (e.g. NDRPDR).
163 | | ... | Return type: integer.
167 | | ... | \| \${ppta} = \| Get Packets Per Transaction Aggregated \|
169 | | ${ppta} = | Get Variable Value | \${packets_per_transaction_aggregated}
171 | | Return From Keyword If | "${ppta}" != "0" | ${ppta}
172 | | # TODO: Insert TCP computation from packet size here.
173 | | ${pptad} = | Get Packets Per Transaction And Direction
174 | | ${traffic_directions} = | Get Traffic Directions
175 | | # We do not support ASTF profiles with multiple transactions, yet.
176 | | ${ppta} = | Evaluate | ${pptad} * ${traffic_directions}
177 | | Return From Keyword | ${ppta}
179 | Get Packets Per Transaction And Direction
181 | | ... | Return value of \${packets_per_transaction_and_direction},
182 | | ... | or ${1} if not defined.
184 | | ... | The return value is used when computing max rate (TPS),
185 | | ... | so for asymmetric transaction use the more numerous direction.
186 | | ... | Return type: integer.
190 | | ... | \| \${pptad} = \| Get Packets Per Transaction And Direction \|
192 | | ${pptad} = | Get Variable Value | \${packets_per_transaction_and_direction}
194 | | Return From Keyword | ${pptad}
196 | Get Ramp Up Duration
198 | | ... | Return value of \${ramp_up_duration},
199 | | ... | or ${0.0} if not defined.
201 | | ... | The return value determines the required duration of ramp-up phase.
202 | | ... | Typically used to prepare a specific state on DUT.
203 | | ... | If the value is zero, ramp-up phase is either skipped or size-limited.
204 | | ... | Return type: float.
208 | | ... | \| \${ramp_up_duration} = \| Get Ramp Up Duration \|
210 | | ${ramp_up_duration} = | Get Variable Value | \${ramp_up_duration} | ${0.0}
211 | | Return From Keyword | ${ramp_up_duration}
215 | | ... | Return value of \${ramp_up_rate},
216 | | ... | if not defined, return zero.
218 | | ... | The return value determines the rate for ramp-up phase.
219 | | ... | Typically used to limit the rate when max rate
220 | | ... | would lose packets in the ramp up phase, thus not setting
221 | | ... | the DUT state correctly.
222 | | ... | If the value is zero, ramp-up phase should be skipped.
223 | | ... | Return type: float.
227 | | ... | \| \${ramp_up_rate} = \| Get Ramp Up Rate \|
229 | | ${ramp_up_rate} = | Get Variable Value | \${ramp_up_rate} | ${0.0}
230 | | Return From Keyword | ${ramp_up_rate}
232 | Get Rate For Teardown
234 | | ... | Return value of \${rate_for_teardown},
235 | | ... | if not defined (or zero) return the soft min rate.
237 | | ... | The return value determines the rate for teardown trial,
238 | | ... | that is executed if a perf test fails.
239 | | ... | The \${rate_for_teardown} is usually not defined in suite,
240 | | ... | but search keywords set it in places where failure can occur,
241 | | ... | so the trial is done at the rate interesting for the failure.
242 | | ... | Return type: float.
246 | | ... | \| \${rate_for_teardown} = \| Get Rate For Teardown \|
248 | | ${rate_for_teardown} = | Get Variable Value | \${rate_for_teardown} | ${0.0}
249 | | Return From Keyword If | ${rate_for_teardown} | ${rate_for_teardown}
250 | | Run Keyword And Return | Get Min Rate Soft
254 | | ... | Return value of \${resetter} variable,
255 | | ... | or \${None} if not defined.
257 | | ... | If not \${None}, the returned value is callable.
258 | | ... | Its use is to reset DUT to initial conditions,
259 | | ... | for example to remove NAT sessions created in the previous trial.
263 | | ... | \| \${resetter} = \| Get Resetter \|
265 | | ${resetter} = | Get Variable Value | \${resetter} | ${None}
266 | | Return From Keyword | ${resetter}
268 | Get Runtime Duration
270 | | ... | Return value of \${runtime_duration} variable,
271 | | ... | if not defined return ${1.0}.
273 | | ... | The return value controls the duration of runtime trial,
274 | | ... | which also acts as a warmup. Usually one second is enough,
275 | | ... | but some suites need longer time to set up state on DUT.
276 | | ... | Return type: float.
280 | | ... | \| \${runtime_duration} = \| Get Runtime Duration \|
282 | | ${runtime_duration} = | Get Variable Value | \${runtime_duration} | ${1.0}
283 | | Return From Keyword | ${runtime_duration}
287 | | ... | Return value of \${runtime_rate} variable,
288 | | ... | if not defined return the max rate.
290 | | ... | The return value controls the rate (TPS unidir) of runtime trial,
291 | | ... | which also acts as a warmup. No plans to ever use a different rate,
292 | | ... | but keywords look better if access to such values is uniform.
293 | | ... | Return type: float.
297 | | ... | \| \${runtime_rate} = \| Get Runtime Rate \|
299 | | ${runtime_rate} = | Get Variable Value | \${runtime_rate} | ${0.0}
300 | | Return From Keyword If | ${runtime_rate} | ${runtime_rate}
301 | | Run Keyword And Return | Get Max Rate
303 | Get Traffic Directions
305 | | ... | Return value of \${traffic_directions},
306 | | ... | or ${2} if not defined.
308 | | ... | The return value used when parsing for measurement results.
309 | | ... | This needs to be known already in profile driver,
310 | | ... | as bidirectional parsing may fail on unidirectional traffic.
311 | | ... | Return type: integer.
315 | | ... | \| \${traffic_directions} = \| Get Traffic Directions \|
317 | | ${traffic_directions} = | Get Variable Value | \${traffic_directions} | ${2}
318 | | Return From Keyword | ${traffic_directions}
320 | Get Transaction Duration
322 | | ... | Return value of \${transaction_duration} variable,
323 | | ... | or \${0.0} if not defined.
325 | | ... | The return value is the expected duration of single (ASTF) transaction
326 | | ... | if it is not negligible for overall trial duration computation.
327 | | ... | Most tests use very short transactions (without explicit delays),
328 | | ... | so the zero default works (and suite saves one line
329 | | ... | of Variables table).
330 | | ... | Return type: float.
334 | | ... | \| \${transaction_duration} = \| Get Transaction Duration \|
336 | | ${transaction_duration} = | Get Variable Value | \${transaction_duration}
338 | | Return From Keyword | ${transaction_duration}
340 | Get Transaction Scale
342 | | ... | Return value of \${transaction_scale} variable,
343 | | ... | or \${0} if not defined.
345 | | ... | Zero return value means the number of transactions is not limited,
346 | | ... | which is true for most STL TRex profiles (transaction is a packet).
347 | | ... | Nonzero return value means the number of transactions is fixed,
348 | | ... | for example in stateful NAT scale tests.
349 | | ... | Return type: integer.
353 | | ... | \| \${transaction_scale} = \| Get Transaction Scale \|
355 | | ${transaction_scale} = | Get Variable Value | \${transaction_scale} | ${0}
356 | | Return From Keyword | ${transaction_scale}
358 | Get Transaction Type
360 | | ... | Return value of \${transaction_type} variable,
361 | | ... | or "packet" if not defined.
363 | | ... | The return value describes the type of transaction
364 | | ... | the test is executed. For example "packet" means a transaction
365 | | ... | is just a single packet. For more sophisticated transactions,
366 | | ... | the logic to determine the number of passed transactions
367 | | ... | is different from merely counting the packets received from DUT.
368 | | ... | Return type: string.
372 | | ... | \| \${transaction_type} = \| Get Transaction Type \|
374 | | ${transaction_type} = | Get Variable Value | \${transaction_type} | packet
375 | | Return From Keyword | ${transaction_type}
379 | | ... | Return value of \${use_latency} variable,
380 | | ... | if not defined return the value from Constants.
382 | | ... | The return value controls whether latency streams are active
383 | | ... | during the main search.
384 | | ... | Return type: bool.
388 | | ... | \| \${use_latency} = \| Get Use Latency \|
390 | | ${use_latency} = | Get Variable Value | ${use_latency} | ${PERF_USE_LATENCY}
391 | | Return From Keyword | ${use_latency}
395 | | ... | For jumbo frames detection, the maximal packet size is relevant,
396 | | ... | encapsulation overhead (if any) has effect.
398 | | ... | This keyword computes jumbo boolean (some suites need that for
399 | | ... | configuration decisions).
400 | | ... | To streamline suite autogeneration, both input and output values
401 | | ... | are communicated as test (or broader scope) variables,
402 | | ... | instead of explicit arguments and return values.
404 | | ... | *Test (or broader scope) variables read:*
405 | | ... | - overhead - Overhead in bytes; default value: 0. Type: integer
406 | | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
409 | | ... | *Test variables set:*
410 | | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be
411 | | ... | enabled. Type: boolean
415 | | ... | \| Set Jumbo \|
417 | | # Already called by Set Max Rate And Jumbo, but some suites (e.g. device)
418 | | # are calling this directly.
419 | | Set Numeric Frame Sizes
420 | | ${jumbo} = | Set Variable If | ${max_frame_size} < 1522
421 | | ... | ${False} | ${True}
422 | | Set Test Variable | \${jumbo}
424 | Set Max Rate And Jumbo
426 | | ... | Input framesize can be either integer in case of a single packet
427 | | ... | in stream, or IMIX string defining mix of packets.
428 | | ... | For jumbo frames detection, the maximal packet size is relevant.
429 | | ... | For maximal transmit rate, the average packet size is relevant.
430 | | ... | In both cases, encapsulation overhead (if any) has effect.
431 | | ... | The maximal rate is computed from NIC name.
432 | | ... | The implementation works by mapping from exact
433 | | ... | whitelisted NIC names.
434 | | ... | The mapping is hardcoded in nic_limits.yaml
435 | | ... | TODO: Make the mapping from NIC names case insensistive.
437 | | ... | This keyword computes maximal unidirectional transmit rate
438 | | ... | and jumbo boolean (some suites need that for configuration decisions).
439 | | ... | To streamline suite autogeneration, both input and output values
440 | | ... | are communicated as test (or broader scope) variables,
441 | | ... | instead of explicit arguments and return values.
443 | | ... | If this keyword detects the test is interested in (unidirectional)
444 | | ... | transactons per second maximal rate (tps), that is returned (not pps).
446 | | ... | *Test (or broader scope) variables read:*
447 | | ... | - nic_name - Name of bottleneck NIC. Type: string
448 | | ... | - overhead - Overhead in bytes; default value: 0. Type: integer
449 | | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
451 | | ... | - packets_per_transaction_and_direction - Pps-tps conversion.
452 | | ... | Optional, default 1.
454 | | ... | *Test variables set:*
455 | | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps].
456 | | ... | This never exceeds bandwidth on TG-DUT nor DUT-DUT links.
457 | | ... | Type: float
458 | | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be
459 | | ... | enabled. Type: boolean
460 | | ... | avg_frame_size - Average frame size including overhead. Type: float
461 | | ... | max_frame_size - Maximal frame size including overhead. Type: float
465 | | ... | \| Set Max Rate And Jumbo \|
467 | | # TODO: Re-check overhead values in suites with both traffics encapsulated.
468 | | # TODO: Improve layered setup to detect encap/decap and update overhead.
469 | | ${pps_limit} = | Get From Dictionary
470 | | ... | ${NIC_NAME_TO_PPS_LIMIT} | ${nic_name}
471 | | ${bps_limit} = | Get From Dictionary
472 | | ... | ${NIC_NAME_TO_BPS_LIMIT} | ${nic_name}
473 | | Set Numeric Frame Sizes
474 | | # We need to add 20B (Ethernet preamble and inter-frame gap)
475 | | # to avg_frame_size
476 | | ${rate} = | Evaluate | ${bps_limit} / ((${avg_frame_size} + 20.0) * 8)
477 | | ${max_rate} = | Set Variable If | ${rate} > ${pps_limit}
478 | | ... | ${pps_limit} | ${rate}
479 | | ${pptad} = | Get Packets Per Transaction And Direction
480 | | ${max_rate} = | Evaluate | ${max_rate} / ${pptad}
481 | | Set Test Variable | \${max_rate}
484 | Set Numeric Frame Sizes
486 | | ... | Framesize can be either integer in case of a single packet
487 | | ... | in stream, or set of packets in case of IMIX type or simmilar.
488 | | ... | For jumbo decisions, we need a numeric size of the biggest packet.
489 | | ... | For max rate decisions, we need a numeric average packet size.
490 | | ... | This keyword computes both and sets them as test variables.
492 | | ... | Each suite sets a value named \${overhead},
493 | | ... | which describes by how many bytes the frames on DUT-DUT link
494 | | ... | are larger (due to encapsulation) than those
495 | | ... | on the primary TG-DUT link. But for some suites that value
496 | | ... | can be negaive (if TG-DUT is encapsulated more heavily).
497 | | ... | For calculations in this keyword, we need largest sizes
498 | | ... | across links, so zero is used if \${overhead} is negative.
500 | | ... | *Test variables read:*
501 | | ... | - frame_size - Framesize. Type: integer or string
502 | | ... | - overhead - Overhead in bytes; default value: ${0}. Type: integer
504 | | ... | *Test variables set*
505 | | ... | avg_frame_size - Average frame size including overhead. Type: float
506 | | ... | max_frame_size - Maximal frame size including overhead. Type: float
510 | | ... | \| Set Numeric Frame Sizes \|
512 | | ${max_overhead} = | Set Variable If | ${overhead} >= 0 | ${overhead} | ${0}
513 | | ${bare_avg_frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1'
514 | | ... | Set Variable | ${353.83333}
516 | | ... | Convert To Number | ${frame_size}
517 | | ${avg_frame_size} = | Evaluate | $bare_avg_frame_size + $max_overhead
518 | | Set Test Variable | \${avg_frame_size}
519 | | ${bare_max_frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1'
520 | | ... | Set Variable | ${1518}
522 | | ... | Convert To Number | ${frame_size}
523 | | ${max_frame_size} = | Evaluate | $bare_max_frame_size + $max_overhead
524 | | Set Test Variable | ${max_frame_size}
526 | Set Rates For Policer
528 | | ... | Policer tests need these values,
529 | | ... | currently computed from \${avg_frame_size}.
530 | | ... | TODO: Verify the units match and computation is correct.
532 | | ... | *Test (or broader scope) variables read:*
533 | | ... | - avg_frame_size - Average L2 Frame Size [B]. Type: float
534 | | ... | Set by Set Max Rate And Jumbo keyword.
536 | | ... | *Test variables set:*
537 | | ... | - eb - Excess burst rate for policer. Type: float
538 | | ... | - cb - Committed burst rate for policer. Type: float
542 | | ... | \| Set Rates For Policer \|
544 | | Set Test Variable | \${eb} | ${avg_frame_size}
545 | | Set Test Variable | \${cb} | ${avg_frame_size}