Code Review / csit.git / blobdiff
? search:
summary | shortlog | log | commit | commitdiff | review | tree
raw | inline | side by side
CSIT-102: Add latency measurement to performance testing
[csit.git] / resources / libraries / robot / performance.robot
diff --git a/resources/libraries/robot/performance.robot b/resources/libraries/robot/performance.robot
index cb9abe5..63295b7 100644 (file)
--- a/resources/libraries/robot/performance.robot
+++ b/resources/libraries/robot/performance.robot
@@ -31,50 +31,65 @@
 | | [Documentation] | Setup performance rates as Suite Variables
 | | ...
 | | ... | _NOTE:_ This KW sets following suite variables:
-| | ... | - ${10Ge_linerate_pps_64B} - Maximum number of packet per second
-| | ... |                              for 10GE with 64B L2 Frame.
-| | ... | - ${10Ge_linerate_pps_68B} - Maximum number of packet per second
-| | ... |                              for 10GE with 68B L2 Frame.
-| | ... | - ${10Ge_linerate_pps_78B} - Maximum number of packet per second
-| | ... |                              for 10GE with 78B L2 Frame.
-| | ... | - ${10Ge_linerate_pps_1518B} - Maximum number of packet per second
-| | ... |                                for 10GE with 1518B L2 Frame.
-| | ... | - ${10Ge_linerate_pps_1522B} - Maximum number of packet per second
-| | ... |                                for 10GE with 1522B L2 Frame.
-| | ... | - ${10Ge_linerate_pps_9000B} - Maximum number of packet per second
-| | ... |                                for 10GE with 9000B L2 Frame.
-| | ... | - ${10Ge_linerate_pps_9004B} - Maximum number of packet per second
-| | ... |                                for 10GE with 9004B L2 Frame.
-| | ...
-| | ${10Ge_linerate_pps_64B}= | Set Variable | 14880952
-| | ${10Ge_linerate_pps_68B}= | Set Variable | 14204545
-| | ${10Ge_linerate_pps_78B}= | Set Variable | 12755102
-| | ${10Ge_linerate_pps_1518B}= | Set Variable | 812743
-| | ${10Ge_linerate_pps_1522B}= | Set Variable | 810635
-| | ${10Ge_linerate_pps_9000B}= | Set Variable | 138580
-| | ${10Ge_linerate_pps_9004B}= | Set Variable | 138519
-| | ${40Ge_linerate_pps_64B}= | Set Variable | 59523809
-| | ${40Ge_linerate_pps_68B}= | Set Variable | 56818181
-| | ${40Ge_linerate_pps_78B}= | Set Variable | 51020408
-| | ${40Ge_linerate_pps_1518B}= | Set Variable | 3250975
-| | ${40Ge_linerate_pps_1522B}= | Set Variable | 3242542
-| | ${40Ge_linerate_pps_9000B}= | Set Variable | 554323
-| | ${40Ge_linerate_pps_9004B}= | Set Variable | 554078
-
-| | Set Suite Variable | ${10Ge_linerate_pps_64B}
-| | Set Suite Variable | ${10Ge_linerate_pps_68B}
-| | Set Suite Variable | ${10Ge_linerate_pps_78B}
-| | Set Suite Variable | ${10Ge_linerate_pps_1518B}
-| | Set Suite Variable | ${10Ge_linerate_pps_1522B}
-| | Set Suite Variable | ${10Ge_linerate_pps_9000B}
-| | Set Suite Variable | ${10Ge_linerate_pps_9004B}
-| | Set Suite Variable | ${40Ge_linerate_pps_64B}
-| | Set Suite Variable | ${40Ge_linerate_pps_68B}
-| | Set Suite Variable | ${40Ge_linerate_pps_78B}
-| | Set Suite Variable | ${40Ge_linerate_pps_1518B}
-| | Set Suite Variable | ${40Ge_linerate_pps_1522B}
-| | Set Suite Variable | ${40Ge_linerate_pps_9000B}
-| | Set Suite Variable | ${40Ge_linerate_pps_9004B}
+| | ... | - 10Ge_linerate_pps_64B - Maximum number of packet per second
+| | ... |                           for 10GE with 64B L2 Frame.
+| | ... | - 10Ge_linerate_pps_68B - Maximum number of packet per second
+| | ... |                           for 10GE with 68B L2 Frame.
+| | ... | - 10Ge_linerate_pps_72B - Maximum number of packet per second
+| | ... |                           for 10GE with 72B L2 Frame.
+| | ... | - 10Ge_linerate_pps_78B - Maximum number of packet per second
+| | ... |                           for 10GE with 78B L2 Frame.
+| | ... | - 10Ge_linerate_pps_1518B - Maximum number of packet per second
+| | ... |                             for 10GE with 1518B L2 Frame.
+| | ... | - 10Ge_linerate_pps_1522B - Maximum number of packet per second
+| | ... |                             for 10GE with 1522B L2 Frame.
+| | ... | - 10Ge_linerate_pps_1526B - Maximum number of packet per second
+| | ... |                             for 10GE with 1526B L2 Frame.
+| | ... | - 10Ge_linerate_pps_9000B - Maximum number of packet per second
+| | ... |                             for 10GE with 9000B L2 Frame.
+| | ... | - 10Ge_linerate_pps_9004B - Maximum number of packet per second
+| | ... |                             for 10GE with 9004B L2 Frame.
+| | ... | - 10Ge_linerate_pps_9008B - Maximum number of packet per second
+| | ... |                             for 10GE with 9008B L2 Frame.
+| | ... | - 40Ge_linerate_pps_64B - Maximum number of packet per second
+| | ... |                           for 40GE with 64B L2 Frame.
+| | ... | - 40Ge_linerate_pps_68B - Maximum number of packet per second
+| | ... |                           for 40GE with 68B L2 Frame.
+| | ... | - 40Ge_linerate_pps_72B - Maximum number of packet per second
+| | ... |                           for 40GE with 72B L2 Frame.
+| | ... | - 40Ge_linerate_pps_78B - Maximum number of packet per second
+| | ... |                           for 40GE with 78B L2 Frame.
+| | ... | - 40Ge_linerate_pps_1518B - Maximum number of packet per second
+| | ... |                             for 40GE with 1518B L2 Frame.
+| | ... | - 40Ge_linerate_pps_1522B - Maximum number of packet per second
+| | ... |                             for 40GE with 1522B L2 Frame.
+| | ... | - 40Ge_linerate_pps_1526B - Maximum number of packet per second
+| | ... |                             for 40GE with 1526B L2 Frame.
+| | ... | - 40Ge_linerate_pps_9000B - Maximum number of packet per second
+| | ... |                             for 40GE with 9000B L2 Frame.
+| | ... | - 40Ge_linerate_pps_9004B - Maximum number of packet per second
+| | ... |                             for 40GE with 9004B L2 Frame.
+| | ... | - 40Ge_linerate_pps_9008B - Maximum number of packet per second
+| | ... |                             for 40GE with 9008B L2 Frame.
+| | ...
+| | Set Suite Variable | ${10Ge_linerate_pps_64B} | 14880952
+| | Set Suite Variable | ${10Ge_linerate_pps_68B} | 14204545
+| | Set Suite Variable | ${10Ge_linerate_pps_72B} | 13586956
+| | Set Suite Variable | ${10Ge_linerate_pps_78B} | 12755102
+| | Set Suite Variable | ${10Ge_linerate_pps_1518B} | 812743
+| | Set Suite Variable | ${10Ge_linerate_pps_1522B} | 810635
+| | Set Suite Variable | ${10Ge_linerate_pps_9000B} | 138580
+| | Set Suite Variable | ${10Ge_linerate_pps_9004B} | 138519
+| | Set Suite Variable | ${10Ge_linerate_pps_9008B} | 138458
+| | Set Suite Variable | ${40Ge_linerate_pps_64B} | 59523809
+| | Set Suite Variable | ${40Ge_linerate_pps_68B} | 56818181
+| | Set Suite Variable | ${40Ge_linerate_pps_72B} | 54347826
+| | Set Suite Variable | ${40Ge_linerate_pps_78B} | 51020408
+| | Set Suite Variable | ${40Ge_linerate_pps_1518B} | 3250975
+| | Set Suite Variable | ${40Ge_linerate_pps_1522B} | 3242542
+| | Set Suite Variable | ${40Ge_linerate_pps_9000B} | 554323
+| | Set Suite Variable | ${40Ge_linerate_pps_9004B} | 554078
+| | Set Suite Variable | ${40Ge_linerate_pps_9008B} | 553832
 
 | Setup performance global Variables
 | | [Documentation] | Setup performance global Variables
@@ -83,10 +98,8 @@
 | | ... | - ${glob_loss_acceptance} - Loss acceptance treshold
 | | ... | - ${glob_loss_acceptance_type} - Loss acceptance treshold type
 | | ...
-| | ${glob_loss_acceptance}= | Set Variable | 0.5
-| | ${glob_loss_acceptance_type}= | Set Variable | percentage
-| | Set Suite Variable | ${glob_loss_acceptance}
-| | Set Suite Variable | ${glob_loss_acceptance_type}
+| | Set Suite Variable | ${glob_loss_acceptance} | 0.5
+| | Set Suite Variable | ${glob_loss_acceptance_type} | percentage
 
 | 2-node circular Topology Variables Setup
 | | Append Nodes | ${nodes['TG']} | ${nodes['DUT1']} | ${nodes['TG']}
@@ -330,8 +343,9 @@
 | | Set Search Frame Size | ${framesize}
 | | Set Search Rate Type pps
 | | Linear Search | ${start_rate} | ${topology_type}
-| | ${rate_per_stream}= | Verify Search Result
+| | ${rate_per_stream} | ${latency}= | Verify Search Result
 | | Display result of NDR search | ${rate_per_stream} | ${framesize} | 2
+| | ...                          | ${latency}
 | | Traffic should pass with no loss | ${duration} | ${rate_per_stream}pps
 | | ...                              | ${framesize} | ${topology_type}
 | | ...                              | fail_on_loss=${False}
@@ -370,9 +384,10 @@
 | | Run Keyword If | '${loss_acceptance_type}' == 'percentage'
 | | ...            | Set Loss Acceptance Type Percentage
 | | Linear Search | ${start_rate} | ${topology_type}
-| | ${rate_per_stream}= | Verify Search Result
+| | ${rate_per_stream} | ${latency}= | Verify Search Result
 | | Display result of PDR search | ${rate_per_stream} | ${framesize} | 2
 | | ...                          | ${loss_acceptance} | ${loss_acceptance_type}
+| | ...                          | ${latency}
 | | Traffic should pass with partial loss | ${duration} | ${rate_per_stream}pps
 | | ...                                   | ${framesize} | ${topology_type}
 | | ...                                   | ${loss_acceptance}
@@ -408,8 +423,9 @@
 | | Set Search Rate Type pps
 | | Set Binary Convergence Threshold | ${threshold}
 | | Binary Search | ${binary_min} | ${binary_max} | ${topology_type}
-| | ${rate_per_stream}= | Verify Search Result
+| | ${rate_per_stream} | ${latency}= | Verify Search Result
 | | Display result of NDR search | ${rate_per_stream} | ${framesize} | 2
+| | ...                          | ${latency}
 | | Traffic should pass with no loss | ${duration} | ${rate_per_stream}pps
 | | ...                              | ${framesize} | ${topology_type}
 | | ...                              | fail_on_loss=${False}
@@ -450,9 +466,10 @@
 | | ...            | Set Loss Acceptance Type Percentage
 | | Set Binary Convergence Threshold | ${threshold}
 | | Binary Search | ${binary_min} | ${binary_max} | ${topology_type}
-| | ${rate_per_stream}= | Verify Search Result
+| | ${rate_per_stream} | ${latency}= | Verify Search Result
 | | Display result of PDR search | ${rate_per_stream} | ${framesize} | 2
 | | ...                          | ${loss_acceptance} | ${loss_acceptance_type}
+| | ...                          | ${latency}
 | | Traffic should pass with partial loss | ${duration} | ${rate_per_stream}pps
 | | ...                                   | ${framesize} | ${topology_type}
 | | ...                                   | ${loss_acceptance}
@@ -489,8 +506,9 @@
 | | Set Search Rate Type pps
 | | Set Binary Convergence Threshold | ${threshold}
 | | Combined Search | ${start_rate} | ${topology_type}
-| | ${rate_per_stream}= | Verify Search Result
+| | ${rate_per_stream} | ${latency}= | Verify Search Result
 | | Display result of NDR search | ${rate_per_stream} | ${framesize} | 2
+| | ...                          | ${latency}
 | | Traffic should pass with no loss | ${duration} | ${rate_per_stream}pps
 | | ...                              | ${framesize} | ${topology_type}
 | | ...                              | fail_on_loss=${False}
@@ -533,9 +551,10 @@
 | | ...            | Set Loss Acceptance Type Percentage
 | | Set Binary Convergence Threshold | ${threshold}
 | | Combined Search | ${start_rate} | ${topology_type}
-| | ${rate_per_stream}= | Verify Search Result
+| | ${rate_per_stream} | ${latency}= | Verify Search Result
 | | Display result of PDR search | ${rate_per_stream} | ${framesize} | 2
 | | ...                          | ${loss_acceptance} | ${loss_acceptance_type}
+| | ...                          | ${latency}
 | | Traffic should pass with partial loss | ${duration} | ${rate_per_stream}pps
 | | ...                                   | ${framesize} | ${topology_type}
 | | ...                                   | ${loss_acceptance}
@@ -550,19 +569,27 @@
 | | ... | - ${rate_per_stream} - Measured rate per stream [pps]. Type: string
 | | ... | - ${framesize} - L2 Frame Size [B]. Type: integer
 | | ... | - ${nr_streams} - Total number of streams. Type: integer
+| | ... | - ${latency} - Latency stats. Type: dictionary
 | | ...
 | | ... | *Return:*
 | | ... | - No value returned
 | | ...
 | | ... | *Example:*
 | | ...
-| | ... | \| Display result of NDR search \| 4400000 \| 64 \| 2
+| | ... | \| Display result of NDR search \| 4400000 \| 64 \| 2 \
+| | ... | \| (0, 10/10/10) \|
 | | [Arguments] | ${rate_per_stream} | ${framesize} | ${nr_streams}
+| | ...         | ${latency}
 | | ${rate_total}= | Evaluate | ${rate_per_stream}*${nr_streams}
 | | ${bandwidth_total}= | Evaluate | ${rate_total}*(${framesize}+20)*8/(10**9)
 | | Set Test Message | FINAL_RATE: ${rate_total} pps
-| | Set Test Message | (${nr_streams}x ${rate_per_stream} pps) | append=yes
-| | Set Test Message | FINAL_BANDWIDTH: ${bandwidth_total} Gbps | append=yes
+| | Set Test Message | (${nr_streams}x ${rate_per_stream} pps)
+| | ...              | append=yes
+| | Set Test Message | ${\n}FINAL_BANDWIDTH: ${bandwidth_total} Gbps (untagged)
+| | ...              | append=yes
+| | :FOR | ${idx} | ${lat} | IN ENUMERATE | @{latency}
+| | | Set Test Message | ${\n}LATENCY_STREAM_${idx}: ${lat} usec (min/avg/max)
+| | ...                | append=yes
 
 | Display result of PDR search
 | | [Documentation] | Display result of PDR search in packet per seconds (total
@@ -574,6 +601,7 @@
 | | ... | - ${nr_streams} - Total number of streams. Type: integer
 | | ... | - ${loss_acceptance} - Accepted loss during search. Type: float
 | | ... | - ${loss_acceptance_type} - Percentage or frames. Type: string
+| | ... | - ${latency} - Latency stats. Type: dictionary
 | | ...
 | | ... | *Return:*
 | | ... | - No value returned
@@ -581,14 +609,19 @@
 | | ... | *Example:*
 | | ...
 | | ... | \| Display result of PDR search \| 4400000 \| 64 \| 2 \| 0.5 \
-| | ... | \| percentage
+| | ... | \| percentage \| (0, 10/10/10) \|
 | | [Arguments] | ${rate_per_stream} | ${framesize} | ${nr_streams}
-| | ...         | ${loss_acceptance} | ${loss_acceptance_type}
+| | ...         | ${loss_acceptance} | ${loss_acceptance_type} | ${latency}
 | | ${rate_total}= | Evaluate | ${rate_per_stream}*${nr_streams}
 | | ${bandwidth_total}= | Evaluate | ${rate_total}*(${framesize}+20)*8/(10**9)
 | | Set Test Message | FINAL_RATE: ${rate_total} pps
-| | Set Test Message | (${nr_streams}x ${rate_per_stream} pps) | append=yes
-| | Set Test Message | FINAL_BANDWIDTH: ${bandwidth_total} Gbps | append=yes
+| | Set Test Message | (${nr_streams}x ${rate_per_stream} pps)
+| | ...              | append=yes
+| | Set Test Message | ${\n}FINAL_BANDWIDTH: ${bandwidth_total} Gbps (untagged)
+| | ...              | append=yes
+| | :FOR | ${idx} | ${lat} | IN ENUMERATE | @{latency}
+| | | Set Test Message | ${\n}LATENCY_STREAM_${idx}: ${lat} usec (min/avg/max)
+| | ...                | append=yes
 | | Set Test Message | ${\n}LOSS_ACCEPTANCE: ${loss_acceptance} ${loss_acceptance_type}
 | | ...              | append=yes
 
@@ -653,7 +686,8 @@
 | Clear and show runtime counters with running traffic
 | | [Arguments] | ${duration} | ${rate} | ${framesize} | ${topology_type}
 | | Send traffic on tg | -1 | ${rate} | ${framesize}
-| | ...                | ${topology_type} | warmup_time=0 | async_call=True
+| | ...                | ${topology_type} | warmup_time=0 | async_call=${True}
+| | ...                | latency=${False}
 | | Clear runtime counters on all DUTs
 | | Sleep | ${duration}
 | | Show runtime counters on all DUTs
Integration tests