This kind of DUT state is dynamically created based on incoming traffic,
it affects how DUT handles the traffic, and (unlike telemetry counters)
it has uneven impact on CPU load.
-Typical example is NAT where opening sessions takes more CPU than
-forwarding packet on existing sessions.
+Typical example is NAT, where detecting new sessions takes more CPU than
+forwarding packet on existing (open or recently closed) sessions.
We call DUT configurations with this kind of state "stateful",
and configurations without them "stateless".
(Even though stateless configurations contain state described in previous
Tests aiming at sustain performance need to make sure DUT state is created.
We achieve this via a ramp-up trial, specific purpose of which
-is to create the state. Subsequent trials need no specific handling,
-as state remains the same.
+is to create the state.
-For the state to be set completely, it is important DUT (nor TG) loses
-no packets, we achieve this by setting the profile multiplier (TPS from now on)
-to low enough value.
+Subsequent trials need no specific handling, as long as the state
+remains the same. But some state can time-out, so additional ramp-up
+trials are inserted whenever the code detects the state can time-out.
+Note that a trial with zero loss refreshes the state,
+so only the time since the last non-zero loss trial is tracked.
+
+For the state to be set completely, it is important both DUT and TG
+do not lose any packets. We achieve this by setting the profile multiplier
+(TPS from now on) to low enough value.
It is also important each state-affecting packet is sent.
For size-limited traffic profile it is guaranteed by the size limit.
For continuous traffic, we set a long enough duration (based on TPS).
-At the end of the ramp-up trial, we check telemetry to confirm
-the state has been created as expected.
-Test fails if the state is not complete.
+At the end of the ramp-up trial, we check DUT state to confirm
+it has been created as expected.
+Test fails if the state is not (completely) created.
State Reset
___________
If it is set, each search algorithm (including MRR) will invoke it
before all trial measurements (both main and telemetry ones).
Any configuration keyword enabling a feature with DUT state
-will check whether a test variable for ramp-up (duration) is present.
+will check whether a test variable for ramp-up rate is present.
If it is present, resetter is not set.
If it is not present, the keyword sets the apropriate resetter value.
This logic makes sure either ramp-up or state reset are used.
The second is protocol such as TCP ramping up their throughput to utilize
the bandwidth available. This is the original meaning of "ramp up"
in the NGFW draft (see above).
-In existing tests we are not distinguishing such phases,
-trial measurment reports the telemetry from the whole trial
-(e.g. throughput is time averaged value).
+In existing tests we are not using this meaning of TCP ramp-up.
+Instead we use only small transactions, and large enough initial window
+so TCP acts as ramped-up already.
-The third is TCP increasing throughput due to retransmissions triggered by
-packet loss. In CSIT we currently try to avoid this behavior
+The third is TCP increasing offered load due to retransmissions triggered by
+packet loss. In CSIT we again try to avoid this behavior
by using small enough data to transfer, so overlap of multiple transactions
(primary cause of packet loss) is unlikely.
-But in MRR tests packet loss is still probable.
-Once again, we rely on using telemetry from the whole trial,
-resulting in time averaged throughput values.
+But in MRR tests, packet loss and non-constant offered load are still expected.
Stateless DUT configuratons
~~~~~~~~~~~~~~~~~~~~~~~~~~~
-These are simply configurations, which do not set any resetter value
+These are simple configurations, which do not set any resetter value
(even if ramp-up duration is not configured).
Majority of existing tests are of this type, using continuous traffic profiles.
In order to identify limits of Trex performance,
we have added suites with stateless DUT configuration (VPP ip4base)
subjected to size-limited ASTF traffic.
-The discovered throughputs serve as a basis of comparison
+The discovered rates serve as a basis of comparison
for evaluating the results for stateful DUT configurations (VPP NAT44ed)
subjected to the same traffic profiles.
comparison.
- Some stateful DUT configurations (NAT44DET, NAT44ED unidirectional)
- are tested using stateless traffic profiles.
+ are tested using stateless traffic profiles and continuous traffic.
- The rest of stateful DUT configurations (NAT44ED bidirectional)
- are tested using stateful traffic profiles.
+ are tested using stateful traffic profiles and size limited traffic.
Code Review / csit.git / blobdiff