3 <!-- MarkdownTOC autolink="true" -->
5 - [Tests for NAT44ED](#tests-for-nat44ed)
6 - [CPS Test Objectives](#cps-test-objectives)
7 - [Input Parameters](#input-parameters)
8 - [Stateful traffic profiles](#stateful-traffic-profiles)
9 - [UDP CPS Tests](#udp-cps-tests)
10 - [UDP TRex Measurements](#udp-trex-measurements)
11 - [Counters](#counters)
12 - [Calculations](#calculations)
16 - [UDP VPP Telemetry](#udp-vpp-telemetry)
17 - [Counters](#counters-1)
19 - [TCP/IP CPS Tests](#tcpip-cps-tests)
20 - [TCP/IP TRex Measurements](#tcpip-trex-measurements)
21 - [Counters](#counters-2)
22 - [Calculations](#calculations-1)
23 - [CPS Trial PASS](#cps-trial-pass)
24 - [CPS-MRR](#cps-mrr-1)
25 - [CPS-PDR](#cps-pdr-1)
26 - [CPS-NDR](#cps-ndr-1)
27 - [TCP/IP VPP Telemetry](#tcpip-vpp-telemetry)
28 - [Counters](#counters-3)
35 Two types of stateful tests are developed for NAT44ED (source network address
36 and port translation IPv4 to IPv4 with 5-tuple session state):
38 - Connections-Per-Second (CPS), discovering the maximum rate of creating
39 NAT44ED sessions. Measured separately for UDP and TCP connections and
40 for different session scale.
42 - Packets-Per-Second (PPS), discovering the maximum rate of
43 simultaneously creating NAT44ED sessions and transfering bulk of data
44 packets across the corresponding connections. Measured separately for
45 UDP and TCP connections with different session scale and different data
46 packet sizes per each connection. Current code is using 64B only for UDP
47 and default MSS 1460B for TCP/IP.
49 This note describes CPS tests.
51 ## CPS Test Objectives
53 Discover DUT's highest sustain rate of creating fully functional NAT44ED
54 5-tuple stateful session entries. Session entry is considered fully
55 functional, if packets associated with this entry are NAT44ED processed
56 by DUT and forwarded in both directions without loss.
58 Similarly to packet throughput tests, three CPS rates are discovered:
60 - CPS-MRR, verified connection rate at maximal connection attempt rate,
61 regardless of an amount of not established connections. (Connections
62 per Second - Maximum Receive Rate.)
63 - CPS-NDR, maximal connection attempt rate at which all connections get
64 established. (Connections per Second - Non Drop Rate.)
65 - CPS-PDR, maximal connection attempt rate at which ratio of not
66 established connections to attempted connections is below configured
67 threshold. (Connections per Second - Partial Drop Rate.)
71 - `max_cps_rate`, maximum rate of attempting connections, to be used by
72 traffic generator, limited by traffic generator capabilities, Ethernet
73 link(s) rate and NIC model.
74 - `min_cps_rate`, minimum rate of establishing connections to be used for
75 measurements. Search fails if lower transmit rate needs to be used to
77 - `target_session_number`, maximum number of sessions to be established and
79 - `target_loss_ratio`, maximum acceptable connections loss ratio search
80 criteria for PDR measurements with UDP tests. Indicates packet drop
81 impact on connection establishment rate.
82 - `final_relative_width`, required measurement resolution expressed as
83 (lower_bound, upper_bound) interval width relative to upper_bound.
84 - stateful traffic profiles, TRex ASTF program defining the connection
85 per L4 protocol tested (TCP, UDP), including connect and
88 ## Stateful traffic profiles
90 TRex ASTF program defines following TCP and UDP transactions for
91 discovering NAT44ED CPS limits:
94 - connect(syn,syn-ack,ack)
95 - pkts client tx 2, rx 1
96 - pkts server tx 1, rx 2
97 - delay (note: optional, currently not implemented)
99 - close(fin,fin-ack,ack,ack)
100 - pkts client tx 2, rx 2
101 - pkts server tx 1, rx 2
103 - connect_and_close(req,ack)
104 - pkts client tx 1, rx 1
105 - pkts server tx 1, rx 1
107 TRex ASTF program configuration parameters:
109 - `limit` of connections, set to `target_session_number`.
110 - `multiplier`, represents `trial_cps_rate`, a number of connections per
111 second to be executed per trial. Multiplier applies to connect phases.
112 Close phases occur automatically based on arrival of the last packet
113 expected per session.
114 - IPv4 source and destination address and port ranges matching the
115 limit of connections.
116 - Source and destination addresses changing packet-by-packet with two
117 separate profiles i) incrementing sequentially pair-wise
118 (implemented) and ii) changed randomly (with seed) pair-wise (not
120 - Source port changing randomly within the range.
121 - `trial_duration`, function of `target_session_number` and `multiplier`
122 - `multiplier`, subject of the search, value in the range (`min_cps_rate`,`max_cps_rate`)
123 - `target_setup_duration` = `target_session_number` / `trial_cps_rate`
125 - `trial_duration` = `target_setup_duration` + `late_traffic_start_correction`
126 - `late_traffic_start_correction` = 0.1115 seconds (hardcoded for now)
128 - `trial_duration` = 2 * `target_setup_duration` + `late_traffic_start_correction`
129 - `late_traffic_start_correction` = 0.1115 seconds (hardcoded for now)
133 ### UDP TRex Measurements
137 Following TRex ASTF counters are collected by UDP CPS tests for automated
138 results evaluation (r) and debugging purposes (d):
141 - (r) `opackets`, TRex UDP transaction start
142 - (r) `ipackets`, TRex UDP transaction finish
147 - (d) `m_active_flows`
149 - (d) `m_traffic_duration`, includes TRex ramp-up overhead, and it can
150 be quite far from the actual traffic duration
151 - (d) `udps_connects`
157 - (d) `udps_keepdrops`, TRex out of capacity, dropping UDP KAs(?)
159 Vratko Polak: Yes, although the traffic profile should have set large
160 enough keepalive value so zero KA packets are actually sent within the
161 trial. I did not actually check the value is large enough for the worst
162 case (ndrpdr search hitting min multiplier of 9001).
164 - (d) `err_rx_throttled`, TRex out of capacity, throttling workers due
167 Vratko Polak: I think this is TRex receiving the packet on L2 level, but
168 then dropping it because L7 buffers are full. Such packet increases
169 ipackets, but does not increase any L7 counter (even if traffic profile
170 wants to receive that packet). But this is just me guessing. TRex docs
171 say "rx thread was throttled due too many packets in NIC rx queue", and
172 I did no experiments/investigation to confirm my hypothesis fits with
173 the observed counters.
175 - (d) `err_c_nf_throttled`, Number of client side flows that were not
176 opened due to flow-table overflow(?)
177 - (d) `err_flow_overflow`, too many flows(?)
179 - (d) `m_active_flows`
181 - (r) `m_traffic_duration`
188 - (d) `err_rx_throttled`, TRex out of capacity, throttling workers due
191 [TRex ASTF counters reference](https://trex-tgn.cisco.com/trex/doc/trex_astf.html#_counters_reference).
193 TRex counters are polled once TRex confirms traffic is stopped, after it
194 is explicitly instructed to stop it. Early attempts to use periodic TRex
195 counter polling affected TRex behaviour and test results, hence counter
196 polling is consider as invasive.
200 - Interface packet loss
201 - pktloss_ratio = (c_opackets - c_ipackets) / c_opackets
202 - UDP session packet loss (currently not used)
203 - UDP session byte loss (currently not used)
204 - UDP session integrity (currently not used)
208 Reported MRR values are calculated as follows:
210 CPS-MRR = `c_ipackets` / `s_traffic_duration`, where
211 `s_traffic_duration` = TRex Traffic Server `m_traffic_duration`.
213 In order to ensure a determnistic region of TRex ASTF operation, a
214 separate set of tests is run for each traffic profile, with vpp-ip4base
215 DUT instead of vpp-nat44ed, to auto-discover the maximum rate TRex ASTF
216 traffic profile is capable of. Result of this test is used as a side
217 reference to compare with the results of NAT44ed CPS-MRR tests.
221 CPS-PDR values are discovered using MLRsearch, a binary search optimized
222 for the overall test duration.
224 CPS-PDR = max(`trial_cps_rate`) found for `pktloss_ratio` <
225 `target_loss_ratio`, according to MLRsearch criteria for PDR.
227 Measurements to be reported in the CPS-PDR result test message:
233 CPS-NDR values are also discovered using MLRsearch.
235 CPS-NDR = max(`trial_cps_rate`) found for `pktloss_ratio` = 0, according
236 to MLRsearch criteria for PDR.
238 Measurements to be reported in the CPS-NDR result test message:
242 ### UDP VPP Telemetry
246 - VPP show nat44 summary
249 max translations per thread: 81920
250 max translations per user: 81920
251 total timed out sessions: 0
252 total sessions: 64514
253 total tcp sessions: 0
254 total tcp established sessions: 0
255 total tcp transitory sessions: 0
256 total tcp transitory (WAIT-CLOSED) sessions: 0
257 total tcp transitory (CLOSED) sessions: 0
258 total udp sessions: 64514
259 total icmp sessions: 0
265 show hardware verbose (10.30.51.54 - /run/vpp/api.sock):
266 Name Idx Link Hardware
267 avf-0/3b/2/0 1 up avf-0/3b/2/0
269 Ethernet address 3c:fe:bd:f9:00:00
270 flags: initialized admin-up vaddr-dma link-up rx-interrupts
271 offload features: l2 vlan rx-polling rss-pf
272 num-queue-pairs 3 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64
280 avf-0/3b/a/0 2 up avf-0/3b/a/0
282 Ethernet address 3c:fe:bd:f9:01:00
283 flags: initialized admin-up vaddr-dma link-up rx-interrupts
284 offload features: l2 vlan rx-polling rss-pf
285 num-queue-pairs 3 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64
298 Thread 1 vpp_wk_0 (lcore 2)
299 Time 21.5, 10 sec internal node vector rate 0.00 loops/sec 6740197.88
300 vector rates in 4.2183e3, out 3.7118e3, drop 0.0000e0, punt 0.0000e0
301 Name State Calls Vectors Suspends Clocks Vectors/Call
302 avf-0/3b/2/0-output active 277 34387 0 1.96e1 124.14
303 avf-0/3b/2/0-tx active 277 34387 0 3.54e1 124.14
304 avf-0/3b/a/0-output active 380 45245 0 1.92e1 119.07
305 avf-0/3b/a/0-tx active 380 45245 0 3.36e1 119.07
306 avf-input polling 144384995 90499 0 3.03e5 0.00
307 ethernet-input active 381 90499 0 1.91e1 237.53
308 ip4-input-no-checksum active 381 90499 0 4.94e1 237.53
309 ip4-lookup active 521 79632 0 3.76e1 152.84
310 ip4-rewrite active 521 79632 0 4.19e1 152.84
311 ip4-sv-reassembly-feature active 381 90499 0 3.78e1 237.53
312 nat44-ed-in2out active 380 45245 0 1.98e2 119.07
313 nat44-ed-in2out-slowpath active 380 45245 0 2.31e3 119.07
314 nat44-ed-out2in active 277 34387 0 1.89e2 124.14
315 nat44-in2out-worker-handoff active 381 90499 0 9.42e1 237.53
316 unix-epoll-input polling 140863 0 0 1.61e3 0.00
318 Thread 2 vpp_wk_1 (lcore 58)
319 Time 21.5, 10 sec internal node vector rate 0.00 loops/sec 6733488.17
320 vector rates in 3.3365e3, out 3.5604e3, drop 0.0000e0, punt 0.0000e0
321 Name State Calls Vectors Suspends Clocks Vectors/Call
322 avf-0/3b/2/0-output active 276 31129 0 2.03e1 112.79
323 avf-0/3b/2/0-tx active 276 31129 0 3.63e1 112.79
324 avf-0/3b/a/0-output active 332 45254 0 1.87e1 136.31
325 avf-0/3b/a/0-tx active 332 45254 0 3.48e1 136.31
326 avf-input polling 166439403 71581 0 4.42e5 0.00
327 ethernet-input active 277 65516 0 1.89e1 236.52
328 ip4-input-no-checksum active 277 65516 0 4.95e1 236.52
329 ip4-lookup active 455 76383 0 3.75e1 167.87
330 ip4-rewrite active 455 76383 0 4.20e1 167.87
331 ip4-sv-reassembly-feature active 277 65516 0 3.85e1 236.52
332 nat44-ed-in2out active 377 45254 0 1.97e2 120.04
333 nat44-ed-in2out-slowpath active 332 45254 0 2.39e3 136.31
334 nat44-ed-out2in active 276 31129 0 1.83e2 112.79
335 nat44-out2in-worker-handoff active 277 65516 0 2.17e2 236.52
336 unix-epoll-input polling 140817 0 0 1.60e3 0.00
345 32258 nat44-in2out-worker-handoff same worker
346 32256 nat44-in2out-worker-handoff do handoff
347 32258 nat44-ed-out2in good out2in packets processed
348 32258 nat44-ed-out2in UDP packets
349 32258 nat44-ed-in2out-slowpath good in2out packets processed
350 32258 nat44-ed-in2out-slowpath UDP packets
351 32256 nat44-out2in-worker-handoff same worker
352 32258 nat44-out2in-worker-handoff do handoff
353 32256 nat44-ed-out2in good out2in packets processed
354 32256 nat44-ed-out2in UDP packets
355 32256 nat44-ed-in2out-slowpath good in2out packets processed
356 32256 nat44-ed-in2out-slowpath UDP packets
361 ### TCP/IP TRex Measurements
365 Following TRex ASTF counters are collected by UDP CPS tests for automated
366 results evaluation (r) and debugging purposes (d):
375 - (d) `m_active_flows`
377 - (d) `m_traffic_duration`
378 - (r) `tcps_connattempt`
379 - (d) `tcps_connects`
382 - (d) `m_active_flows`
384 - (r) `m_traffic_duration`
386 - (r) `tcps_connects`
388 - (d) `err_no_template`, server can’t match L7 template no destination port or IP range
390 [TRex ASTF counters reference](https://trex-tgn.cisco.com/trex/doc/trex_astf.html#_counters_reference).
392 TRex counters are polled only once by CSIT after traffic is stopped.
396 TODO WIP Note: Currently s_tcp_connects is used for counting successful
397 sessions. But now I am not sure whether it is correct, as already
398 c_tcps_connects counts NAT sessions that got established (even though
399 TCP is not fully connected yet). Not sure how the counters behave when
400 the third packet is lost and retransmitted.
402 - Interface packet loss
403 - `pktloss_c_s` = `c_opackets` - `s_ipackets`
404 - `pktloss_s_c` = `s_opackets` - `c_ipackets`
405 - `pktloss_ratio` = (`pktloss_s_c` + `pktloss_c_s`) / (`c_opackets` + `s_opackets`)
406 - TCP session integrity
407 - `tcp_attempted_connection_count` = `c_tcps_connattempt`
408 - `tcp_failed_connection_count` = `c_tcps_connects` - `c_tcps_connattempt`
412 TODO WIP Note: Currently any trial measurement fails only if TRex itself
413 fails, or if we fail to parse some counter. No criteria mentioned here
414 is currently planned to be implemented; we rely on bad things leading to
415 too few (maybe zero) passed transactions.
418 PASS of TCP CPS test trial is conditioned on all of the following criteria being met:
420 - PASS-C1 TRex must attempt all configured `target_session_number` in `target_setup_duration` time
421 - IOW TRex must send connect packets at configured `trial_cps_rate`.
422 - PASS-C2 Following TRex errors ARE NOT recorded in Target-Counters:
424 - No errors recorded so far
426 - `err_no_template`, server can’t match L7 template no destination port or IP range
431 Reported MRR values are equal to the following TRex counters from Target-Counters:
435 TODO Add description of separate set of tests for discovering a **safe**
436 CPS-MTR value (Maximum Transmit Rate) for TRex, where TRex errors **are not**
437 observed in Target-Counters.
441 CPS-PDR values are discovered using MLRsearch, a binary search optimized
442 for the overall test duration.
444 CPS-PDR = `trial_cps_rate`, if all of the following conditions are met:
446 - `tcp_failed_connection_count` < `target_loss_ratio`
447 - `pktloss_ratio` < `target_loss_ratio`
449 Measurements to be reported in the CPS-PDR result test message:
457 CPS-NDR values are discovered using MLRsearch, a binary search optimized
458 for the overall test duration.
460 CPS-NDR = `trial_cps_rate`, if all of the following conditions are met:
462 - `tcp_failed_connection_count` = 0
463 - `pktloss_ratio` = 0
465 Measurements to be reported in the CPS-PDR result test message:
471 ### TCP/IP VPP Telemetry
475 - VPP show nat44 summary
478 <TODO add sample output>
484 <TODO add sample output>
490 <TODO add sample output>
498 <TODO add sample output>