Methodology: Edit nat44 test spec doc

- Missing profile specifics and transaction counters.

Change-Id: I6f7378e5fe9d639599c38545b0503354a8a65198
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

diff --git a/docs/report/introduction/methodology_nat44.rst b/docs/report/introduction/methodology_nat44.rst
index 5dc558f..198837d 100644 (file)
--- a/docs/report/introduction/methodology_nat44.rst
+++ b/docs/report/introduction/methodology_nat44.rst
@@ -12,14 +12,17 @@ public range.
 Following quantities are used to describe inside to outside IP address
 and port bindings scenarios:
 
 Following quantities are used to describe inside to outside IP address
 and port bindings scenarios:
 

-- inside-addresses, ports-per-inside-address, number of inside source
-  addresses (representing inside hosts) and number of TCP/UDP source
+- Inside-addresses, number of inside source addresses
+  (representing inside hosts).
+- Ports-per-inside-address, number of TCP/UDP source

   ports per inside source address.
   ports per inside source address.

-- outside-addresses, number of outside (public) source addresses
-  allocated to NAT44. The maximal number of ports-per-outside-address
-  usable for NAT is 64 512 (in non-reserved port range 1024-65535,
-  :rfc:`4787`).
-- sharing-ratio, equal to inside-addresses / outside-addresses.
+- Outside-addresses, number of outside (public) source addresses
+  allocated to NAT44.
+- Ports-per-outside-address, number of TCP/UDP source
+  ports per outside source address. The maximal number of
+  ports-per-outside-address usable for NAT is 64 512
+  (in non-reserved port range 1024-65535, :rfc:`4787`).
+- Sharing-ratio, equal to inside-addresses / outside-addresses.

 
 CSIT NAT44 tests are designed to take into account the maximum number of
 ports (sessions) required per inside host (inside-address) and at the
 
 CSIT NAT44 tests are designed to take into account the maximum number of
 ports (sessions) required per inside host (inside-address) and at the


@@ -43,15 +46,7 @@ Initial CSIT NAT44 tests, including associated TG/TRex traffic profiles,
 are based on ports-per-inside-address set to 63 and the sharing ratio of
 1024. This approach is currently used for all NAT44 tests including
 NAT44det (NAT44 deterministic used for Carrier Grade NAT applications)
 are based on ports-per-inside-address set to 63 and the sharing ratio of
 1024. This approach is currently used for all NAT44 tests including
 NAT44det (NAT44 deterministic used for Carrier Grade NAT applications)

-and NAT44ed.
-
-..
-    .. TODO::
-
-    Note that in the latter case, due to overloading of (ouside-address,
-    outside-port) tuple for different endpoint destinations the actual
-    sharing ratio is likely to different, as it will depend on the
-    destination addresses used by NAT'ed flows.
+and NAT44ed (Endpoint Dependent).

 
 Private address ranges to be used in tests:
 
 
 Private address ranges to be used in tests:
 


@@ -70,13 +65,14 @@ NAT44 Session Scale
 
 NAT44 session scale tested is govern by the following logic:
 
 
 NAT44 session scale tested is govern by the following logic:
 

-- Number of inside addresses/hosts H[i] = (H[i-1] x 2^2) with H(0)=1 024, i = 1,2,3, ...
+- Number of inside-addresses(hosts) H[i] = (H[i-1] x 2^2) with H(0)=1 024,
+  i = 1,2,3, ...

 
 

-  - H[i] = 1 024, 4 096, 16 384, 65 536, 262 144, 1 048 576, ...
+  - H[i] = 1 024, 4 096, 16 384, 65 536, 262 144, ...

 
 

-- Number of sessions S[i](ports-per-host) = H[i] * ports-per-inside-address
+- Number of sessions S[i] = H[i] * ports-per-inside-address

 
 

-  - ports-per-host = 63
+  - ports-per-inside-address = 63

 
 +---+---------+------------+
 | i |   hosts |   sessions |
 
 +---+---------+------------+
 | i |   hosts |   sessions |


@@ -101,6 +97,11 @@ ip4, ip6 and l2, sending UDP packets in both directions
 inside-to-outside and outside-to-inside. See
 :ref:`data_plane_throughput` for more detail.
 
 inside-to-outside and outside-to-inside. See
 :ref:`data_plane_throughput` for more detail.
 

+The inside-to-outside traffic uses single destination address (20.0.0.0)
+and port (1024).
+The inside-to-outside traffic covers whole inside address and port range,
+the outside-to-inside traffic covers whole outside address and port range.
+

 NAT44det translation entries are created during the ramp-up phase
 preceding the throughput test, followed by verification that all entries
 are present, before proceeding to the throughput test. This ensures
 NAT44det translation entries are created during the ramp-up phase
 preceding the throughput test, followed by verification that all entries
 are present, before proceeding to the throughput test. This ensures


@@ -117,30 +118,55 @@ NAT44det scenario tested:
     16515072.
   - [mrr|ndrpdr|soak], MRR, NDRPDR or SOAK test.
 
     16515072.
   - [mrr|ndrpdr|soak], MRR, NDRPDR or SOAK test.
 

+..
+    TODO: The -s{S} part is redundant,
+    we can save space by removing it.
+    TODO: Make traffic profile names resemble suite names more closely.
+

 NAT44 Endpoint-Dependent
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
 NAT44 Endpoint-Dependent
 ^^^^^^^^^^^^^^^^^^^^^^^^
 

-NAT44ed is benchmarked using following methodologies:
-
-- Uni-directional throughput using *stateless* traffic profile.
+..
+    TODO: Is it possible to test a NAT44ed scenario where the outside source
+    address and port is limited to just one value?
+    In theory, as long as every inside source address&port traffic
+    uses a different destination address&port, there will be no conflicts,
+    and we could use bidirectional stateless profiles.
+    Possibly, VPP requires some amount of outside source address&port
+    to remain unused for security reasons. But we can try to see what happens.
+
+In order to excercise NAT44ed ability to translate based on both
+source and destination address and port, the inside-to-outside traffic
+varies also destination address and port. Destination port is the same
+as source port, destination address has the same offset as the source address,
+but applied to different subnet (starting with 20.0.0.0).
+
+As the mapping is not deterministic (for security reasons),
+we cannot easily use stateless bidirectional traffic profiles.
+Outside address and port range is fully covered,
+but we do not know which outside-to-inside source address and port to use
+to hit an open session of a particular outside address and port.
+
+Therefore, NAT44ed is benchmarked using following methodologies:
+
+- Unidirectional throughput using *stateless* traffic profile.

 - Connections-per-second using *stateful* traffic profile.
 - Connections-per-second using *stateful* traffic profile.

-- Bi-directional throughput using *stateful* traffic profile.
+- Bidirectional PPS (see below) using *stateful* traffic profile.

 
 

-Uni-directional NAT44ed throughput tests are using TRex STL (Stateless)
+Unidirectional NAT44ed throughput tests are using TRex STL (Stateless)

 APIs and traffic profiles, but with packets sent only in
 APIs and traffic profiles, but with packets sent only in

-inside-to-outside direction. Due to indeterministic bindings of outside
-to inside (src_addr,src_port) that are created dynamically at flow start
-bidirectional testing is not possible with stateless traffic profiles.
-See :ref:`data_plane_throughput` for more detail.
-
-Similarly to NAT44det, NAT44ed uni-directional throughput tests include
+inside-to-outside direction.
+Similarly to NAT44det, NAT44ed unidirectional throughput tests include

 a ramp-up phase to establish and verify the presence of required NAT44ed
 a ramp-up phase to establish and verify the presence of required NAT44ed

-binding entries. NAT44ed CPS (connections-per-second) and throughput /
-PPS stateful tests do not have a ramp-up phase.
+binding entries.

 
 Stateful NAT44ed tests are using TRex ASTF (Advanced Stateful) APIs and
 traffic profiles, with packets sent in both directions. Tests are run
 with both UDP and TCP/IP sessions.
 
 Stateful NAT44ed tests are using TRex ASTF (Advanced Stateful) APIs and
 traffic profiles, with packets sent in both directions. Tests are run
 with both UDP and TCP/IP sessions.

+As both NAT44ed CPS (connections-per-second) and PPS (packets-per-second)
+stateful tests measure (also) session opening performance,
+they use state reset instead of ramp-up trial.
+That is also the reason why PPS tests are not called throughput tests.

 
 Associated CSIT test cases use the following naming scheme to indicate
 NAT44DET case tested:
 
 Associated CSIT test cases use the following naming scheme to indicate
 NAT44DET case tested:


@@ -164,3 +190,41 @@ NAT44DET case tested:
   - [cps|pps], connections-per-second session establishment rate or
     packets-per-second throughput rate.
   - [mrr|ndrpdr], bidirectional stateful tests MRR, NDRPDR.
   - [cps|pps], connections-per-second session establishment rate or
     packets-per-second throughput rate.
   - [mrr|ndrpdr], bidirectional stateful tests MRR, NDRPDR.

+
+Stateful traffic profiles
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+WiP.
+
+UDP CPS
+~~~~~~~
+
+TCP CPS
+~~~~~~~
+
+UDP PPS
+~~~~~~~
+
+TCP PPS
+~~~~~~~
+
+Ip4base tests
+^^^^^^^^^^^^^
+
+Contrary to stateless traffic profiles, we do not have a simple limit
+that would guarantee TRex is able to send traffic at specified load.
+For that reason, we have added tests where "nat44ed" is replaced by "ip4base".
+Instead of NAT44ed processing, the tests set minimalistic IPv4 routes,
+so that packets are forwarded in both inside-to-outside and outside-to-inside
+directions.
+
+The packets arrive to server end of TRex with different source address&port
+than in NAT44ed tests (no translation to outside values is done with ip4base),
+but those are not specified in the stateful traffic profiles.
+The server end uses the received address&port as destination
+for outside-to-inside traffic. Therefore the same stateful traffic profile
+works for both NAT44ed and ip4base test (of the same scale).
+
+The NAT44ed results are displayed together with corresponding ip4base results.
+If they are similar, TRex is probably the bottleneck.
+If NAT44ed result is visibly smaller, it describes the real VPP performance.