csit rls1704 report - updated csit_release_notes.rst and overview.rst files.

Change-Id: I0b5005a4c8dc566e559638d981fb0e8a7b079499
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>

diff --git a/docs/report/test_configuration/vpp_functional_configuration/index.rst b/docs/report/test_configuration/vpp_functional_configuration/index.rst
index c13f2d4..89ccbef 100644 (file)
--- a/docs/report/test_configuration/vpp_functional_configuration/index.rst
+++ b/docs/report/test_configuration/vpp_functional_configuration/index.rst
@@ -1,5 +1,5 @@
-VPP Functional Configuration
-============================
+VPP Functional Test Configs
+===========================

 
 .. note::
 
 
 .. note::
 

diff --git a/docs/report/test_configuration/vpp_performance_configuration/index.rst b/docs/report/test_configuration/vpp_performance_configuration/index.rst
index 1d1b03f..c5009cf 100644 (file)
--- a/docs/report/test_configuration/vpp_performance_configuration/index.rst
+++ b/docs/report/test_configuration/vpp_performance_configuration/index.rst
@@ -1,5 +1,5 @@
-VPP Performance Configuration
-=============================
+VPP Performance Test Configs
+============================

 
 .. note::
 
 
 .. note::
 

diff --git a/docs/report/testpmd_performance_tests/csit_release_notes.rst b/docs/report/testpmd_performance_tests/csit_release_notes.rst
index ba78a1b..7f87d24 100644 (file)
--- a/docs/report/testpmd_performance_tests/csit_release_notes.rst
+++ b/docs/report/testpmd_performance_tests/csit_release_notes.rst
@@ -8,3 +8,18 @@ Changes in CSIT |release|
 
     - new NICs - 2p40GE Intel xl710, 2p10GE Intel x710
 
 
     - new NICs - 2p40GE Intel xl710, 2p10GE Intel x710
 

+Known Issues
+------------
+
+Here is the list of known issues in CSIT |release| for Testpmd performance tests:
+
++---+---------------------------------------------------+------------+-----------------------------------------------------------------+
+| # | Issue                                             | Jira ID    | Description                                                     |
++---+---------------------------------------------------+------------+-----------------------------------------------------------------+
+| 1 | Testpmd in 1t1c and 2t2c setups - large variation | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |
+|   | of discovered NDR throughput values across        |            | throughput. Applies to XL710 and X710 NICs, no issues observed  |
+|   | multiple test runs with xl710 and x710 NICs.      |            | on x520 NICs.                                                   |
++---+---------------------------------------------------+------------+-----------------------------------------------------------------+
+| 2 | Lower than expected NDR throughput with xl710     | CSIT-?     | Suspected NIC firmware or DPDK driver issue affecting NDR       |
+|   | and x710 NICs, compared to x520 NICs.             |            | throughput. Applies to XL710 and X710 NICs.                     |
++---+---------------------------------------------------+------------+-----------------------------------------------------------------+
\ No newline at end of file

diff --git a/docs/report/vpp_functional_tests/csit_release_notes.rst b/docs/report/vpp_functional_tests/csit_release_notes.rst
index 3b2d02e..8f63b8e 100644 (file)
--- a/docs/report/vpp_functional_tests/csit_release_notes.rst
+++ b/docs/report/vpp_functional_tests/csit_release_notes.rst
@@ -4,17 +4,24 @@ CSIT Release Notes
 Changes in CSIT |release|
 -------------------------
 
 Changes in CSIT |release|
 -------------------------
 

-#. Introduction of Centos7 tests
+#. VPP functional test environment changes

 
 #. Implemented VAT command history collection for every test case as part of teardown
 
 
 #. Implemented VAT command history collection for every test case as part of teardown
 

+    - Introduction of Centos7 tests in VIRL environment.
+
+#. VPP performance test framework changes
+
+    - Added VAT command history collection for every test case as part of teardown.
+

 #. Added VPP functional tests
 
 #. Added VPP functional tests
 

-    - IPv4 routed-forwarding with dot1q VLAN sub-interfaces
-    - L2BD switched-forwarding with dot1q VLAN sub-interfaces and vhost-user to VM
-    - IPv4 routed-forwarding with vhost-user interfaces to VM
-    - Vhost-User interface re-connect tests
+    - IPv4 routed-forwarding with dot1q VLAN sub-interfaces.
+    - L2BD switched-forwarding with dot1q VLAN sub-interfaces and vhost-user to VM.
+    - IPv4 routed-forwarding with vhost-user interfaces to VM.
+    - Vhost-user interface re-connect tests.

 
 

+>>>>>>> csit rls1704 report - updated csit_release_notes.rst and overview.rst files.

 Known Issues
 ------------
 
 Known Issues
 ------------
 

diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst
index 10e01ab..7c17e0f 100644 (file)
--- a/docs/report/vpp_performance_tests/csit_release_notes.rst
+++ b/docs/report/vpp_performance_tests/csit_release_notes.rst
@@ -6,31 +6,36 @@ Changes in CSIT |release|
 \r
 #. VPP performance test environment changes\r
 \r
 \r
 #. VPP performance test environment changes\r
 \r

-    - Further VM and vhost-user test environment optimizations - Qemu virtio\r
+    - Further optimizations of VM and vhost-user test environment - Qemu virtio\r

       queue size increased from default value of 256 to 1024.\r
       queue size increased from default value of 256 to 1024.\r

-    - Addition of HW cryptodev devices in all three LF FD.io physical testbeds.\r
+    - Addition of HW cryptodev devices - Intel QAT 8950 50G - in all three\r
+      LF FD.io physical testbeds.\r

 \r
 \r

-#. Added tests\r
+#. VPP performance test framework changes\r

 \r
 \r

-    - CGNAT\r
+    - Added VAT command history collection for every test case as part of teardown.\r
+\r
+#. Added VPP performance tests\r
+\r
+    - **CGNAT**\r

 \r
       - Carrier Grade Network Address Translation tests with varying number\r
         of users and ports per user: 1u-15p, 10u-15p, 100u-15p, 1000u-15p,\r
         2000u-15p, 4000u-15p - with Intel x520 NIC.\r
 \r
 \r
       - Carrier Grade Network Address Translation tests with varying number\r
         of users and ports per user: 1u-15p, 10u-15p, 100u-15p, 1000u-15p,\r
         2000u-15p, 4000u-15p - with Intel x520 NIC.\r
 \r

-    - vhost-user tests with one VM\r
+    - **vhost-user tests with one VM**\r

 \r
       - L2 Bridge Domain switched-forwarding with Intel x710 NIC, Intel x520 NIC,\r
         Intel xl710 NIC.\r
       - VXLAN and L2 Bridge Domain switched-forwarding with Intel x520 NIC.\r
 \r
 \r
       - L2 Bridge Domain switched-forwarding with Intel x710 NIC, Intel x520 NIC,\r
         Intel xl710 NIC.\r
       - VXLAN and L2 Bridge Domain switched-forwarding with Intel x520 NIC.\r
 \r

-    - vhost-user tests with two VM service chain\r
+    - **vhost-user tests with two VMs service chain**\r

 \r
       - L2 cross-connect switched-forwarding with Intel x520 NIC, Intel xl710 NIC.\r
       - L2 Bridge Domain switched-forwarding with Intel x520 NIC, Intel xl710 NIC.\r
       - IPv4 routed-forwarding with Intel x520 NIC, Intel xl710 NIC.\r
 \r
 \r
       - L2 cross-connect switched-forwarding with Intel x520 NIC, Intel xl710 NIC.\r
       - L2 Bridge Domain switched-forwarding with Intel x520 NIC, Intel xl710 NIC.\r
       - IPv4 routed-forwarding with Intel x520 NIC, Intel xl710 NIC.\r
 \r

-    - IPSec encryption with\r
+    - **IPSec encryption with**\r

 \r
       - AES-GCM, CBC-SHA1 ciphers, in combination with IPv4 routed-forwarding\r
         with Intel xl710 NIC.\r
 \r
       - AES-GCM, CBC-SHA1 ciphers, in combination with IPv4 routed-forwarding\r
         with Intel xl710 NIC.\r


@@ -205,22 +210,21 @@ Here is the list of known issues in CSIT |release| for VPP performance tests:
 |   | for ip4scale200k, ip4scale2m scale IPv4 routed- |            | Observed frequency: all test runs.                              |\r
 |   | forwarding tests. ip4scale20k tests are fine.   |            |                                                                 |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 |   | for ip4scale200k, ip4scale2m scale IPv4 routed- |            | Observed frequency: all test runs.                              |\r
 |   | forwarding tests. ip4scale20k tests are fine.   |            |                                                                 |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r

-| 2 | VAT API timeouts during ip6scale2m scale IPv6   |            | Needs fixing VPP VAT API timeouts for large volume of IPv6      |\r
-|   | routed-forwarding tests when volume adding IPv6 | VPP-?      | routes.                                                         |\r
+| 2 | VAT API timeouts during ip6scale2m scale IPv6   | VPP-712    | Needs fixing VPP VAT API timeouts for large volume of IPv6      |\r
+|   | routed-forwarding tests when volume adding IPv6 |            | routes.                                                         |\r

 |   | routes - 2M in this case. ip6scale2kk works.    |            |                                                                 |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 |   | routes - 2M in this case. ip6scale2kk works.    |            |                                                                 |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r

-| 3 | Vic1385 and Vic1227 low performance             | CSIT-?     | Low NDR performance.                                            |\r
+| 3 | Vic1385 and Vic1227 low performance             | VPP-664    | Low NDR performance.                                            |\r

 |   |                                                 |            |                                       .                         |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 |   |                                                 |            |                                       .                         |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r

-| 4 | Sporadic NDR discovery test failures on x520    | CSIT-?     | Suspected issue with HW settings (BIOS, FW) in LF               |\r
+| 4 | Sporadic NDR discovery test failures on x520    | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |\r

 |   |                                                 |            | infrastructure. Issue can't be replicated outside LF.           |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 |   |                                                 |            | infrastructure. Issue can't be replicated outside LF.           |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r

-| 5 | Testpmd - Non-repeatible zig-zagging NDR        | CSIT-?     | Suspected NIC firmware or driver issue affecting NDR            |\r
-|   | throughput in multi-thread/-core tests          |            | in multi-thread/-core operation. Need to update to latest       |\r
-|   | - 2t2c - for some tested NICs.                  |            | firmware in NICs. Applies to XL710 and X710 NICs.               |\r
+| 5 | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |\r
+|   | of discovered NDR throughput values across      |            | throughput. Applies to XL710 and X710 NICs, x520 NICs are fine. |\r
+|   | multiple test runs with xl710 and x710 NICs.    |            |                                       .                         |\r

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r

-| 6 | VPP - Non-repeatible zig-zagging NDR            | CSIT-?     | Suspected NIC firmware or driver issue affecting NDR            |\r
-|   | throughput in multi-thread/-core tests          |            | in multi-thread/-core operation. Need to update to latest       |\r
-|   | - 2t2c - for some tested NICs.                  |            | firmware in NICs. Applies to XL710 and X710 NICs.               |\r
+| 6 | Lower than expected NDR and PDR throughput with | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |\r
+|   | xl710 and x710 NICs, compared to x520 NICs.     |            | PDR throughput. Applies to XL710 and X710 NICs.                 |\r

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 \r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 \r

diff --git a/docs/report/vpp_performance_tests/overview.rst b/docs/report/vpp_performance_tests/overview.rst
index 96a9377..56ffda0 100644 (file)
--- a/docs/report/vpp_performance_tests/overview.rst
+++ b/docs/report/vpp_performance_tests/overview.rst
@@ -238,11 +238,6 @@ suites:
       switching to/from four vhost interfaces and two VMs, NDR throughput
       discovery.
 
       switching to/from four vhost interfaces and two VMs, NDR throughput
       discovery.
 

-Methodology: TRex Traffic Generator Usage
------------------------------------------
-
-TODO Description to be added.
-

 Methodology: Multi-Thread and Multi-Core
 ----------------------------------------
 
 Methodology: Multi-Thread and Multi-Core
 ----------------------------------------
 


@@ -338,5 +333,56 @@ guest dealing with data plan.
 Methodology: IPSec with Intel QAT HW cards
 ------------------------------------------
 
 Methodology: IPSec with Intel QAT HW cards
 ------------------------------------------
 

-TODO Description to be added.
-Intel QAT 8950 50G (Walnut Hill)
\ No newline at end of file
+VPP IPSec performance tests are using DPDK cryptodev device driver in
+combination with HW cryptodev devices - Intel QAT 8950 50G - present in
+LF FD.io physical testbeds. DPDK cryptodev can be used for all IPSec
+data plane functions supported by VPP.
+
+Currently CSIT |release| implements following IPSec test cases:
+
+- AES-GCM, CBC-SHA1 ciphers, in combination with IPv4 routed-forwarding
+  with Intel xl710 NIC.
+- CBC-SHA1 ciphers, in combination with LISP-GPE overlay tunneling for
+  IPv4-over-IPv4 with Intel xl710 NIC.
+
+Methodology: TRex Traffic Generator Usage
+-----------------------------------------
+
+The `TRex traffic generator <https://wiki.fd.io/view/TRex>`_ is used for all
+CSIT performance tests. TRex stateless mode is used to measure NDR and PDR
+throughputs using binary search (NDR and PDR discovery tests) and for quick
+checks of DUT performance against the reference NDRs (NDR check tests) for
+specific configuration.
+
+TRex is installed and run on the TG compute node. The typical procedure is:
+
+    - If the TRex is not already installed on TG, it is installed in the
+      suite setup phase - see `TRex intallation <https://gerrit.fd.io/r/gitweb?p=csit.git;a=blob;f=resources/tools/t-rex/t-rex-installer.sh;h=8090b7568327ac5f869e82664bc51b24f89f603f;hb=refs/heads/rls1704>`_.
+    - TRex configuration is set in its configuration file::
+
+      /etc/trex_cfg.yaml
+
+    - TRex is started in the background mode::
+
+      sh -c 'cd /opt/trex-core-2.22/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --iom 0 > /dev/null 2>&1 &' > /dev/null
+
+    - There are traffic streams dynamically prepared for each test. The traffic
+      is sent and the statistics obtained using trex_stl_lib.api.STLClient.
+
+**Measuring packet loss**
+
+    - Create an instance of STLClient
+    - Connect to the client
+    - Add all streams
+    - Clear statistics
+    - Send the traffic for defined time
+    - Get the statistics
+
+If there is a warm-up phase required, the traffic is sent also before test and
+the statistics are ignored.
+
+**Measuring latency**
+
+If measurement of latency is requested, two more packet streams are created (one
+for each direction) with TRex flow_stats parameter set to STLFlowLatencyStats. In
+that case, returned statistics will also include min/avg/max latency values.