CSIT rls1704 report - small updates

Change-Id: I80b5aaf2bab77a6131fca8cf4e903ef6aecd1195
Signed-off-by: Jan Gelety <jgelety@cisco.com>
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>

diff --git a/docs/report/testpmd_performance_tests/csit_release_notes.rst b/docs/report/testpmd_performance_tests/csit_release_notes.rst
index 7f87d24..72bef87 100644 (file)
--- a/docs/report/testpmd_performance_tests/csit_release_notes.rst
+++ b/docs/report/testpmd_performance_tests/csit_release_notes.rst
@@ -20,6 +20,6 @@ Here is the list of known issues in CSIT |release| for Testpmd performance tests
 |   | of discovered NDR throughput values across        |            | throughput. Applies to XL710 and X710 NICs, no issues observed  |
 |   | multiple test runs with xl710 and x710 NICs.      |            | on x520 NICs.                                                   |
 +---+---------------------------------------------------+------------+-----------------------------------------------------------------+
 |   | of discovered NDR throughput values across        |            | throughput. Applies to XL710 and X710 NICs, no issues observed  |
 |   | multiple test runs with xl710 and x710 NICs.      |            | on x520 NICs.                                                   |
 +---+---------------------------------------------------+------------+-----------------------------------------------------------------+

-| 2 | Lower than expected NDR throughput with xl710     | CSIT-?     | Suspected NIC firmware or DPDK driver issue affecting NDR       |
+| 2 | Lower than expected NDR throughput with xl710     | CSIT-571   | Suspected NIC firmware or DPDK driver issue affecting NDR       |

 |   | and x710 NICs, compared to x520 NICs.             |            | throughput. Applies to XL710 and X710 NICs.                     |
 +---+---------------------------------------------------+------------+-----------------------------------------------------------------+
\ No newline at end of file
 |   | and x710 NICs, compared to x520 NICs.             |            | throughput. Applies to XL710 and X710 NICs.                     |
 +---+---------------------------------------------------+------------+-----------------------------------------------------------------+
\ No newline at end of file

diff --git a/docs/report/vpp_functional_tests/csit_release_notes.rst b/docs/report/vpp_functional_tests/csit_release_notes.rst
index 8f63b8e..25ca27c 100644 (file)
--- a/docs/report/vpp_functional_tests/csit_release_notes.rst
+++ b/docs/report/vpp_functional_tests/csit_release_notes.rst
@@ -6,11 +6,10 @@ Changes in CSIT |release|
 
 #. VPP functional test environment changes
 
 
 #. VPP functional test environment changes
 

-#. Implemented VAT command history collection for every test case as part of teardown
-
+    - Implemented VAT command history collection for every test case as part of teardown.

     - Introduction of Centos7 tests in VIRL environment.
 
     - Introduction of Centos7 tests in VIRL environment.
 

-#. VPP performance test framework changes
+#. VPP functional test framework changes

 
     - Added VAT command history collection for every test case as part of teardown.
 
 
     - Added VAT command history collection for every test case as part of teardown.
 


@@ -21,7 +20,6 @@ Changes in CSIT |release|
     - IPv4 routed-forwarding with vhost-user interfaces to VM.
     - Vhost-user interface re-connect tests.
 
     - IPv4 routed-forwarding with vhost-user interfaces to VM.
     - Vhost-user interface re-connect tests.
 

->>>>>>> csit rls1704 report - updated csit_release_notes.rst and overview.rst files.

 Known Issues
 ------------
 
 Known Issues
 ------------
 


@@ -55,17 +53,21 @@ Here is the list of known issues in CSIT |release| for VPP functional tests in V
 |   |                                                 |          | destination address is reported.                     |
 +---+-------------------------------------------------+----------+------------------------------------------------------+
 | 6 | IPv4 + VLAN: Processing of tagged frame doesn't | CSIT-564 | Dot1q tagged packets are thrown away in case of IPv4 |
 |   |                                                 |          | destination address is reported.                     |
 +---+-------------------------------------------------+----------+------------------------------------------------------+
 | 6 | IPv4 + VLAN: Processing of tagged frame doesn't | CSIT-564 | Dot1q tagged packets are thrown away in case of IPv4 |

-|   | work with virtio driver                         |          | routing on interface binded to virtio driver.        |
+|   | work with virtio driver                         |          | routing on interface binded to virtio driver. Issue  |
+|   |                                                 |          | with VIRL test simulation environment                |

 +---+-------------------------------------------------+----------+------------------------------------------------------+
 +---+-------------------------------------------------+----------+------------------------------------------------------+

-| 7 | VHOST-user: QEMU re-connect doesn't work        | CSIT-565 | Used QEMU 2.5.0 doesn't support re-connection. Usage |
-|   |                                                 |          | of QEMU 2.7.0 is not recommended by vpp-dev team at  |
-|   |                                                 |          | the moment.                                          |
+| 7 | Vhost-user: QEMU reconnect doesn't work         | CSIT-565 | Using QEMU 2.5.0 that does not support vhost-user    |
+|   |                                                 |          | reconnect. It requires moving CSIT VIRL environment  |
+|   |                                                 |          | to QEMU 2.7.0.                                       |

 +---+-------------------------------------------------+----------+------------------------------------------------------+
 +---+-------------------------------------------------+----------+------------------------------------------------------+

-| 8 | Centos7: LISP and VXLAN test failures           | CSIT-566 | We can observe crash of SUT1 VM during LISP tests    |
-|   |                                                 |          | execution, that leads to failure of all other tests  |
-|   |                                                 |          | (sometimes observed  during VXLAN tests execution    |
-|   |                                                 |          | too). NOTE: Possible root cause identified after     |
-|   |                                                 |          | upgrade of one of VIRL servers. When fix is          |
-|   |                                                 |          | confirmed in a short time it will be merged to CSIT  |
-|   |                                                 |          | rls1704 branch.                                      |
+| 8 | Centos7: LISP and VXLAN sporadic test failures  | CSIT-566 | Observing sporadic crashes of DUT1 VM during LISP    |
+|   |                                                 |          | and VXLAN test executions, what leads to failure of  |
+|   |                                                 |          | all other tests in the suite. NOTE: After upgrading  |
+|   |                                                 |          | one of the VIRL servers host from Ubuntu14.04 to     |
+|   |                                                 |          | 16.04, DUT1 VM stops crashing, but tests keep        |
+|   |                                                 |          | failing sporadically 1-in-100. Possible root cause   |
+|   |                                                 |          | identified - unexpected IPv6 RA packets upsetting    |
+|   |                                                 |          | some test component. Currently suspecting            |
+|   |                                                 |          | environment or CSIT issue, but can not exclude VPP,  |
+|   |                                                 |          | further troubleshooting in progress.                 |

 +---+-------------------------------------------------+----------+------------------------------------------------------+
 +---+-------------------------------------------------+----------+------------------------------------------------------+

diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst
index 7c17e0f..b4fc9de 100644 (file)
--- a/docs/report/vpp_performance_tests/csit_release_notes.rst
+++ b/docs/report/vpp_performance_tests/csit_release_notes.rst
@@ -215,14 +215,14 @@ Here is the list of known issues in CSIT |release| for VPP performance tests:
 |   | routes - 2M in this case. ip6scale2kk works.    |            |                                                                 |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 | 3 | Vic1385 and Vic1227 low performance             | VPP-664    | Low NDR performance.                                            |\r
 |   | routes - 2M in this case. ip6scale2kk works.    |            |                                                                 |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 | 3 | Vic1385 and Vic1227 low performance             | VPP-664    | Low NDR performance.                                            |\r

-|   |                                                 |            |                                       .                         |\r
+|   |                                                 |            |                                                                 |\r

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 | 4 | Sporadic NDR discovery test failures on x520    | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |\r
 |   |                                                 |            | infrastructure. Issue can't be replicated outside LF.           |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 | 5 | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |\r
 |   | of discovered NDR throughput values across      |            | throughput. Applies to XL710 and X710 NICs, x520 NICs are fine. |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 | 4 | Sporadic NDR discovery test failures on x520    | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |\r
 |   |                                                 |            | infrastructure. Issue can't be replicated outside LF.           |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 | 5 | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |\r
 |   | of discovered NDR throughput values across      |            | throughput. Applies to XL710 and X710 NICs, x520 NICs are fine. |\r

-|   | multiple test runs with xl710 and x710 NICs.    |            |                                       .                         |\r
+|   | multiple test runs with xl710 and x710 NICs.    |            |                                                                 |\r

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 | 6 | Lower than expected NDR and PDR throughput with | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |\r
 |   | xl710 and x710 NICs, compared to x520 NICs.     |            | PDR throughput. Applies to XL710 and X710 NICs.                 |\r
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+\r
 | 6 | Lower than expected NDR and PDR throughput with | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |\r
 |   | xl710 and x710 NICs, compared to x520 NICs.     |            | PDR throughput. Applies to XL710 and X710 NICs.                 |\r

diff --git a/docs/report/vpp_unit_tests/overview.rst b/docs/report/vpp_unit_tests/overview.rst
index 2848db7..4ef50f5 100644 (file)
--- a/docs/report/vpp_unit_tests/overview.rst
+++ b/docs/report/vpp_unit_tests/overview.rst
@@ -33,7 +33,7 @@ Unit Tests Coverage
 -------------------\r
 \r
 Following VPP functional test areas are covered in VPP unit test code included\r
 -------------------\r
 \r
 Following VPP functional test areas are covered in VPP unit test code included\r

-in VPP rls1701 with results listed in this report:\r
+in VPP rls1704 with results listed in this report:\r

 \r
 - CRUD Loopback - create, read, update, delete Loopback interfaces.\r
 - Flow-per-packet plugin - collect and report L2 and IP4 flow statistics.\r
 \r
 - CRUD Loopback - create, read, update, delete Loopback interfaces.\r
 - Flow-per-packet plugin - collect and report L2 and IP4 flow statistics.\r

diff --git a/resources/tools/report_gen/run_report.sh b/resources/tools/report_gen/run_report.sh
index 00fbdfd..4aa15b7 100755 (executable)
--- a/resources/tools/report_gen/run_report.sh
+++ b/resources/tools/report_gen/run_report.sh
@@ -238,7 +238,7 @@ sed -i -e "s@###LINK###@${JEN_URL}\/${JEN_JOB}\/${JEN_BUILD[-1]}@g" \
 ### FUNCTIONAL SOURCE DATA
 
 JEN_JOB='csit-vpp-functional-1704-ubuntu1604-virl'
 ### FUNCTIONAL SOURCE DATA
 
 JEN_JOB='csit-vpp-functional-1704-ubuntu1604-virl'

-JEN_BUILD=57
+JEN_BUILD=63

 
 if [[ ${DEBUG} -eq 1 ]] ;
 then
 
 if [[ ${DEBUG} -eq 1 ]] ;
 then