Add known issues to report 18.07

Change-Id: Ide101c80e2ef13021f25096da6afd9e4364ee263
Signed-off-by: Tibor Frank <tifrank@cisco.com>

diff --git a/docs/report/introduction/index.rst b/docs/report/introduction/index.rst
index ba6599c..81a7dc3 100644 (file)
--- a/docs/report/introduction/index.rst
+++ b/docs/report/introduction/index.rst
@@ -3,6 +3,7 @@ Introduction
 
 .. toctree::
 
+    report_history
     introduction
     test_scenarios_overview
     physical_testbeds

diff --git a/docs/report/introduction/report_history.rst b/docs/report/introduction/report_history.rst
new file mode 100644 (file)
index 0000000..d54f526
--- /dev/null
+++ b/docs/report/introduction/report_history.rst
@@ -0,0 +1,15 @@
+Report History
+==============
+
++---------+--------------------------------------------------------------------+
+| Version | Change                                                             |
++=========+====================================================================+
+| 1.1     | The known issues                                                   |
+|         | `CSIT-1246 <https://jira.fd.io/browse/CSIT-1246>`_,                |
+|         | `CSIT-1253 <https://jira.fd.io/browse/CSIT-1253>`_,                |
+|         | `CSIT-1266 <https://jira.fd.io/browse/CSIT-1266>`_,                |
+|         | `CSIT-1267 <https://jira.fd.io/browse/CSIT-1267>`_                 |
+|         | added to :ref:`vpp_known_issues`.                                  |
++---------+--------------------------------------------------------------------+
+| 1.0     | Initial version                                                    |
++---------+--------------------------------------------------------------------+

diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst
index 32bbbbc..379072c 100644 (file)
--- a/docs/report/vpp_performance_tests/csit_release_notes.rst
+++ b/docs/report/vpp_performance_tests/csit_release_notes.rst
@@ -54,38 +54,53 @@ Changes in |csit-release|
    - **General Code Housekeeping**: Ongoing RF keywords optimizations,
      removal of redundant RF keywords.
 
+.. _vpp_known_issues:
+
 Known Issues
 ------------
 
 List of known issues in |csit-release| for VPP performance tests:
 
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| # | JiraID                                  | Issue Description                                                                                                               |
-+===+=========================================+=================================================================================================================================+
-| 1 | `CSIT-570                               | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue.                         |
-|   | <https://jira.fd.io/browse/CSIT-570>`_  | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds.                           |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 2 | `VPP-1361                               | High failure rate of api call sw_interface_set_flags [admin-up|link-up].                                                        |
-|   | <https://jira.fd.io/browse/VPP-1361>`_  | Failure rate: 30-40% of tests failing due to interfaces not in link-up state after API call sw_interface_set_flags.             |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 3 | `CSIT-1234                              | VPP IPSecHW scale interface mode 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04.        |
-|   | <https://jira.fd.io/browse/CSIT-1234>`_ | ip4ipsecscale1000tnl-ip4base-int 1core CSIT-18.07/18.04 relative change: NDR -31%, PDR -32%, MRR -38%.                          |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 4 | `CSIT-1242                              | VPP xl710 ip4base test 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04.                  |
-|   | <https://jira.fd.io/browse/CSIT-1242>`_ | xl710 ip4base 1core CSIT-18.07/18.04 relative change: NDR -29%, high stdev.                                                     |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 5 | `CSIT-1243                              | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds.                      |
-|   | <https://jira.fd.io/browse/CSIT-1243>`_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%.                                                          |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 6 | `CSIT-1244                              | VPP lispip4 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds.                    |
-|   | <https://jira.fd.io/browse/CSIT-1244>`_ | ip4lispip4-ip4base 2core 3n-skx/3n-hsw relative change: NDR -11%, PDR -18%.                                                     |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 7 | `CSIT-1245                              | VPP srv6proxy-stat and srv6proxy-masq, much higher NDR and PDR 78B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. |
-|   | <https://jira.fd.io/browse/CSIT-1245>`_ | Due to wrong test suite configuration in dynamic-proxy mode. Artefact of suite code refactoring.                                |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 8 | `CSIT-1246                              | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07.        |
-|   | <https://jira.fd.io/browse/CSIT-1246>`_ | Past vpp-agent releases are not compatible either.                                                                              |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 9 | `CSIT-1253                              | VPP lbdpdk link bonding tests failing due to interfaces not coming up.                                                          |
-|   | <https://jira.fd.io/browse/CSIT-1253>`_ | VPP lbdpdk link bonding tests relying on DPDK functionality for bonding fail.                                                   |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| #  | JiraID                                  | Issue Description                                                                                                               |
++====+=========================================+=================================================================================================================================+
+| 1  | `CSIT-570                               | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue.                         |
+|    | <https://jira.fd.io/browse/CSIT-570>`_  | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds.                           |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 2  | `VPP-1361                               | High failure rate of api call sw_interface_set_flags [admin-up|link-up].                                                        |
+|    | <https://jira.fd.io/browse/VPP-1361>`_  | Failure rate: 30-40% of tests failing due to interfaces not in link-up state after API call sw_interface_set_flags.             |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 3  | `CSIT-1234                              | VPP IPSecHW scale interface mode 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04.        |
+|    | <https://jira.fd.io/browse/CSIT-1234>`_ | ip4ipsecscale1000tnl-ip4base-int 1core CSIT-18.07/18.04 relative change: NDR -31%, PDR -32%, MRR -38%.                          |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 4  | `CSIT-1242                              | VPP xl710 ip4base test 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04.                  |
+|    | <https://jira.fd.io/browse/CSIT-1242>`_ | xl710 ip4base 1core CSIT-18.07/18.04 relative change: NDR -29%, high stdev.                                                     |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 5  | `CSIT-1243                              | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds.                      |
+|    | <https://jira.fd.io/browse/CSIT-1243>`_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%.                                                          |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 6  | `CSIT-1244                              | VPP lispip4 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds.                    |
+|    | <https://jira.fd.io/browse/CSIT-1244>`_ | ip4lispip4-ip4base 2core 3n-skx/3n-hsw relative change: NDR -11%, PDR -18%.                                                     |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 7  | `CSIT-1245                              | VPP srv6proxy-stat and srv6proxy-masq, much higher NDR and PDR 78B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. |
+|    | <https://jira.fd.io/browse/CSIT-1245>`_ | Due to wrong test suite configuration in dynamic-proxy mode. Artefact of suite code refactoring.                                |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 8  | `CSIT-1246                              | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07.        |
+|    | <https://jira.fd.io/browse/CSIT-1246>`_ | Past vpp-agent releases are not compatible either.                                                                              |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 9  | `CSIT-1253                              | VPP lbdpdk link bonding tests failing due to interfaces not coming up.                                                          |
+|    | <https://jira.fd.io/browse/CSIT-1253>`_ | VPP lbdpdk link bonding tests relying on DPDK functionality for bonding fail.                                                   |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 10 | `CSIT-1266                              | Because of wrong Multi Seq configuration in the suite "csit/tests/vpp/perf/ip4/2n1l-25ge2p1xxv710-ethip4-ip4base-mrr.robot",    |
+|    | <https://jira.fd.io/browse/CSIT-1266>`_ | the data generated by this suite is incorrect. The bug was fixed by the `patch 14440 <https://gerrit.fd.io/r/#/c/14440>`_.      |
+|    |                                         | The incorrect data will be replaced by new generated correct data.                                                              |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 11 | `CSIT-1267                              | Because of the wrong NIC configuration in suites                                                                                |
+|    | <https://jira.fd.io/browse/CSIT-1267>`_ | tests/vpp/perf/vm_vhost/10ge2p1x710-1lbvpplacp-dot1q-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr.robot,                             |
+|    |                                         | tests/vpp/perf/vm_vhost/10ge2p1x710-1lbvpplacp-dot1q-l2bdbasemaclrn-eth-2vhostvr1024-1vm-ndrpdr.robot,                          |
+|    |                                         | tests/vpp/perf/vm_vhost/10ge2p1x710-1lbvpplacp-dot1q-l2xcbase-eth-2vhostvr1024-1vm-mrr.robot,                                   |
+|    |                                         | tests/vpp/perf/vm_vhost/10ge2p1x710-1lbvpplacp-dot1q-l2xcbase-eth-2vhostvr1024-1vm-ndrpdr.robot                                 |
+|    |                                         | the data generated by this suites is incorrect.                                                                                 |
+|    |                                         | The bug was fixed by the `patch 14395 <https://gerrit.fd.io/r/#/c/14395>`_.                                                     |
+|    |                                         | The incorrect data will be replaced by new generated correct data.                                                              |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+

diff --git a/resources/tools/presentation/run_report.sh b/resources/tools/presentation/run_report.sh
index 9435ce0..2792190 100755 (executable)
--- a/resources/tools/presentation/run_report.sh
+++ b/resources/tools/presentation/run_report.sh
@@ -38,7 +38,7 @@ export PYTHONPATH=`pwd`
 python pal.py \
     --specification specification.yaml \
     --release ${RELEASE} \
-    --version "1.0" \
+    --version "1.1" \
     --logging INFO \
     --force