From: Tibor Frank Date: Wed, 5 Sep 2018 11:21:19 +0000 (+0200) Subject: Add known issues to report 18.07 X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=commitdiff_plain;h=79ecf968f27fca0a75d6db5ab40920ba04873448 Add known issues to report 18.07 Change-Id: Ide101c80e2ef13021f25096da6afd9e4364ee263 Signed-off-by: Tibor Frank --- diff --git a/docs/report/introduction/index.rst b/docs/report/introduction/index.rst index ba6599cfc8..81a7dc3c49 100644 --- a/docs/report/introduction/index.rst +++ b/docs/report/introduction/index.rst @@ -3,6 +3,7 @@ Introduction .. toctree:: + report_history introduction test_scenarios_overview physical_testbeds diff --git a/docs/report/introduction/report_history.rst b/docs/report/introduction/report_history.rst new file mode 100644 index 0000000000..d54f526f55 --- /dev/null +++ b/docs/report/introduction/report_history.rst @@ -0,0 +1,15 @@ +Report History +============== + ++---------+--------------------------------------------------------------------+ +| Version | Change | ++=========+====================================================================+ +| 1.1 | The known issues | +| | `CSIT-1246 `_, | +| | `CSIT-1253 `_, | +| | `CSIT-1266 `_, | +| | `CSIT-1267 `_ | +| | added to :ref:`vpp_known_issues`. | ++---------+--------------------------------------------------------------------+ +| 1.0 | Initial version | ++---------+--------------------------------------------------------------------+ diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index 32bbbbcfb6..379072c119 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -54,38 +54,53 @@ Changes in |csit-release| - **General Code Housekeeping**: Ongoing RF keywords optimizations, removal of redundant RF keywords. +.. _vpp_known_issues: + Known Issues ------------ List of known issues in |csit-release| for VPP performance tests: -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| # | JiraID | Issue Description | -+===+=========================================+=================================================================================================================================+ -| 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. | -| | `_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 2 | `VPP-1361 | High failure rate of api call sw_interface_set_flags [admin-up|link-up]. | -| | `_ | Failure rate: 30-40% of tests failing due to interfaces not in link-up state after API call sw_interface_set_flags. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 3 | `CSIT-1234 | VPP IPSecHW scale interface mode 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | -| | `_ | ip4ipsecscale1000tnl-ip4base-int 1core CSIT-18.07/18.04 relative change: NDR -31%, PDR -32%, MRR -38%. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 4 | `CSIT-1242 | VPP xl710 ip4base test 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | -| | `_ | xl710 ip4base 1core CSIT-18.07/18.04 relative change: NDR -29%, high stdev. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 5 | `CSIT-1243 | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. | -| | `_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 6 | `CSIT-1244 | VPP lispip4 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. | -| | `_ | ip4lispip4-ip4base 2core 3n-skx/3n-hsw relative change: NDR -11%, PDR -18%. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 7 | `CSIT-1245 | VPP srv6proxy-stat and srv6proxy-masq, much higher NDR and PDR 78B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | -| | `_ | Due to wrong test suite configuration in dynamic-proxy mode. Artefact of suite code refactoring. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 8 | `CSIT-1246 | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07. | -| | `_ | Past vpp-agent releases are not compatible either. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 9 | `CSIT-1253 | VPP lbdpdk link bonding tests failing due to interfaces not coming up. | -| | `_ | VPP lbdpdk link bonding tests relying on DPDK functionality for bonding fail. | -+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| # | JiraID | Issue Description | ++====+=========================================+=================================================================================================================================+ +| 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. | +| | `_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 2 | `VPP-1361 | High failure rate of api call sw_interface_set_flags [admin-up|link-up]. | +| | `_ | Failure rate: 30-40% of tests failing due to interfaces not in link-up state after API call sw_interface_set_flags. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 3 | `CSIT-1234 | VPP IPSecHW scale interface mode 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | +| | `_ | ip4ipsecscale1000tnl-ip4base-int 1core CSIT-18.07/18.04 relative change: NDR -31%, PDR -32%, MRR -38%. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 4 | `CSIT-1242 | VPP xl710 ip4base test 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | +| | `_ | xl710 ip4base 1core CSIT-18.07/18.04 relative change: NDR -29%, high stdev. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 5 | `CSIT-1243 | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. | +| | `_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 6 | `CSIT-1244 | VPP lispip4 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. | +| | `_ | ip4lispip4-ip4base 2core 3n-skx/3n-hsw relative change: NDR -11%, PDR -18%. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 7 | `CSIT-1245 | VPP srv6proxy-stat and srv6proxy-masq, much higher NDR and PDR 78B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | +| | `_ | Due to wrong test suite configuration in dynamic-proxy mode. Artefact of suite code refactoring. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 8 | `CSIT-1246 | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07. | +| | `_ | Past vpp-agent releases are not compatible either. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 9 | `CSIT-1253 | VPP lbdpdk link bonding tests failing due to interfaces not coming up. | +| | `_ | VPP lbdpdk link bonding tests relying on DPDK functionality for bonding fail. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 10 | `CSIT-1266 | Because of wrong Multi Seq configuration in the suite "csit/tests/vpp/perf/ip4/2n1l-25ge2p1xxv710-ethip4-ip4base-mrr.robot", | +| | `_ | the data generated by this suite is incorrect. The bug was fixed by the `patch 14440 `_. | +| | | The incorrect data will be replaced by new generated correct data. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 11 | `CSIT-1267 | Because of the wrong NIC configuration in suites | +| | `_ | tests/vpp/perf/vm_vhost/10ge2p1x710-1lbvpplacp-dot1q-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr.robot, | +| | | tests/vpp/perf/vm_vhost/10ge2p1x710-1lbvpplacp-dot1q-l2bdbasemaclrn-eth-2vhostvr1024-1vm-ndrpdr.robot, | +| | | tests/vpp/perf/vm_vhost/10ge2p1x710-1lbvpplacp-dot1q-l2xcbase-eth-2vhostvr1024-1vm-mrr.robot, | +| | | tests/vpp/perf/vm_vhost/10ge2p1x710-1lbvpplacp-dot1q-l2xcbase-eth-2vhostvr1024-1vm-ndrpdr.robot | +| | | the data generated by this suites is incorrect. | +| | | The bug was fixed by the `patch 14395 `_. | +| | | The incorrect data will be replaced by new generated correct data. | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ diff --git a/resources/tools/presentation/run_report.sh b/resources/tools/presentation/run_report.sh index 9435ce01a6..2792190e3e 100755 --- a/resources/tools/presentation/run_report.sh +++ b/resources/tools/presentation/run_report.sh @@ -38,7 +38,7 @@ export PYTHONPATH=`pwd` python pal.py \ --specification specification.yaml \ --release ${RELEASE} \ - --version "1.0" \ + --version "1.1" \ --logging INFO \ --force