From: Peter Mikus Date: Tue, 25 Jul 2017 09:54:26 +0000 (+0200) Subject: CSIT-618 Release report update VIII X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=commitdiff_plain;h=1e2bf354994a7ec53c1acee2d1c2fe2208f7c905 CSIT-618 Release report update VIII Change-Id: Idec390861aa5ac8a1a485d5996fbfef444ac8a29 Signed-off-by: Peter Mikus --- diff --git a/docs/report/dpdk_performance_tests/test_environment.rst b/docs/report/dpdk_performance_tests/test_environment.rst index e8ed76d761..a8c0a485d2 100644 --- a/docs/report/dpdk_performance_tests/test_environment.rst +++ b/docs/report/dpdk_performance_tests/test_environment.rst @@ -369,7 +369,8 @@ Below a subset of the running configuration: - **isolcpus=-** used for all cpu cores apart from first core of each socket used for running VPP worker threads and Qemu/LXC - processes https://www.kernel.org/doc/Documentation/kernel-parameters.txt + processes + https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt - **intel_pstate=disable** - [X86] Do not enable intel_pstate as the default scaling driver for the supported processors. Intel P-State driver decide what P-state (CPU core power state) to use based on requesting policy from the @@ -387,7 +388,7 @@ Below a subset of the running configuration: - **rcu_nocbs** - [KNL] In kernels built with CONFIG_RCU_NOCB_CPU=y, set the specified list of CPUs to be no-callback CPUs, that never queue RCU callbacks (read-copy update). - https://www.kernel.org/doc/Documentation/kernel-parameters.txt + https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt **Applied command line boot parameters:** diff --git a/docs/report/honeycomb_functional_tests/test_environment.rst b/docs/report/honeycomb_functional_tests/test_environment.rst index 92431c6c32..b55bfad602 100644 --- a/docs/report/honeycomb_functional_tests/test_environment.rst +++ b/docs/report/honeycomb_functional_tests/test_environment.rst @@ -13,7 +13,7 @@ versions: Current VPP tests have been executed on a single VM operating system and version only, as described in the following paragraphs. -In CSIT terminology, the VM operating system for both SUTs and TG that +In CSIT terminology, the VM operating system for both DUTs and TG that |vpp-release| has been tested with, is the following: |virl-image-ubuntu| @@ -23,10 +23,8 @@ package versions are those that would have been installed by a "apt-get update", "apt-get upgrade" on that day), produced by CSIT disk image build scripts. The exact list of installed packages and their versions (including the Linux -kernel package version) are included in CSIT source repository: +kernel package version) are included in `VIRL images lists`_. - resources/tools/disk-image-builder/ubuntu/lists/|virl-image-ubuntu| - -A replica of this VM image can be built by running the "build.sh" script in CSIT -repository resources/tools/disk-image-builder/ubuntu. +A replica of this VM image can be built by running the "build.sh" script in +`VIRL nested`_. diff --git a/docs/report/honeycomb_performance_tests/csit_release_notes.rst b/docs/report/honeycomb_performance_tests/csit_release_notes.rst index 51b62a7a6a..75d5e9eb3b 100644 --- a/docs/report/honeycomb_performance_tests/csit_release_notes.rst +++ b/docs/report/honeycomb_performance_tests/csit_release_notes.rst @@ -4,7 +4,7 @@ CSIT Release Notes Changes in CSIT |release| ------------------------- -#. First release with honeycomb performance testing +#. First release with Honeycomb performance testing. Known Issues ------------ diff --git a/docs/report/honeycomb_performance_tests/test_environment.rst b/docs/report/honeycomb_performance_tests/test_environment.rst index 1cafe26aa4..db187b2c1b 100644 --- a/docs/report/honeycomb_performance_tests/test_environment.rst +++ b/docs/report/honeycomb_performance_tests/test_environment.rst @@ -2,7 +2,7 @@ Test Environment ================ To execute performance tests, there are three identical testbeds, each testbed -consists of two SUTs and one TG. +consists of two DUTs and one TG. Server HW Configuration ----------------------- diff --git a/docs/report/introduction/csit_design.rst b/docs/report/introduction/csit_design.rst index baba58f904..7586e1dc8f 100644 --- a/docs/report/introduction/csit_design.rst +++ b/docs/report/introduction/csit_design.rst @@ -1,3 +1,5 @@ +.. _csit-design: + CSIT Design =========== diff --git a/docs/report/nsh_sfc_functional_tests/overview.rst b/docs/report/nsh_sfc_functional_tests/overview.rst index c9d6ce7502..79403a2272 100644 --- a/docs/report/nsh_sfc_functional_tests/overview.rst +++ b/docs/report/nsh_sfc_functional_tests/overview.rst @@ -100,19 +100,19 @@ NSH_SFC Functional Tests Coverage Following NSH_SFC functional test areas are covered in the CSIT |release| with results listed in this report: -- **NSH SFC Classifier** - TG send some TCP packets to test NSH SFC +- **NSH SFC Classifier** - TG sends some TCP packets to test NSH SFC Classifier functional. DUT1 will receive these packets from one NIC and loopback the VXLAN-GPE-NSH encapsulated packets to the TG from other NIC. - Test case count: 7 -- **NSH SFC Proxy Inbound** - TG send some VXLAN-GPE-NSH encapsulated packets +- **NSH SFC Proxy Inbound** - TG sends some VXLAN-GPE-NSH encapsulated packets to test NSH SFC Proxy Inbound functional. DUT1 will receive these packets from one NIC and loopback the VXLAN encapsulated packets to the TG from other NIC. - Test case count: 6 -- **NSH SFC Proxy Outbound** - TG send some VXLAN encapsulated packets to test +- **NSH SFC Proxy Outbound** - TG sends some VXLAN encapsulated packets to test NSH SFC Proxy Outbound functional. DUT1 will receive these packets from one NIC and loopback the VXLAN-GPE-NSH encapsulated packets to the TG from other NIC. - Test case count: 6 -- **NSH SFC Service Function Forward** - TG send some VXLAN-GPE-NSH encapsulated +- **NSH SFC Service Function Forward** - TG sends some VXLAN-GPE-NSH encapsulated packets to test NSH SFC Service Function Forward functional. DUT1 will receive these packets from one NIC and swap the VXLAN-GPE-NSH header, after that DUT1 loopback the VXLAN-GPE-NSH encapsulated packtes to the TG from other NIC. diff --git a/docs/report/vpp_functional_tests/csit_release_notes.rst b/docs/report/vpp_functional_tests/csit_release_notes.rst index 08fe87053d..c547258091 100644 --- a/docs/report/vpp_functional_tests/csit_release_notes.rst +++ b/docs/report/vpp_functional_tests/csit_release_notes.rst @@ -23,8 +23,8 @@ Changes in CSIT |release| on CSIT wiki page `Design_Optimizations `_; - - For more detail see the CSIT Framework Design section in this - report; + - For more detail see the :ref:`CSIT Framework Design ` section + in this report; Known Issues ------------ @@ -68,6 +68,6 @@ Here is the list of known issues in CSIT |release| for VPP functional tests in V | | | | environment or CSIT issue, but can not exclude VPP, | | | | | further troubleshooting in progress. | +---+-------------------------------------------------+----------+------------------------------------------------------+ -| 7 | IPSEC: Tests cover old crypto code path | -- | There are used default conf settings so IPSEC tests | +| 7 | IPSEC: Tests cover old crypto code path | CSIT-733 | There are used default conf settings so IPSEC tests | | | | | use old security code not the new Crypto SW code. | +---+-------------------------------------------------+----------+------------------------------------------------------+ diff --git a/docs/report/vpp_functional_tests/test_environment.rst b/docs/report/vpp_functional_tests/test_environment.rst index 93e3955f6f..15c27ec708 100644 --- a/docs/report/vpp_functional_tests/test_environment.rst +++ b/docs/report/vpp_functional_tests/test_environment.rst @@ -42,9 +42,7 @@ VIRL simulations. SUT Configuration - VIRL Guest VM --------------------------------- -Configuration of the SUT VMs is defined in file - - /csit/resources/tools/virl/topologies/double-ring-nested.xenial.virl +Configurations of the SUT VMs is defined in `VIRL topologies directory`_ - List of SUT VM interfaces::: @@ -65,48 +63,43 @@ SUT Configuration - VIRL Guest OS Linux In CSIT terminology, the VM operating system for both SUTs that |vpp-release| has been tested with, is the following: -**#. Ubuntu VIRL image** - -This image implies Ubuntu 16.04.1 LTS, current as of yyyy-mm-dd (that is, -package versions are those that would have been installed by a "apt-get update", -"apt-get upgrade" on that day), produced by CSIT disk image build scripts. - -The exact list of installed packages and their versions (including the Linux -kernel package version) are included in CSIT source repository: - - resources/tools/disk-image-builder/ubuntu/lists/|virl-image-ubuntu| - -A replica of this VM image can be built by running the "build.sh" script in CSIT -repository resources/tools/disk-image-builder/ubuntu. - -**#. CentOS VIRL image** - -The Centos7.3 image is ready to be used but no tests running on it now. -Corresponding Jenkins jobs are under preparation. - -The exact list of installed packages and their versions (including the Linux -kernel package version) are included in CSIT source repository: - - resources/tools/disk-image-builder/ubuntu/lists/|virl-image-centos| - -A replica of this VM image can be built by running the "build.sh" script in CSIT -repository resources/tools/disk-image-builder/centos. - -**#. Nested VM image** - -In addition to the "main" VM image, tests which require VPP to communicate to a -VM over a vhost-user interface, utilize a "nested" VM image. - -This "nested" VM is dynamically created and destroyed as part of a test case, -and therefore the "nested" VM image is optimized to be small, lightweight and -have a short boot time. The "nested" VM image is not built around any -established Linux distribution, but is based on `BuildRoot -`_, a tool for building embedded Linux systems. Just as -for the "main" image, scripts to produce an identical replica of the "nested" -image are included in CSIT GIT repository, and the image can be rebuilt using -the "build.sh" script at: - - resources/tools/disk-image-builder/ubuntu/lists/nested +#. **Ubuntu VIRL image** + + This image implies Ubuntu 16.04.1 LTS, current as of yyyy-mm-dd (that is, + package versions are those that would have been installed by a "apt-get + update", "apt-get upgrade" on that day), produced by CSIT disk image build + scripts. + + The exact list of installed packages and their versions (including the Linux + kernel package version) are included in `VIRL images lists`_. + + A replica of this VM image can be built by running the "build.sh" script in + CSIT repository. + +#. **CentOS VIRL image** + + The Centos7.3 image is ready to be used but no tests running on it now. + Corresponding Jenkins jobs are under preparation. + + The exact list of installed packages and their versions (including the Linux + kernel package version) are included in `VIRL images lists`_. + + A replica of this VM image can be built by running the "build.sh" script in + CSIT repository. + +#. **Nested VM image** + + In addition to the "main" VM image, tests which require VPP to communicate to + a VM over a vhost-user interface, utilize a "nested" VM image. + + This "nested" VM is dynamically created and destroyed as part of a test case, + and therefore the "nested" VM image is optimized to be small, lightweight and + have a short boot time. The "nested" VM image is not built around any + established Linux distribution, but is based on `BuildRoot + `_, a tool for building embedded Linux systems. Just + as for the "main" image, scripts to produce an identical replica of the + "nested" image are included in CSIT GIT repository, and the image can be + rebuilt using the "build.sh" script at `VIRL nested`_. DUT Configuration - VPP ----------------------- @@ -117,9 +110,7 @@ Under Test (DUT) node. **DUT port configuration** Port configuration of DUTs is defined in topology file that is generated per -VIRL simulation based on the definition stored in file - - /csit/resources/tools/virl/topologies/double-ring-nested.xenial.yaml +VIRL simulation based on the definition stored in `VIRL topologies directory`_. Example of DUT nodes configuration::: @@ -373,7 +364,7 @@ Traffic scripts of test cases are executed on this VM. **TG VM configuration** -Configuration of the TG VMs is defined in file +Configuration of the TG VMs is defined in `VIRL topologies directory`_. /csit/resources/tools/virl/topologies/double-ring-nested.xenial.virl @@ -389,9 +380,7 @@ Configuration of the TG VMs is defined in file **TG node port configuration** Port configuration of TG is defined in topology file that is generated per VIRL -simulation based on the definition stored in file - - /csit/resources/tools/virl/topologies/double-ring-nested.xenial.yaml +simulation based on the definition stored in `VIRL topologies directory`_. Example of TG node configuration::: diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index 3daff961b4..0ee3e82f59 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -31,8 +31,8 @@ Changes in CSIT |release| on CSIT wiki page `Design_Optimizations `_. - - For more detail see the CSIT Framework Design section in this - report; + - For more detail see the :ref:`CSIT Framework Design ` section + in this report; #. Changes to CSIT driver for TRex Traffic Generator: @@ -155,27 +155,27 @@ Here is the list of known issues in CSIT |release| for VPP performance tests: +---+-------------------------------------------------+------------+-----------------------------------------------------------------+ | # | Issue | Jira ID | Description | +---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 1 | Security-groups acl-plugin scale tests failure | CSIT-xxx | VPP with 2 worker threads crashes during security-groups | +| 1 | Security-groups acl-plugin scale tests failure | CSIT-731 | VPP with 2 worker threads crashes during security-groups | | | with stateful acls if VPP with 2 worker threads | VPP-912 | iaclNsf and oaclNsf tests with 100k flows. | +---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| 2 | VPP fails memif tests in 4 worker 2 core setup | CSIT-xxx | VPP with 4 worker threads running on 2 physical cores crashes | -| | | VPP-xxx | during memif tests. Initial debugging points to DPDK code | +| 2 | VPP fails memif tests in 4 worker 2 core setup | CSIT-732 | VPP with 4 worker threads running on 2 physical cores crashes | +| | | VPP-920 | during memif tests. Initial debugging points to DPDK code | +---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| X | NDR discovery test failures 1518B frame size | VPP-663 | VPP reporting errors: dpdk-input Rx ip checksum errors. | +| 3 | NDR discovery test failures 1518B frame size | VPP-663 | VPP reporting errors: dpdk-input Rx ip checksum errors. | | | for ip4scale200k, ip4scale2m scale IPv4 routed- | | Observed frequency: all test runs. | | | forwarding tests. ip4scale20k tests are fine. | | | +---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| X | Vic1385 and Vic1227 low performance. | VPP-664 | Low NDR performance. | +| 4 | Vic1385 and Vic1227 low performance. | VPP-664 | Low NDR performance. | | | | | | +---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| X | Sporadic NDR discovery test failures on x520. | CSIT-750 | Suspected issue with HW settings (BIOS, FW) in LF | +| 5 | Sporadic NDR discovery test failures on x520. | CSIT-750 | Suspected issue with HW settings (BIOS, FW) in LF | | | | | infrastructure. Issue can't be replicated outside LF. | +---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| X | VPP in 2t2c setups - large variation | CSIT-568 | Suspected NIC firmware or DPDK driver issue affecting NDR | +| 6 | VPP in 2t2c setups - large variation | CSIT-568 | Suspected NIC firmware or DPDK driver issue affecting NDR | | | of discovered NDR throughput values across | | throughput. Applies to XL710 and X710 NICs, x520 NICs are fine. | | | multiple test runs with xl710 and x710 NICs. | | | +---+-------------------------------------------------+------------+-----------------------------------------------------------------+ -| X | Lower than expected NDR and PDR throughput with | CSIT-569 | Suspected NIC firmware or DPDK driver issue affecting NDR and | +| 7 | Lower than expected NDR and PDR throughput with | CSIT-569 | Suspected NIC firmware or DPDK driver issue affecting NDR and | | | xl710 and x710 NICs, compared to x520 NICs. | | PDR throughput. Applies to XL710 and X710 NICs. | +---+-------------------------------------------------+------------+-----------------------------------------------------------------+ diff --git a/docs/report/vpp_performance_tests/overview.rst b/docs/report/vpp_performance_tests/overview.rst index 98a4d06681..5cf7b1d355 100644 --- a/docs/report/vpp_performance_tests/overview.rst +++ b/docs/report/vpp_performance_tests/overview.rst @@ -361,8 +361,9 @@ TRex is installed and run on the TG compute node. The typical procedure is: $ sh -c 'cd /opt/trex-core-2.25/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --iom 0 > /dev/null 2>&1 &' > /dev/null -- There are traffic streams dynamically prepared for each test. The traffic - is sent and the statistics obtained using trex_stl_lib.api.STLClient. +- There are traffic streams dynamically prepared for each test, based on traffic + profiles. The traffic is sent and the statistics obtained using + trex_stl_lib.api.STLClient. **Measuring packet loss** diff --git a/docs/report/vpp_performance_tests/test_environment.rst b/docs/report/vpp_performance_tests/test_environment.rst index 9392259a97..20ea77f31a 100644 --- a/docs/report/vpp_performance_tests/test_environment.rst +++ b/docs/report/vpp_performance_tests/test_environment.rst @@ -251,7 +251,8 @@ Below a subset of the running configuration: - **isolcpus=-** used for all cpu cores apart from first core of each socket used for running VPP worker threads and Qemu/LXC - processes https://www.kernel.org/doc/Documentation/kernel-parameters.txt + processes + https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt - **intel_pstate=disable** - [X86] Do not enable intel_pstate as the default scaling driver for the supported processors. Intel P-State driver decide what P-state (CPU core power state) to use based on requesting policy from the @@ -269,7 +270,7 @@ Below a subset of the running configuration: - **rcu_nocbs** - [KNL] In kernels built with CONFIG_RCU_NOCB_CPU=y, set the specified list of CPUs to be no-callback CPUs, that never queue RCU callbacks (read-copy update). - https://www.kernel.org/doc/Documentation/kernel-parameters.txt + https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt **Applied command line boot parameters:** diff --git a/resources/tools/report_gen/conf.py b/resources/tools/report_gen/conf.py index b16cc4e42f..89dd45824f 100644 --- a/resources/tools/report_gen/conf.py +++ b/resources/tools/report_gen/conf.py @@ -69,8 +69,11 @@ rst_epilog = """ .. _tag documentation rst file: https://git.fd.io/csit/tree/docs/tag_documentation.rst?h=rls1707 .. _TRex intallation: https://git.fd.io/csit/tree/resources/tools/trex/trex_installer.sh?h=rls1707 .. _TRex driver: https://git.fd.io/csit/tree/resources/tools/trex/trex_stateless_profile.py?h=rls1707 -.. _CSIT Honeycomb Functional Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.vpp.func.html -.. _CSIT Honeycomb Performance Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.vpp.perf.html +.. _VIRL topologies directory: https://git.fd.io/csit/tree/resources/tools/virl/topologies/?h=rls1707 +.. _VIRL images lists: https://git.fd.io/csit/tree/resources/tools/disk-image-builder/ubuntu/lists/?h=rls1707 +.. _VIRL nested: https://git.fd.io/csit/tree/resources/tools/disk-image-builder/nested/?h=rls1707 +.. _CSIT Honeycomb Functional Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.vpp.func.honeycomb.html +.. _CSIT Honeycomb Performance Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.vpp.perf.honeycomb.html .. _CSIT DPDK Performance Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.dpdk.perf.html .. _CSIT VPP Functional Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.vpp.func.html .. _CSIT VPP Performance Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.vpp.perf.html @@ -83,7 +86,6 @@ rst_epilog = """ .. _FD.io test executor honeycomb performance jobs: https://jenkins.fd.io/view/hc2vpp/job/hc2vpp-csit-perf-master-ubuntu1604/lastSuccessfulBuild .. _FD.io test executor NSH_SFC functional jobs: https://jenkins.fd.io/view/csit/job/csit-nsh_sfc-verify-func-1707-ubuntu1604-virl/lastSuccessfulBuild .. _FD.io VPP compile job: https://jenkins.fd.io/view/vpp/job/vpp-merge-1707-ubuntu1604/ -.. _FD.io VPP compile job: https://jenkins.fd.io/view/vpp/job/vpp-merge-1707-ubuntu1604/ .. _CSIT Testbed Setup: https://git.fd.io/csit/tree/resources/tools/testbed-setup/README.md?h=rls1707 """ diff --git a/resources/tools/report_gen/run_report.cfg b/resources/tools/report_gen/run_report.cfg index 57b2be0f94..5cd6b37c58 100644 --- a/resources/tools/report_gen/run_report.cfg +++ b/resources/tools/report_gen/run_report.cfg @@ -54,8 +54,8 @@ JOB[PERF,HC]=hc2vpp-csit-perf-master-ubuntu1604 JOB[PERF,HC,BLD]="8 9" JOB[FUNC,HC]=hc2vpp-csit-integration-1707-ubuntu1604 JOB[FUNC,HC,BLD]=lastSuccessfulBuild -JOB[FUNC,NSH]=hc2vpp-csit-integration-1707-ubuntu1604 -JOB[FUNC,NSH,BLD]=lastSuccessfulBuild +JOB[FUNC,NSH]=csit-nsh_sfc-verify-func-1707-ubuntu1604-virl +JOB[FUNC,NSH,BLD]=2 JOB[1704,PERF,VPP]=csit-vpp-perf-1704-all JOB[1704,VPP,BLD]="6 7 8 9 10 12 14 15 16 17"