From: Peter Mikus Date: Mon, 17 Jul 2017 12:14:23 +0000 (+0200) Subject: CSIT-618 CSIT rls1707 Report - Update II X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=commitdiff_plain;h=4457640600a2b875da17e97dba2dcab8a411eb95;hp=997a1485635dd74aaf090ca5de608b6fdc25b74b CSIT-618 CSIT rls1707 Report - Update II Change-Id: I50a210930c0571ca910ead1724de7848c0911646 Signed-off-by: Peter Mikus --- diff --git a/docs/report/detailed_test_results/index.rst b/docs/report/detailed_test_results/index.rst index a609ef40d4..9a986bcd65 100644 --- a/docs/report/detailed_test_results/index.rst +++ b/docs/report/detailed_test_results/index.rst @@ -7,4 +7,5 @@ Detailed Test Results dpdk_performance_results/index vpp_functional_results/index honeycomb_functional_results/index + nshsfc_functional_results/index vpp_unittest_results/index diff --git a/docs/report/detailed_test_results/nshsfc_functional_results/index.rst b/docs/report/detailed_test_results/nshsfc_functional_results/index.rst new file mode 100644 index 0000000000..11c168a92c --- /dev/null +++ b/docs/report/detailed_test_results/nshsfc_functional_results/index.rst @@ -0,0 +1,14 @@ +NSH SFC Functional Results +========================== + +.. note:: + + Data sources for reported test results: i) FD.io test executor jobs + `###JOB### + <###LINK###>`_ + , ii) archived FD.io jobs test result `output files + <../../_static/archive/>`_. + +.. toctree:: + + nshsfc_functional_results diff --git a/docs/report/introduction/overview.rst b/docs/report/introduction/overview.rst index 7954686ad3..4aac352d0c 100644 --- a/docs/report/introduction/overview.rst +++ b/docs/report/introduction/overview.rst @@ -1,82 +1,82 @@ -Overview -======== - -This is the **F**\ast **D**\ata **I**/**O** Project (**FD.io**) **C**\ontinuous -**S**\ystem **I**\ntegration and **T**\esting (**CSIT**) project report for CSIT -|release| system testing of |vpp-release|. - -The report describes CSIT functional and performance tests and their -continuous execution delivered in CSIT |release|. A high-level overview is -provided for each CSIT test environment running in Linux Foundation (LF) FD.io -Continuous Performance Labs. This is followed by summary of all executed tests -against the |vpp-release| and associated FD.io projects and sub-systems -(HoneyComb, DPDK), CSIT |release| release notes, result highlights and known -issues discovered in CSIT. More detailed description of each environment, -pointers to CSIT test code documentation and detailed test resuls with links to -the source data files are also provided. - -CSIT |release| report contains following main sections and sub-sections: - -#. **Introduction** - general introduction to CSIT project; *Overview* - - this section; *CSIT Test Naming* - CSIT general naming convention for test - suites and test cases, important to recognize the high-level test content - and interpret reported results; *General Notes* - general notes related to - this report. - -#. **VPP Performance Tests** - VPP performance tests executed in physical - FD.io testbeds; *Overview* - tested topologies, test coverage and naming - specifics, methodology for multi-core, packet throughput and latency, and - KVM VM vhost tests; *CSIT Release Notes* - changes in CSIT |release|, added - tests, performance changes, environment or methodology changes, known CSIT - issues; *Packet Throughput Graphs* and *Packet Latency - Graphs* - plotted NDR, PDR throughput and latency results from multiple - test job executions; *Test Environment* - environment description; - *Documentation* - CSIT source code documentation for VPP performance tests. - -#. **DPDK Performance Tests** - DPDK performance tests executed in - physical FD.io testbeds; *Overview* - tested topologies, test coverage; - *CSIT Release Notes* - changes in CSIT |release|, any known CSIT issues; - *Packet Throughput Graphs* and *Packet Latency Graphs* - - plotted NDR, PDR throughput and latency results from multiple test job - executions; *Test Environment* - environment description; *Documentation* - - CSIT source code documentation for DPDK performance tests. - -#. **VPP Functional Tests** - VPP functional tests executed in virtual - FD.io testbeds; *Overview* - tested virtual topologies, test coverage and - naming specifics; *CSIT Release Notes* - changes in CSIT |release|, added - tests, environment or methodology changes, known CSIT issues, tests to be - added; *Test Environment* - environment description ; *Documentation* - - source code documentation for VPP functional tests. - -#. **HoneyComb Functional Tests** - HoneyComb functional tests executed in - virtual FD.io testbeds; *Overview* - tested virtual topologies, test - coverage and naming specifics; *CSIT Release Notes* - changes in CSIT - |release|, added tests, environment or methodology changes, known CSIT issues; - *Test Environment* - environment description ; - *Documentation* - source code documentation for Honeycomb functional tests. - -#. **VPP Unit Tests** - refers to VPP functional unit tests executed as - part of vpp make test verify option within the FD.io VPP project; listed in - this report to give a more complete view about executed VPP functional tests; - *Overview* - short overview of unit test framework and executed tests; - *Documentation* - source code documentation of VPP unit tests. - -#. **NSH_SFC Functional Tests** - NSH_SFC functional tests executed in - virtual FD.io testbeds; *Overview* - tested virtual topologies, test - coverage and naming specifics; *CSIT Release Notes* - changes in CSIT - |release|, added tests, environment or methodology changes, known CSIT issues; - *Test Environment* - environment description ; - *Documentation* - source code documentation for NSH_SFC functional tests. - -#. **Detailed Test Results** - auto-generated results from CSIT jobs - executions using CSIT Robot Framework output files as source data; *VPP - Performance Results*, *DPDK Performance Results*, *VPP Functional - Results*, *HoneyComb Functional Results*, *VPPtest Functional Results*. - -#. **Test Configuration** - auto-generated DUT configuration data from CSIT jobs - executions using CSIT Robot Framework output files as source data; *VPP - Performance Test Configs*, *VPP Functional Test Configs*. - -#. **Test Operational Data** - auto-generated DUT operational data from CSIT jobs - executions using CSIT Robot Framework output files as source data; *VPP - Performance Operational Data*. +Overview +======== + +This is the **F**\ast **D**\ata **I**/**O** Project (**FD.io**) **C**\ontinuous +**S**\ystem **I**\ntegration and **T**\esting (**CSIT**) project report for CSIT +|release| system testing of |vpp-release|. + +The report describes CSIT functional and performance tests and their +continuous execution delivered in CSIT |release|. A high-level overview is +provided for each CSIT test environment running in Linux Foundation (LF) FD.io +Continuous Performance Labs. This is followed by summary of all executed tests +against the |vpp-release| and associated FD.io projects and sub-systems +(HoneyComb, DPDK, NSH_SFC), CSIT |release| release notes, result highlights and +known issues discovered in CSIT. More detailed description of each environment, +pointers to CSIT test code documentation and detailed test resuls with links to +the source data files are also provided. + +CSIT |release| report contains following main sections and sub-sections: + +#. **Introduction** - general introduction to CSIT project; *Overview* - + this section; *CSIT Test Naming* - CSIT general naming convention for test + suites and test cases, important to recognize the high-level test content + and interpret reported results; *General Notes* - general notes related to + this report. + +#. **VPP Performance Tests** - VPP performance tests executed in physical + FD.io testbeds; *Overview* - tested topologies, test coverage and naming + specifics, methodology for multi-core, packet throughput and latency, and + KVM VM vhost tests; *CSIT Release Notes* - changes in CSIT |release|, added + tests, performance changes, environment or methodology changes, known CSIT + issues; *Packet Throughput Graphs* and *Packet Latency + Graphs* - plotted NDR, PDR throughput and latency results from multiple + test job executions; *Test Environment* - environment description; + *Documentation* - CSIT source code documentation for VPP performance tests. + +#. **DPDK Performance Tests** - DPDK performance tests executed in + physical FD.io testbeds; *Overview* - tested topologies, test coverage; + *CSIT Release Notes* - changes in CSIT |release|, any known CSIT issues; + *Packet Throughput Graphs* and *Packet Latency Graphs* + - plotted NDR, PDR throughput and latency results from multiple test job + executions; *Test Environment* - environment description; *Documentation* - + CSIT source code documentation for DPDK performance tests. + +#. **VPP Functional Tests** - VPP functional tests executed in virtual + FD.io testbeds; *Overview* - tested virtual topologies, test coverage and + naming specifics; *CSIT Release Notes* - changes in CSIT |release|, added + tests, environment or methodology changes, known CSIT issues, tests to be + added; *Test Environment* - environment description ; *Documentation* - + source code documentation for VPP functional tests. + +#. **HoneyComb Functional Tests** - HoneyComb functional tests executed in + virtual FD.io testbeds; *Overview* - tested virtual topologies, test + coverage and naming specifics; *CSIT Release Notes* - changes in CSIT + |release|, added tests, environment or methodology changes, known CSIT issues; + *Test Environment* - environment description ; + *Documentation* - source code documentation for Honeycomb functional tests. + +#. **VPP Unit Tests** - refers to VPP functional unit tests executed as + part of vpp make test verify option within the FD.io VPP project; listed in + this report to give a more complete view about executed VPP functional tests; + *Overview* - short overview of unit test framework and executed tests; + *Documentation* - source code documentation of VPP unit tests. + +#. **NSH_SFC Functional Tests** - NSH_SFC functional tests executed in + virtual FD.io testbeds; *Overview* - tested virtual topologies, test + coverage and naming specifics; *CSIT Release Notes* - changes in CSIT + |release|, added tests, environment or methodology changes, known CSIT issues; + *Test Environment* - environment description ; + *Documentation* - source code documentation for NSH_SFC functional tests. + +#. **Detailed Test Results** - auto-generated results from CSIT jobs + executions using CSIT Robot Framework output files as source data; *VPP + Performance Results*, *DPDK Performance Results*, *VPP Functional + Results*, *HoneyComb Functional Results*, *VPPtest Functional Results*. + +#. **Test Configuration** - auto-generated DUT configuration data from CSIT jobs + executions using CSIT Robot Framework output files as source data; *VPP + Performance Test Configs*, *VPP Functional Test Configs*. + +#. **Test Operational Data** - auto-generated DUT operational data from CSIT jobs + executions using CSIT Robot Framework output files as source data; *VPP + Performance Operational Data*. diff --git a/resources/tools/report_gen/run_report.sh b/resources/tools/report_gen/run_report.sh index 494ec5650a..ece60368da 100755 --- a/resources/tools/report_gen/run_report.sh +++ b/resources/tools/report_gen/run_report.sh @@ -21,17 +21,19 @@ PLOT_VPP_SOURCE_DIR=${WORKING_DIR}/vpp_plot PLOT_DPDK_SOURCE_DIR=${WORKING_DIR}/dpdk_plot DTR_SOURCE_DIR=${SOURCE_DIR}/detailed_test_results -DTR_PERF_SOURCE_DIR=${DTR_SOURCE_DIR}/vpp_performance_results DTR_DPDK_SOURCE_DIR=${DTR_SOURCE_DIR}/dpdk_performance_results -DTR_FUNC_SOURCE_DIR=${DTR_SOURCE_DIR}/vpp_functional_results -DTR_HONEYCOMB_SOURCE_DIR=${DTR_SOURCE_DIR}/honeycomb_functional_results +DTR_VPP_PERF_SOURCE_DIR=${DTR_SOURCE_DIR}/vpp_performance_results +DTR_VPP_FUNC_SOURCE_DIR=${DTR_SOURCE_DIR}/vpp_functional_results +DTR_HC_PERF_SOURCE_DIR=${DTR_SOURCE_DIR}/honeycomb_performance_results +DTR_HC_FUNC_SOURCE_DIR=${DTR_SOURCE_DIR}/honeycomb_functional_results +DTR_NSHSFC_FUNC_SOURCE_DIR=${DTR_SOURCE_DIR}/nshsfc_functional_results DTC_SOURCE_DIR=${SOURCE_DIR}/test_configuration -DTC_PERF_SOURCE_DIR=${DTC_SOURCE_DIR}/vpp_performance_configuration -DTC_FUNC_SOURCE_DIR=${DTC_SOURCE_DIR}/vpp_functional_configuration +DTC_VPP_PERF_SOURCE_DIR=${DTC_SOURCE_DIR}/vpp_performance_configuration +DTC_VPP_FUNC_SOURCE_DIR=${DTC_SOURCE_DIR}/vpp_functional_configuration DTO_SOURCE_DIR=${SOURCE_DIR}/test_operational_data -DTO_PERF_SOURCE_OPER_DIR=${DTO_SOURCE_DIR}/vpp_performance_operational_data +DTO_VPP_PERF_SOURCE_OPER_DIR=${DTO_SOURCE_DIR}/vpp_performance_operational_data # Jenkins links CSIT_JEN_URL='https://jenkins.fd.io/view/csit/job' @@ -90,34 +92,34 @@ done unzip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${JEN_FBUILD}.zip -d ${WORKING_DIR}/ python run_robot_data.py -i ${WORKING_DIR}/robot-plugin/output.xml \ - --output ${DTR_PERF_SOURCE_DIR}/vpp_performance_results.rst \ + --output ${DTR_VPP_PERF_SOURCE_DIR}/vpp_performance_results.rst \ --formatting rst --start 4 --level 2 python run_robot_teardown_data.py -i ${WORKING_DIR}/robot-plugin/output.xml \ - --output ${DTC_PERF_SOURCE_DIR}/vpp_performance_configuration.rst \ + --output ${DTC_VPP_PERF_SOURCE_DIR}/vpp_performance_configuration.rst \ --data "VAT_H" -f "rst" --start 4 --level 2 python run_robot_teardown_data.py -i ${WORKING_DIR}/robot-plugin/output.xml \ - -o ${DTO_PERF_SOURCE_OPER_DIR}/vpp_performance_operational_data.rst \ + -o ${DTO_VPP_PERF_SOURCE_OPER_DIR}/vpp_performance_operational_data.rst \ --data "SH_RUN" -f "rst" --start 4 --level 2 if [[ ${DEBUG} -eq 0 ]] ; then sed -i -e "s@###JOB###@${JEN_JOB}\/${JEN_FBUILD}@g" \ - ${DTR_PERF_SOURCE_DIR}/index.rst + ${DTR_VPP_PERF_SOURCE_DIR}/index.rst sed -i -e "s@###LINK###@${CSIT_JEN_URL}\/${JEN_JOB}\/${JEN_FBUILD}@g" \ - ${DTR_PERF_SOURCE_DIR}/index.rst + ${DTR_VPP_PERF_SOURCE_DIR}/index.rst sed -i -e "s@###JOB###@${JEN_JOB}\/${JEN_FBUILD}@g" \ - ${DTC_PERF_SOURCE_DIR}/index.rst + ${DTC_VPP_PERF_SOURCE_DIR}/index.rst sed -i -e "s@###LINK###@${CSIT_JEN_URL}\/${JEN_JOB}\/${JEN_FBUILD}@g" \ - ${DTC_PERF_SOURCE_DIR}/index.rst + ${DTC_VPP_PERF_SOURCE_DIR}/index.rst sed -i -e "s@###JOB###@${JEN_JOB}\/${JEN_FBUILD}@g" \ - ${DTO_PERF_SOURCE_OPER_DIR}/index.rst + ${DTO_VPP_PERF_SOURCE_OPER_DIR}/index.rst sed -i -e "s@###LINK###@${CSIT_JEN_URL}\/${JEN_JOB}\/${JEN_FBUILD}@g" \ - ${DTO_PERF_SOURCE_OPER_DIR}/index.rst + ${DTO_VPP_PERF_SOURCE_OPER_DIR}/index.rst fi ### DPDK PERFORMANCE SOURCE DATA JEN_JOB='csit-dpdk-perf-1707-all' -JEN_BUILD=(1 2 3 4) +JEN_BUILD=(1 2 3 4 5 6 7 8 9 10) for i in "${JEN_BUILD[@]}"; do curl --fail -fs ${CSIT_JEN_URL}/${JEN_JOB}/${i}/robot/report/output_perf_data.xml \ @@ -154,21 +156,21 @@ else -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${JEN_BUILD}.zip sed -i -e "s@###JOB###@${JEN_JOB}\/${JEN_BUILD}@g" \ - ${DTR_FUNC_SOURCE_DIR}/index.rst + ${DTR_VPP_FUNC_SOURCE_DIR}/index.rst sed -i -e "s@###LINK###@${CSIT_JEN_URL}\/${JEN_JOB}\/${JEN_BUILD}@g" \ - ${DTR_FUNC_SOURCE_DIR}/index.rst + ${DTR_VPP_FUNC_SOURCE_DIR}/index.rst sed -i -e "s@###JOB###@${JEN_JOB}\/${JEN_BUILD}@g" \ - ${DTC_FUNC_SOURCE_DIR}/index.rst + ${DTC_VPP_FUNC_SOURCE_DIR}/index.rst sed -i -e "s@###LINK###@${CSIT_JEN_URL}\/${JEN_JOB}\/${JEN_BUILD}@g" \ - ${DTC_FUNC_SOURCE_DIR}/index.rst + ${DTC_VPP_FUNC_SOURCE_DIR}/index.rst fi unzip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${JEN_BUILD}.zip -d ${WORKING_DIR}/ python run_robot_data.py -i ${WORKING_DIR}/robot-plugin/output.xml \ - --output ${DTR_FUNC_SOURCE_DIR}/vpp_functional_results.rst \ + --output ${DTR_VPP_FUNC_SOURCE_DIR}/vpp_functional_results.rst \ --formatting rst --start 5 --level 2 python run_robot_teardown_data.py -i ${WORKING_DIR}/robot-plugin/output.xml \ - --output ${DTC_FUNC_SOURCE_DIR}/vpp_functional_configuration.rst \ + --output ${DTC_VPP_FUNC_SOURCE_DIR}/vpp_functional_configuration.rst \ --data "VAT_H" -f "rst" --start 5 --level 2 ### HONEYCOMB SOURCE DATA @@ -184,16 +186,41 @@ else -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${JEN_BUILD}.zip sed -i -e "s@###JOB###@${JEN_JOB}\/${JEN_BUILD}@g" \ - ${DTR_HONEYCOMB_SOURCE_DIR}/index.rst + ${DTR_HC_FUNC_SOURCE_DIR}/index.rst sed -i -e "s@###LINK###@${HC_JEN_URL}\/${JEN_JOB}\/${JEN_BUILD}@g" \ - ${DTR_HONEYCOMB_SOURCE_DIR}/index.rst + ${DTR_HC_FUNC_SOURCE_DIR}/index.rst fi unzip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${JEN_BUILD}.zip -d ${WORKING_DIR}/ python run_robot_data.py -i ${WORKING_DIR}/robot-plugin/output.xml \ - --output ${DTR_HONEYCOMB_SOURCE_DIR}/honeycomb_functional_results.rst \ + --output ${DTR_HC_FUNC_SOURCE_DIR}/honeycomb_functional_results.rst \ --formatting rst --start 5 --level 2 +### NSH_SFC SOURCE DATA + +JEN_JOB='csit-nsh_sfc-verify-func-1707-ubuntu1604-virl' +JEN_BUILD='lastSuccessfulBuild' + +if [[ ${DEBUG} -eq 1 ]] ; +then + cp ./${JEN_JOB}-${JEN_BUILD}.zip ${STATIC_DIR_ARCH}/${JEN_JOB}-${JEN_BUILD}.zip +else + curl -fs ${CSIT_JEN_URL}/${JEN_JOB}/${JEN_BUILD}/robot/report/\*zip\*/robot-plugin.zip \ + -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${JEN_BUILD}.zip + + sed -i -e "s@###JOB###@${JEN_JOB}\/${JEN_BUILD}@g" \ + ${DTR_NSHSFC_FUNC_SOURCE_DIR}/index.rst + sed -i -e "s@###LINK###@${HC_JEN_URL}\/${JEN_JOB}\/${JEN_BUILD}@g" \ + ${DTR_NSHSFC_FUNC_SOURCE_DIR}/index.rst +fi + +unzip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${JEN_BUILD}.zip -d ${WORKING_DIR}/ +python run_robot_data.py -i ${WORKING_DIR}/robot-plugin/output.xml \ + --output ${DTR_NSHSFC_FUNC_SOURCE_DIR}/nshsfc_functional_results.rst \ + --formatting rst --start 5 --level 2 + + + # Delete temporary json files find ${SOURCE_DIR} -name "*.json" -type f -delete