From 61407a72649b7b355472350c2186aa36bf8182e7 Mon Sep 17 00:00:00 2001 From: Peter Mikus Date: Mon, 24 Jul 2017 14:22:02 +0200 Subject: [PATCH] CSIT-618 Release report update VII Change-Id: Ide7771b59cc3ee7619502a6805c8401528a8c874 Signed-off-by: Peter Mikus --- .../vpp_functional_tests/csit_release_notes.rst | 2 +- .../packet_latency_graphs/l2.rst | 20 ++++-- .../packet_throughput_graphs/ipsec.rst | 5 -- .../packet_throughput_graphs/l2.rst | 44 ++++++++---- resources/tools/report_gen/run_report.sh | 78 ++++++++++++++++------ 5 files changed, 105 insertions(+), 44 deletions(-) diff --git a/docs/report/vpp_functional_tests/csit_release_notes.rst b/docs/report/vpp_functional_tests/csit_release_notes.rst index fd71d78f40..08fe87053d 100644 --- a/docs/report/vpp_functional_tests/csit_release_notes.rst +++ b/docs/report/vpp_functional_tests/csit_release_notes.rst @@ -21,7 +21,7 @@ Changes in CSIT |release| - Complete CSIT framework code revision and optimizations as descried on CSIT wiki page - _`Design_Optimizations `_; + `Design_Optimizations `_; - For more detail see the CSIT Framework Design section in this report; diff --git a/docs/report/vpp_performance_tests/packet_latency_graphs/l2.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/l2.rst index c46fd03a38..d6e76a7c88 100644 --- a/docs/report/vpp_performance_tests/packet_latency_graphs/l2.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/l2.rst @@ -11,9 +11,15 @@ VPP packet latency in 1t1c setup (1thread, 1core) is presented in the graph belo .. raw:: html - + -*Figure 1. VPP 1thread 1core - packet latency for Phy-to-Phy L2 Ethernet Switching.* +*Figure 1a. VPP 1thread 1core - packet latency for Phy-to-Phy L2 Ethernet Switching.* + +.. raw:: html + + + +*Figure 1b. VPP 1thread 1core - packet latency for Phy-to-Phy L2 Ethernet Switching.* CSIT source code for the test cases used for above plots can be found in CSIT git repository: @@ -25,9 +31,15 @@ VPP packet latency in 2t2c setup (2thread, 2core) is presented in the graph belo .. raw:: html - + + +*Figure 2a. VPP 2threads 2cores - packet latency for Phy-to-Phy L2 Ethernet Switching.* + +.. raw:: html + + -*Figure 2. VPP 2threads 2cores - packet latency for Phy-to-Phy L2 Ethernet Switching.* +*Figure 2b. VPP 2threads 2cores - packet latency for Phy-to-Phy L2 Ethernet Switching.* CSIT source code for the test cases used for above plots can be found in CSIT git repository: diff --git a/docs/report/vpp_performance_tests/packet_throughput_graphs/ipsec.rst b/docs/report/vpp_performance_tests/packet_throughput_graphs/ipsec.rst index ebd787547a..eba3a40e87 100644 --- a/docs/report/vpp_performance_tests/packet_throughput_graphs/ipsec.rst +++ b/docs/report/vpp_performance_tests/packet_throughput_graphs/ipsec.rst @@ -10,11 +10,6 @@ Performance is reported for VPP running in multiple configurations of VPP worker thread(s), a.k.a. VPP data plane thread(s), and their physical CPU core(s) placement. -Note: only 1,000 scale IPSec tunnel test cases are plotted. Single IPSec -tunnel test cases are listed in regex outputs, but marked {NOT PLOTTED} -due to the current limitation of auto-plotting scripts. For all test -result data please refere to **Detailed Test Results** section. - NDR Throughput ~~~~~~~~~~~~~~ diff --git a/docs/report/vpp_performance_tests/packet_throughput_graphs/l2.rst b/docs/report/vpp_performance_tests/packet_throughput_graphs/l2.rst index 37ac157f90..4acc65a003 100644 --- a/docs/report/vpp_performance_tests/packet_throughput_graphs/l2.rst +++ b/docs/report/vpp_performance_tests/packet_throughput_graphs/l2.rst @@ -15,9 +15,16 @@ in the graph below. .. raw:: html - + -*Figure 1. VPP 1thread 1core - NDR Throughput for Phy-to-Phy L2 Ethernet +*Figure 1a. VPP 1thread 1core - NDR Throughput for Phy-to-Phy L2 Ethernet +Switching.* + +.. raw:: html + + + +*Figure 1b. VPP 1thread 1core - NDR Throughput for Phy-to-Phy L2 Ethernet Switching.* CSIT source code for the test cases used for above plots can be found in CSIT @@ -31,9 +38,16 @@ in the graph below. .. raw:: html - + + +*Figure 2a. VPP 2threads 2cores - NDR Throughput for Phy-to-Phy L2 Ethernet +Switching.* + +.. raw:: html + + -*Figure 2. VPP 2threads 2cores - NDR Throughput for Phy-to-Phy L2 Ethernet +*Figure 2b. VPP 2threads 2cores - NDR Throughput for Phy-to-Phy L2 Ethernet Switching.* CSIT source code for the test cases used for above plots can be found in CSIT @@ -50,9 +64,16 @@ in the graph below. PDR measured for 0.5% packet loss ratio. .. raw:: html - + -*Figure 3. VPP 1thread 1core - PDR Throughput for Phy-to-Phy L2 Ethernet +*Figure 3a. VPP 1thread 1core - PDR Throughput for Phy-to-Phy L2 Ethernet +Switching.* + +.. raw:: html + + + +*Figure 3b. VPP 1thread 1core - PDR Throughput for Phy-to-Phy L2 Ethernet Switching.* CSIT source code for the test cases used for above plots can be found in CSIT @@ -66,13 +87,12 @@ in the graph below. PDR measured for 0.5% packet loss ratio. .. raw:: html - + -*Figure 4. VPP 2thread 2core - PDR Throughput for Phy-to-Phy L2 Ethernet +*Figure 4a. VPP 2thread 2core - PDR Throughput for Phy-to-Phy L2 Ethernet Switching.* -CSIT source code for the test cases used for above plots can be found in CSIT -git repository: +.. raw:: html + + -.. program-output:: cd ../../ && set -x && cd tests/vpp/perf/l2 && grep -E "64B-2t2c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-(iacl50-state(ful|less)-flows10k.*|oacl50-state(ful|less)-flows10k.*|eth.*)*pdrdisc" * - :shell: diff --git a/resources/tools/report_gen/run_report.sh b/resources/tools/report_gen/run_report.sh index 9076526eda..2737373496 100755 --- a/resources/tools/report_gen/run_report.sh +++ b/resources/tools/report_gen/run_report.sh @@ -3,7 +3,7 @@ set -x # Build locally without jenkins integrations -DEBUG=0 +DEBUG=1 # Script directory SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" @@ -55,8 +55,8 @@ blds=${JOB[PERF,VPP,BLD]} for i in ${blds[@]}; do curl --silent ${URL[JENKINS,CSIT]}/${JOB[PERF,VPP]}/${i}/robot/report/output_perf_data.xml \ --output ${DIR[PLOT,VPP]}/${JOB[PERF,VPP]}-${i}.xml - curl --silent ${URL[JENKINS,CSIT]}/${JOB[PERF,VPP]}/${i}/robot/report/output_perf_data.json \ - --output ${DIR[PLOT,VPP]}/${JOB[PERF,VPP]}-${i}.json + #curl --silent ${URL[JENKINS,CSIT]}/${JOB[PERF,VPP]}/${i}/robot/report/output_perf_data.json \ + # --output ${DIR[PLOT,VPP]}/${JOB[PERF,VPP]}-${i}.json if [[ ${DEBUG} -eq 0 ]] ; then curl --fail --silent ${URL[JENKINS,CSIT]}/${JOB[PERF,VPP]}/${i}/robot/report/\*zip\*/robot-plugin.zip \ @@ -81,8 +81,8 @@ blds=${JOB[PERF,DPDK,BLD]} for i in ${blds[@]}; do curl --silent ${URL[JENKINS,CSIT]}/${JOB[PERF,DPDK]}/${i}/robot/report/output_perf_data.xml \ --output ${DIR[PLOT,DPDK]}/${JOB[PERF,DPDK]}-${i}.xml - curl --silent ${URL[JENKINS,CSIT]}/${JOB[PERF,DPDK]}/${i}/robot/report/output_perf_data.json \ - --output ${DIR[PLOT,DPDK]}/${JOB[PERF,DPDK]}-${i}.json + #curl --silent ${URL[JENKINS,CSIT]}/${JOB[PERF,DPDK]}/${i}/robot/report/output_perf_data.json \ + # --output ${DIR[PLOT,DPDK]}/${JOB[PERF,DPDK]}-${i}.json if [[ ${DEBUG} -eq 0 ]] ; then curl --fail --silent ${URL[JENKINS,CSIT]}/${JOB[PERF,DPDK]}/${i}/robot/report/\*zip\*/robot-plugin.zip \ @@ -196,9 +196,12 @@ then fi # Generate tables for performance improvements -./run_improvments_tables.py \ - --input ${DIR[DTR,PERF,VPP,IMPRV]} \ - --output ${DIR[DTR,PERF,VPP,IMPRV]} +if [[ ${DEBUG} -eq 0 ]] ; +then + ./run_improvments_tables.py \ + --input ${DIR[DTR,PERF,VPP,IMPRV]} \ + --output ${DIR[DTR,PERF,VPP,IMPRV]} +fi # Delete temporary json files find ${DIR[RST]} -name "*.json" -type f -delete @@ -228,25 +231,45 @@ _EOF # Plot packets per second -# VPP L2 +# VPP L2 sel1 python run_plot.py --input ${DIR[PLOT,VPP]} \ - --output ${DIR[STATIC,VPP]}/64B-1t1c-l2-ndrdisc \ + --output ${DIR[STATIC,VPP]}/64B-1t1c-l2-sel1-ndrdisc \ --title "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc" \ - --xpath '//*[@framesize="64B" and (contains(@tags,"BASE") or (contains(@tags,"FEATURE") and contains(@tags,"ACL50") and contains(@tags,"10k_FLOWS"))) and contains(@tags,"NDRDISC") and contains(@tags,"1T1C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' + --xpath '//*[@framesize="64B" and contains(@tags,"BASE") and contains(@tags,"NDRDISC") and contains(@tags,"1T1C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' python run_plot.py --input ${DIR[PLOT,VPP]} \ - --output ${DIR[STATIC,VPP]}/64B-2t2c-l2-ndrdisc \ + --output ${DIR[STATIC,VPP]}/64B-2t2c-l2-sel1-ndrdisc \ --title "64B-2t2c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc" \ - --xpath '//*[@framesize="64B" and (contains(@tags,"BASE") or (contains(@tags,"FEATURE") and contains(@tags,"ACL50") and contains(@tags,"10k_FLOWS"))) and contains(@tags,"NDRDISC") and contains(@tags,"2T2C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' + --xpath '//*[@framesize="64B" and contains(@tags,"BASE") and contains(@tags,"NDRDISC") and contains(@tags,"2T2C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' python run_plot.py --input ${DIR[PLOT,VPP]} \ - --output ${DIR[STATIC,VPP]}/64B-1t1c-l2-pdrdisc \ + --output ${DIR[STATIC,VPP]}/64B-1t1c-l2-sel1-pdrdisc \ --title "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-pdrdisc" \ - --xpath '//*[@framesize="64B" and (contains(@tags,"BASE") or (contains(@tags,"FEATURE") and contains(@tags,"ACL50") and contains(@tags,"10k_FLOWS"))) and contains(@tags,"PDRDISC") and not(contains(@tags,"NDRDISC")) and contains(@tags,"1T1C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' + --xpath '//*[@framesize="64B" and contains(@tags,"BASE") and contains(@tags,"PDRDISC") and not(contains(@tags,"NDRDISC")) and contains(@tags,"1T1C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' python run_plot.py --input ${DIR[PLOT,VPP]} \ - --output ${DIR[STATIC,VPP]}/64B-2t2c-l2-pdrdisc \ + --output ${DIR[STATIC,VPP]}/64B-2t2c-l2-sel1-pdrdisc \ --title "64B-2t2c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-pdrdisc" \ - --xpath '//*[@framesize="64B" and (contains(@tags,"BASE") or (contains(@tags,"FEATURE") and contains(@tags,"ACL50") and contains(@tags,"10k_FLOWS"))) and contains(@tags,"PDRDISC") and not(contains(@tags,"NDRDISC")) and contains(@tags,"2T2C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' + --xpath '//*[@framesize="64B" and contains(@tags,"BASE") and contains(@tags,"PDRDISC") and not(contains(@tags,"NDRDISC")) and contains(@tags,"2T2C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' + +# VPP L2 sel2 + +python run_plot.py --input ${DIR[PLOT,VPP]} \ + --output ${DIR[STATIC,VPP]}/64B-1t1c-l2-sel2-ndrdisc \ + --title "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc" \ + --xpath '//*[@framesize="64B" and ((contains(@tags,"FEATURE") and contains(@tags,"ACL50") and contains(@tags,"10k_FLOWS"))) and contains(@tags,"NDRDISC") and contains(@tags,"1T1C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' --lower 0 --upper 8000000 +python run_plot.py --input ${DIR[PLOT,VPP]} \ + --output ${DIR[STATIC,VPP]}/64B-2t2c-l2-sel2-ndrdisc \ + --title "64B-2t2c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc" \ + --xpath '//*[@framesize="64B" and ((contains(@tags,"FEATURE") and contains(@tags,"ACL50") and contains(@tags,"10k_FLOWS"))) and contains(@tags,"NDRDISC") and contains(@tags,"2T2C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' --lower 5000000 --upper 12000000 + +python run_plot.py --input ${DIR[PLOT,VPP]} \ + --output ${DIR[STATIC,VPP]}/64B-1t1c-l2-sel2-pdrdisc \ + --title "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-pdrdisc" \ + --xpath '//*[@framesize="64B" and ((contains(@tags,"FEATURE") and contains(@tags,"ACL50") and contains(@tags,"10k_FLOWS"))) and contains(@tags,"PDRDISC") and not(contains(@tags,"NDRDISC")) and contains(@tags,"1T1C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' --lower 0 --upper 8000000 +python run_plot.py --input ${DIR[PLOT,VPP]} \ + --output ${DIR[STATIC,VPP]}/64B-2t2c-l2-sel2-pdrdisc \ + --title "64B-2t2c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-pdrdisc" \ + --xpath '//*[@framesize="64B" and ((contains(@tags,"FEATURE") and contains(@tags,"ACL50") and contains(@tags,"10k_FLOWS"))) and contains(@tags,"NDRDISC") and contains(@tags,"2T2C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' --lower 5000000 --upper 12000000 # VPP IP4 @@ -430,16 +453,27 @@ python run_plot.py --input ${DIR[PLOT,DPDK]} \ # Plot latency -# VPP L2 +# VPP L2 sel1 + +python run_plot.py --input ${DIR[PLOT,VPP]} \ + --output ${DIR[STATIC,VPP]}/64B-1t1c-l2-sel1-ndrdisc-lat50 \ + --title "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc" \ + --xpath '//*[@framesize="64B" and contains(@tags,"BASE") and contains(@tags,"NDRDISC") and contains(@tags,"1T1C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' --latency lat_50 +python run_plot.py --input ${DIR[PLOT,VPP]} \ + --output ${DIR[STATIC,VPP]}/64B-2t2c-l2-sel1-ndrdisc-lat50 \ + --title "64B-2t2c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc" \ + --xpath '//*[@framesize="64B" and contains(@tags,"BASE") and contains(@tags,"NDRDISC") and contains(@tags,"2T2C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' --latency lat_50 + +# VPP L2 sel2 python run_plot.py --input ${DIR[PLOT,VPP]} \ - --output ${DIR[STATIC,VPP]}/64B-1t1c-l2-ndrdisc-lat50 \ + --output ${DIR[STATIC,VPP]}/64B-1t1c-l2-sel2-ndrdisc-lat50 \ --title "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc" \ - --xpath '//*[@framesize="64B" and (contains(@tags,"BASE") or (contains(@tags,"FEATURE") and contains(@tags,"ACL50") and contains(@tags,"10k_FLOWS"))) and contains(@tags,"NDRDISC") and contains(@tags,"1T1C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' --latency lat_50 + --xpath '//*[@framesize="64B" and ((contains(@tags,"FEATURE") and contains(@tags,"ACL50") and contains(@tags,"10k_FLOWS"))) and contains(@tags,"NDRDISC") and contains(@tags,"1T1C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' --latency lat_50 python run_plot.py --input ${DIR[PLOT,VPP]} \ - --output ${DIR[STATIC,VPP]}/64B-2t2c-l2-ndrdisc-lat50 \ + --output ${DIR[STATIC,VPP]}/64B-2t2c-l2-sel2-ndrdisc-lat50 \ --title "64B-2t2c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc" \ - --xpath '//*[@framesize="64B" and (contains(@tags,"BASE") or (contains(@tags,"FEATURE") and contains(@tags,"ACL50") and contains(@tags,"10k_FLOWS"))) and contains(@tags,"NDRDISC") and contains(@tags,"2T2C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' --latency lat_50 + --xpath '//*[@framesize="64B" and ((contains(@tags,"FEATURE") and contains(@tags,"ACL50") and contains(@tags,"10k_FLOWS"))) and contains(@tags,"NDRDISC") and contains(@tags,"2T2C") and (contains(@tags,"L2BDMACSTAT") or contains(@tags,"L2BDMACLRN") or contains(@tags,"L2XCFWD")) and not(contains(@tags,"VHOST"))]' --latency lat_50 # VPP IP4 -- 2.16.6