1 Presentation and Analytics Layer
2 ================================
7 The presentation and analytics layer (PAL) is the fourth layer of CSIT
8 hierarchy. The model of presentation and analytics layer consists of four
11 - sL1 - Data - input data to be processed:
13 - Static content - .rst text files, .svg static figures, and other files
14 stored in the CSIT git repository.
15 - Data to process - .xml files generated by Jenkins jobs executing tests,
16 stored as robot results files (output.xml).
17 - Specification - .yaml file with the models of report elements (tables,
18 plots, layout, ...) generated by this tool. There is also the configuration
19 of the tool and the specification of input data (jobs and builds).
21 - sL2 - Data processing
23 - The data are read from the specified input files (.xml) and stored as
24 multi-indexed `pandas.Series <https://pandas.pydata.org/pandas-docs/stable/
25 generated/pandas.Series.html>`_.
26 - This layer provides also interface to input data and filtering of the input
29 - sL3 - Data presentation - This layer generates the elements specified in the
32 - Tables: .csv files linked to static .rst files.
33 - Plots: .html files generated using plot.ly linked to static .rst files.
35 - sL4 - Report generation - Sphinx generates required formats and versions:
38 - versions: minimal, full (TODO: define the names and scope of versions)
46 \includesvg[width=0.90\textwidth]{../_tmp/src/csit_framework_documentation/pal_layers}
47 \label{fig:pal_layers}
52 .. figure:: pal_layers.svg
62 The report specification file defines which data is used and which outputs are
63 generated. It is human readable and structured. It is easy to add / remove /
64 change items. The specification includes:
66 - Specification of the environment.
67 - Configuration of debug mode (optional).
68 - Specification of input data (jobs, builds, files, ...).
69 - Specification of the output.
70 - What and how is generated:
71 - What: plots, tables.
72 - How: specification of all properties and parameters.
75 Structure of the specification file
76 '''''''''''''''''''''''''''''''''''
78 The specification file is organized as a list of dictionaries distinguished by
102 Each type represents a section. The sections "environment", "debug", "static",
103 "input" and "output" are listed only once in the specification; "table", "file"
104 and "plot" can be there multiple times.
106 Sections "debug", "table", "file" and "plot" are optional.
108 Table(s), files(s) and plot(s) are referred as "elements" in this text. It is
109 possible to define and implement other elements if needed.
115 This section has the following parts:
117 - type: "environment" - says that this is the section "environment".
118 - configuration - configuration of the PAL.
119 - paths - paths used by the PAL.
120 - urls - urls pointing to the data sources.
121 - make-dirs - a list of the directories to be created by the PAL while
122 preparing the environment.
123 - remove-dirs - a list of the directories to be removed while cleaning the
125 - build-dirs - a list of the directories where the results are stored.
127 The structure of the section "Environment" is as follows (example):
136 # - Download of input data files
138 # - Read data from given zip / xml files
139 # - Set the configuration as it is done in normal mode
140 # If the section "type: debug" is missing, CFG[DEBUG] is set to 0.
144 # Top level directories:
148 DIR[BUILD,HTML]: "_build"
149 DIR[BUILD,LATEX]: "_build_latex"
152 DIR[RST]: "../../../docs/report"
154 # Working directories
155 ## Input data files (.zip, .xml)
156 DIR[WORKING,DATA]: "{DIR[WORKING]}/data"
157 ## Static source files from git
158 DIR[WORKING,SRC]: "{DIR[WORKING]}/src"
159 DIR[WORKING,SRC,STATIC]: "{DIR[WORKING,SRC]}/_static"
161 # Static html content
162 DIR[STATIC]: "{DIR[BUILD,HTML]}/_static"
163 DIR[STATIC,VPP]: "{DIR[STATIC]}/vpp"
164 DIR[STATIC,DPDK]: "{DIR[STATIC]}/dpdk"
165 DIR[STATIC,ARCH]: "{DIR[STATIC]}/archive"
167 # Detailed test results
168 DIR[DTR]: "{DIR[WORKING,SRC]}/detailed_test_results"
169 DIR[DTR,PERF,DPDK]: "{DIR[DTR]}/dpdk_performance_results"
170 DIR[DTR,PERF,VPP]: "{DIR[DTR]}/vpp_performance_results"
171 DIR[DTR,PERF,HC]: "{DIR[DTR]}/honeycomb_performance_results"
172 DIR[DTR,FUNC,VPP]: "{DIR[DTR]}/vpp_functional_results"
173 DIR[DTR,FUNC,HC]: "{DIR[DTR]}/honeycomb_functional_results"
174 DIR[DTR,FUNC,NSHSFC]: "{DIR[DTR]}/nshsfc_functional_results"
175 DIR[DTR,PERF,VPP,IMPRV]: "{DIR[WORKING,SRC]}/vpp_performance_tests/performance_improvements"
177 # Detailed test configurations
178 DIR[DTC]: "{DIR[WORKING,SRC]}/test_configuration"
179 DIR[DTC,PERF,VPP]: "{DIR[DTC]}/vpp_performance_configuration"
180 DIR[DTC,FUNC,VPP]: "{DIR[DTC]}/vpp_functional_configuration"
182 # Detailed tests operational data
183 DIR[DTO]: "{DIR[WORKING,SRC]}/test_operational_data"
184 DIR[DTO,PERF,VPP]: "{DIR[DTO]}/vpp_performance_operational_data"
186 # .css patch file to fix tables generated by Sphinx
187 DIR[CSS_PATCH_FILE]: "{DIR[STATIC]}/theme_overrides.css"
188 DIR[CSS_PATCH_FILE2]: "{DIR[WORKING,SRC,STATIC]}/theme_overrides.css"
191 URL[JENKINS,CSIT]: "https://jenkins.fd.io/view/csit/job"
192 URL[JENKINS,HC]: "https://jenkins.fd.io/view/hc2vpp/job"
195 # List the directories which are created while preparing the environment.
196 # All directories MUST be defined in "paths" section.
197 - "DIR[WORKING,DATA]"
203 - "DIR[WORKING,SRC,STATIC]"
206 # List the directories which are deleted while cleaning the environment.
207 # All directories MUST be defined in "paths" section.
211 # List the directories where the results (build) is stored.
212 # All directories MUST be defined in "paths" section.
216 It is possible to use defined items in the definition of other items, e.g.:
220 DIR[WORKING,DATA]: "{DIR[WORKING]}/data"
222 will be automatically changed to
226 DIR[WORKING,DATA]: "_tmp/data"
229 Section: Configuration
230 ''''''''''''''''''''''
232 This section specifies the groups of parameters which are repeatedly used in the
233 elements defined later in the specification file. It has the following parts:
235 - data sets - Specification of data sets used later in element's specifications
236 to define the input data.
237 - plot layouts - Specification of plot layouts used later in plots'
238 specifications to define the plot layout.
240 The structure of the section "Configuration" is as follows (example):
245 type: "configuration"
247 plot-vpp-throughput-latency:
248 csit-vpp-perf-1710-all:
260 csit-vpp-perf-1710-all:
269 gridcolor: "rgb(238, 238, 238)"
270 linecolor: "rgb(238, 238, 238)"
275 tickcolor: "rgb(238, 238, 238)"
277 title: "Indexed Test Cases"
280 gridcolor: "rgb(238, 238, 238)'"
282 linecolor: "rgb(238, 238, 238)"
288 tickcolor: "rgb(238, 238, 238)"
289 title: "Packets Per Second [pps]"
305 The definitions from this sections are used in the elements, e.g.:
311 title: "VPP Performance 64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
312 algorithm: "plot_performance_box"
313 output-file-type: ".html"
314 output-file: "{DIR[STATIC,VPP]}/64B-1t1c-l2-sel1-ndrdisc"
316 "plot-vpp-throughput-latency"
317 filter: "'64B' and ('BASE' or 'SCALE') and 'NDRDISC' and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
323 boxpoints: "outliers"
326 title: "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
334 This section is optional as it configures the debug mode. It is used if one
335 does not want to download input data files and use local files instead.
337 If the debug mode is configured, the "input" section is ignored.
339 This section has the following parts:
341 - type: "debug" - says that this is the section "debug".
344 - input-format - xml or zip.
345 - extract - if "zip" is defined as the input format, this file is extracted
346 from the zip file, otherwise this parameter is ignored.
348 - builds - list of builds from which the data is used. Must include a job
349 name as a key and then a list of builds and their output files.
351 The structure of the section "Debug" is as follows (example):
358 input-format: "zip" # zip or xml
359 extract: "robot-plugin/output.xml" # Only for zip
361 # The files must be in the directory DIR[WORKING,DATA]
362 csit-dpdk-perf-1707-all:
365 file: "csit-dpdk-perf-1707-all__10.xml"
368 file: "csit-dpdk-perf-1707-all__9.xml"
369 csit-nsh_sfc-verify-func-1707-ubuntu1604-virl:
372 file: "csit-nsh_sfc-verify-func-1707-ubuntu1604-virl-2.xml"
373 csit-vpp-functional-1707-ubuntu1604-virl:
375 build: lastSuccessfulBuild
376 file: "csit-vpp-functional-1707-ubuntu1604-virl-lastSuccessfulBuild.xml"
377 hc2vpp-csit-integration-1707-ubuntu1604:
379 build: lastSuccessfulBuild
380 file: "hc2vpp-csit-integration-1707-ubuntu1604-lastSuccessfulBuild.xml"
381 csit-vpp-perf-1707-all:
384 file: "csit-vpp-perf-1707-all__16__output.xml"
387 file: "csit-vpp-perf-1707-all__17__output.xml"
393 This section defines the static content which is stored in git and will be used
394 as a source to generate the report.
396 This section has these parts:
398 - type: "static" - says that this section is the "static".
399 - src-path - path to the static content.
400 - dst-path - destination path where the static content is copied and then
406 src-path: "{DIR[RST]}"
407 dst-path: "{DIR[WORKING,SRC]}"
413 This section defines the data used to generate elements. It is mandatory
414 if the debug mode is not used.
416 This section has the following parts:
418 - type: "input" - says that this section is the "input".
419 - general - parameters common to all builds:
421 - file-name: file to be downloaded.
422 - file-format: format of the downloaded file, ".zip" or ".xml" are supported.
423 - download-path: path to be added to url pointing to the file, e.g.:
424 "{job}/{build}/robot/report/*zip*/{filename}"; {job}, {build} and
425 {filename} are replaced by proper values defined in this section.
426 - extract: file to be extracted from downloaded zip file, e.g.: "output.xml";
427 if xml file is downloaded, this parameter is ignored.
429 - builds - list of jobs (keys) and numbers of builds which output data will be
432 The structure of the section "Input" is as follows (example from 17.07 report):
437 type: "input" # Ignored in debug mode
439 file-name: "robot-plugin.zip"
441 download-path: "{job}/{build}/robot/report/*zip*/{filename}"
442 extract: "robot-plugin/output.xml"
444 csit-vpp-perf-1707-all:
456 csit-dpdk-perf-1707-all:
467 csit-vpp-functional-1707-ubuntu1604-virl:
468 - lastSuccessfulBuild
469 hc2vpp-csit-perf-master-ubuntu1604:
472 hc2vpp-csit-integration-1707-ubuntu1604:
473 - lastSuccessfulBuild
474 csit-nsh_sfc-verify-func-1707-ubuntu1604-virl:
481 This section specifies which format(s) will be generated (html, pdf) and which
482 versions will be generated for each format.
484 This section has the following parts:
486 - type: "output" - says that this section is the "output".
487 - format: html or pdf.
488 - version: defined for each format separately.
490 The structure of the section "Output" is as follows (example):
503 TODO: define the names of versions
506 Content of "minimal" version
507 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
509 TODO: define the name and content of this version
515 This section defines a table to be generated. There can be 0 or more "table"
518 This section has the following parts:
520 - type: "table" - says that this section defines a table.
521 - title: Title of the table.
522 - algorithm: Algorithm which is used to generate the table. The other
523 parameters in this section must provide all information needed by the used
525 - template: (optional) a .csv file used as a template while generating the
527 - output-file-ext: extension of the output file.
528 - output-file: file which the table will be written to.
529 - columns: specification of table columns:
531 - title: The title used in the table header.
532 - data: Specification of the data, it has two parts - command and arguments:
536 - template - take the data from template, arguments:
538 - number of column in the template.
540 - data - take the data from the input data, arguments:
542 - jobs and builds which data will be used.
544 - operation - performs an operation with the data already in the table,
547 - operation to be done, e.g.: mean, stdev, relative_change (compute
548 the relative change between two columns) and display number of data
549 samples ~= number of test jobs. The operations are implemented in the
551 TODO: Move from utils,py to e.g. operations.py
552 - numbers of columns which data will be used (optional).
554 - data: Specify the jobs and builds which data is used to generate the table.
555 - filter: filter based on tags applied on the input data, if "template" is
556 used, filtering is based on the template.
557 - parameters: Only these parameters will be put to the output data structure.
559 The structure of the section "Table" is as follows (example of
560 "table_performance_improvements"):
566 title: "Performance improvements"
567 algorithm: "table_performance_improvements"
568 template: "{DIR[DTR,PERF,VPP,IMPRV]}/tmpl_performance_improvements.csv"
569 output-file-ext: ".csv"
570 output-file: "{DIR[DTR,PERF,VPP,IMPRV]}/performance_improvements"
573 title: "VPP Functionality"
579 title: "VPP-16.09 mean [Mpps]"
582 title: "VPP-17.01 mean [Mpps]"
585 title: "VPP-17.04 mean [Mpps]"
588 title: "VPP-17.07 mean [Mpps]"
589 data: "data csit-vpp-perf-1707-all mean"
591 title: "VPP-17.07 stdev [Mpps]"
592 data: "data csit-vpp-perf-1707-all stdev"
594 title: "17.04 to 17.07 change [%]"
595 data: "operation relative_change 5 4"
597 csit-vpp-perf-1707-all:
612 Example of "table_details" which generates "Detailed Test Results - VPP
613 Performance Results":
619 title: "Detailed Test Results - VPP Performance Results"
620 algorithm: "table_details"
621 output-file-ext: ".csv"
622 output-file: "{DIR[WORKING]}/vpp_performance_results"
626 data: "data test_name"
628 title: "Documentation"
629 data: "data test_documentation"
632 data: "data test_msg"
634 csit-vpp-perf-1707-all:
642 Example of "table_details" which generates "Test configuration - VPP Performance
649 title: "Test configuration - VPP Performance Test Configs"
650 algorithm: "table_details"
651 output-file-ext: ".csv"
652 output-file: "{DIR[WORKING]}/vpp_test_configuration"
658 title: "VPP API Test (VAT) Commands History - Commands Used Per Test Case"
659 data: "data show-run"
661 csit-vpp-perf-1707-all:
673 This section defines a plot to be generated. There can be 0 or more "plot"
676 This section has these parts:
678 - type: "plot" - says that this section defines a plot.
679 - title: Plot title used in the logs. Title which is displayed is in the
681 - output-file-type: format of the output file.
682 - output-file: file which the plot will be written to.
683 - algorithm: Algorithm used to generate the plot. The other parameters in this
684 section must provide all information needed by plot.ly to generate the plot.
690 - These parameters are transparently passed to plot.ly.
692 - data: Specify the jobs and numbers of builds which data is used to generate
694 - filter: filter applied on the input data.
695 - parameters: Only these parameters will be put to the output data structure.
697 The structure of the section "Plot" is as follows (example of a plot showing
698 throughput in a chart box-with-whiskers):
704 title: "VPP Performance 64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
705 algorithm: "plot_performance_box"
706 output-file-type: ".html"
707 output-file: "{DIR[STATIC,VPP]}/64B-1t1c-l2-sel1-ndrdisc"
709 csit-vpp-perf-1707-all:
720 # Keep this formatting, the filter is enclosed with " (quotation mark) and
721 # each tag is enclosed with ' (apostrophe).
722 filter: "'64B' and 'BASE' and 'NDRDISC' and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
728 boxpoints: "outliers"
731 title: "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
736 gridcolor: "rgb(238, 238, 238)"
737 linecolor: "rgb(238, 238, 238)"
742 tickcolor: "rgb(238, 238, 238)"
744 title: "Indexed Test Cases"
747 gridcolor: "rgb(238, 238, 238)'"
749 linecolor: "rgb(238, 238, 238)"
755 tickcolor: "rgb(238, 238, 238)"
756 title: "Packets Per Second [pps]"
772 The structure of the section "Plot" is as follows (example of a plot showing
773 latency in a box chart):
779 title: "VPP Latency 64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
780 algorithm: "plot_latency_box"
781 output-file-type: ".html"
782 output-file: "{DIR[STATIC,VPP]}/64B-1t1c-l2-sel1-ndrdisc-lat50"
784 csit-vpp-perf-1707-all:
795 filter: "'64B' and 'BASE' and 'NDRDISC' and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
802 title: "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
807 gridcolor: "rgb(238, 238, 238)"
808 linecolor: "rgb(238, 238, 238)"
813 tickcolor: "rgb(238, 238, 238)"
815 title: "Indexed Test Cases"
818 gridcolor: "rgb(238, 238, 238)'"
820 linecolor: "rgb(238, 238, 238)"
826 tickcolor: "rgb(238, 238, 238)"
827 title: "Latency min/avg/max [uSec]"
843 The structure of the section "Plot" is as follows (example of a plot showing
844 VPP HTTP server performance in a box chart with pre-defined data
845 "plot-vpp-httlp-server-performance" set and plot layout "plot-cps"):
851 title: "VPP HTTP Server Performance"
852 algorithm: "plot_http_server_performance_box"
853 output-file-type: ".html"
854 output-file: "{DIR[STATIC,VPP]}/http-server-performance-cps"
856 "plot-vpp-httlp-server-performance"
857 # Keep this formatting, the filter is enclosed with " (quotation mark) and
858 # each tag is enclosed with ' (apostrophe).
859 filter: "'HTTP' and 'TCP_CPS'"
865 boxpoints: "outliers"
868 title: "VPP HTTP Server Performance"
876 This section defines a file to be generated. There can be 0 or more "file"
879 This section has the following parts:
881 - type: "file" - says that this section defines a file.
882 - title: Title of the table.
883 - algorithm: Algorithm which is used to generate the file. The other
884 parameters in this section must provide all information needed by the used
886 - output-file-ext: extension of the output file.
887 - output-file: file which the file will be written to.
888 - file-header: The header of the generated .rst file.
889 - dir-tables: The directory with the tables.
890 - data: Specify the jobs and builds which data is used to generate the table.
891 - filter: filter based on tags applied on the input data, if "all" is
892 used, no filtering is done.
893 - parameters: Only these parameters will be put to the output data structure.
894 - chapters: the hierarchy of chapters in the generated file.
895 - start-level: the level of the the top-level chapter.
897 The structure of the section "file" is as follows (example):
903 title: "VPP Performance Results"
904 algorithm: "file_test_results"
905 output-file-ext: ".rst"
906 output-file: "{DIR[DTR,PERF,VPP]}/vpp_performance_results"
907 file-header: "\n.. |br| raw:: html\n\n <br />\n\n\n.. |prein| raw:: html\n\n <pre>\n\n\n.. |preout| raw:: html\n\n </pre>\n\n"
908 dir-tables: "{DIR[DTR,PERF,VPP]}"
910 csit-vpp-perf-1707-all:
917 data-start-level: 2 # 0, 1, 2, ...
918 chapters-start-level: 2 # 0, 1, 2, ...
924 - Manually created / edited files.
925 - .rst files, static .csv files, static pictures (.svg), ...
926 - Stored in CSIT git repository.
928 No more details about the static content in this document.
934 The PAL processes tests results and other information produced by Jenkins jobs.
935 The data are now stored as robot results in Jenkins (TODO: store the data in
936 nexus) either as .zip and / or .xml files.
942 As the first step, the data are downloaded and stored locally (typically on a
943 Jenkins slave). If .zip files are used, the given .xml files are extracted for
946 Parsing of the .xml files is performed by a class derived from
947 "robot.api.ResultVisitor", only necessary methods are overridden. All and only
948 necessary data is extracted from .xml file and stored in a structured form.
950 The parsed data are stored as the multi-indexed pandas.Series data type. Its
951 structure is as follows:
961 "job name", "build", "metadata", "suites", "tests" are indexes to access the
990 Using indexes data["job 1 name"]["build 1"]["tests"] (e.g.:
991 data["csit-vpp-perf-1704-all"]["17"]["tests"]) we get a list of all tests with
994 Data will not be accessible directly using indexes, but using getters and
997 **Structure of metadata:**
1002 "version": "VPP version",
1003 "job": "Jenkins job name"
1004 "build": "Information about the build"
1007 **Structure of suites:**
1013 "doc": "Suite 1 documentation"
1014 "parent": "Suite 1 parent"
1017 "doc": "Suite N documentation"
1018 "parent": "Suite N parent"
1021 **Structure of tests:**
1029 "name": "Test name",
1030 "parent": "Name of the parent of the test",
1031 "doc": "Test documentation"
1032 "msg": "Test message"
1033 "tags": ["tag 1", "tag 2", "tag n"],
1034 "type": "PDR" | "NDR",
1037 "unit": "pps" | "bps" | "percentage"
1046 "50": { # Only for NDR
1051 "10": { # Only for NDR
1063 "50": { # Only for NDR
1068 "10": { # Only for NDR
1075 "lossTolerance": "lossTolerance" # Only for PDR
1076 "vat-history": "DUT1 and DUT2 VAT History"
1078 "show-run": "Show Run"
1090 "name": "Test name",
1091 "parent": "Name of the parent of the test",
1092 "doc": "Test documentation"
1093 "msg": "Test message"
1094 "tags": ["tag 1", "tag 2", "tag n"],
1095 "vat-history": "DUT1 and DUT2 VAT History"
1096 "show-run": "Show Run"
1097 "status": "PASS" | "FAIL"
1104 Note: ID is the lowercase full path to the test.
1110 The first step when generating an element is getting the data needed to
1111 construct the element. The data are filtered from the processed input data.
1113 The data filtering is based on:
1118 - required data - only this data is included in the output.
1120 WARNING: The filtering is based on tags, so be careful with tagging.
1122 For example, the element which specification includes:
1127 csit-vpp-perf-1707-all:
1139 - "'64B' and 'BASE' and 'NDRDISC' and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
1141 will be constructed using data from the job "csit-vpp-perf-1707-all", for all
1142 listed builds and the tests with the list of tags matching the filter
1145 The output data structure for filtered test data is:
1168 Data analytics part implements:
1170 - methods to compute statistical data from the filtered input data.
1173 Throughput Speedup Analysis - Multi-Core with Multi-Threading
1174 '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
1176 Throughput Speedup Analysis (TSA) calculates throughput speedup ratios
1177 for tested 1-, 2- and 4-core multi-threaded VPP configurations using the
1183 N_core_throughput_speedup = -----------------
1186 Multi-core throughput speedup ratios are plotted in grouped bar graphs
1187 for throughput tests with 64B/78B frame size, with number of cores on
1188 X-axis and speedup ratio on Y-axis.
1190 For better comparison multiple test results' data sets are plotted per
1193 - graph type: grouped bars;
1194 - graph X-axis: (testcase index, number of cores);
1195 - graph Y-axis: speedup factor.
1197 Subset of existing performance tests is covered by TSA graphs.
1205 title: "TSA: 64B-*-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
1206 algorithm: "plot_throughput_speedup_analysis"
1207 output-file-type: ".html"
1208 output-file: "{DIR[STATIC,VPP]}/10ge2p1x520-64B-l2-tsa-ndrdisc"
1210 "plot-throughput-speedup-analysis"
1211 filter: "'NIC_Intel-X520-DA2' and '64B' and 'BASE' and 'NDRDISC' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
1217 title: "64B-*-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
1219 "plot-throughput-speedup-analysis"
1222 Comparison of results from two sets of the same test executions
1223 '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
1225 This algorithm enables comparison of results coming from two sets of the
1226 same test executions. It is used to quantify performance changes across
1227 all tests after test environment changes e.g. Operating System
1228 upgrades/patches, Hardware changes.
1230 It is assumed that each set of test executions includes multiple runs
1231 of the same tests, 10 or more, to verify test results repeatibility and
1232 to yield statistically meaningful results data.
1234 Comparison results are presented in a table with a specified number of
1235 the best and the worst relative changes between the two sets. Following table
1236 columns are defined:
1239 - throughput mean values of the reference set;
1240 - throughput standard deviation of the reference set;
1241 - throughput mean values of the set to compare;
1242 - throughput standard deviation of the set to compare;
1243 - relative change of the mean values.
1247 The model specifies:
1249 - type: "table" - means this section defines a table.
1250 - title: Title of the table.
1251 - algorithm: Algorithm which is used to generate the table. The other
1252 parameters in this section must provide all information needed by the used
1254 - output-file-ext: Extension of the output file.
1255 - output-file: File which the table will be written to.
1256 - reference - the builds which are used as the reference for comparison.
1257 - compare - the builds which are compared to the reference.
1258 - data: Specify the sources, jobs and builds, providing data for generating
1260 - filter: Filter based on tags applied on the input data, if "template" is
1261 used, filtering is based on the template.
1262 - parameters: Only these parameters will be put to the output data
1264 - nr-of-tests-shown: Number of the best and the worst tests presented in the
1265 table. Use 0 (zero) to present all tests.
1273 title: "Performance comparison"
1274 algorithm: "table_performance_comparison"
1275 output-file-ext: ".csv"
1276 output-file: "{DIR[DTR,PERF,VPP,IMPRV]}/vpp_performance_comparison"
1278 title: "csit-vpp-perf-1801-all - 1"
1280 csit-vpp-perf-1801-all:
1284 title: "csit-vpp-perf-1801-all - 2"
1286 csit-vpp-perf-1801-all:
1290 "vpp-perf-comparison"
1296 nr-of-tests-shown: 20
1299 Advanced data analytics
1300 ```````````````````````
1302 In the future advanced data analytics (ADA) will be added to analyze the
1303 telemetry data collected from SUT telemetry sources and correlate it to
1304 performance test results.
1308 - describe the concept of ADA.
1309 - add specification.
1315 Generates the plots and tables according to the report models per
1316 specification file. The elements are generated using algorithms and data
1317 specified in their models.
1323 - tables are generated by algorithms implemented in PAL, the model includes the
1324 algorithm and all necessary information.
1325 - output format: csv
1326 - generated tables are stored in specified directories and linked to .rst
1333 - `plot.ly <https://plot.ly/>`_ is currently used to generate plots, the model
1334 includes the type of plot and all the necessary information to render it.
1335 - output format: html.
1336 - generated plots are stored in specified directories and linked to .rst files.
1342 Report is generated using Sphinx and Read_the_Docs template. PAL generates html
1343 and pdf formats. It is possible to define the content of the report by
1344 specifying the version (TODO: define the names and content of versions).
1350 1. Read the specification.
1351 2. Read the input data.
1352 3. Process the input data.
1353 4. For element (plot, table, file) defined in specification:
1355 a. Get the data needed to construct the element using a filter.
1356 b. Generate the element.
1357 c. Store the element.
1359 5. Generate the report.
1360 6. Store the report (Nexus).
1362 The process is model driven. The elements' models (tables, plots, files
1363 and report itself) are defined in the specification file. Script reads
1364 the elements' models from specification file and generates the elements.
1366 It is easy to add elements to be generated in the report. If a new type
1367 of an element is required, only a new algorithm needs to be implemented
1371 Continuous Performance Measurements and Trending
1372 ------------------------------------------------
1374 Performance analysis and trending execution sequence:
1375 `````````````````````````````````````````````````````
1377 CSIT PA runs performance analysis, change detection and trending using specified
1378 trend analysis metrics over the rolling window of last <N> sets of historical
1379 measurement data. PA is defined as follows:
1383 #. By PT job at its completion.
1384 #. Manually from Jenkins UI.
1386 #. Download and parse archived historical data and the new data:
1388 #. New data from latest PT job is evaluated against the rolling window
1389 of <N> sets of historical data.
1390 #. Download RF output.xml files and compressed archived data.
1391 #. Parse out the data filtering test cases listed in PA specification
1392 (part of CSIT PAL specification file).
1394 #. Calculate trend metrics for the rolling window of <N> sets of historical data:
1396 #. Calculate quartiles Q1, Q2, Q3.
1397 #. Trim outliers using IQR.
1398 #. Calculate TMA and TMSD.
1399 #. Calculate normal trending range per test case based on TMA and TMSD.
1401 #. Evaluate new test data against trend metrics:
1403 #. If within the range of (TMA +/- 3*TMSD) => Result = Pass,
1405 #. If below the range => Result = Fail, Reason = Regression.
1406 #. If above the range => Result = Pass, Reason = Progression.
1408 #. Generate and publish results
1410 #. Relay evaluation result to job result.
1411 #. Generate a new set of trend analysis summary graphs and drill-down
1414 #. Summary graphs to include measured values with Normal,
1415 Progression and Regression markers. MM shown in the background if
1417 #. Drill-down graphs to include MM, TMA and TMSD.
1419 #. Publish trend analysis graphs in html format on
1420 https://docs.fd.io/csit/master/trending/.
1423 Parameters to specify:
1424 ``````````````````````
1426 - job to be monitored - the Jenkins job which results are used as input data for
1428 - builds used for trending plot(s) - specified by a list of build numbers or by
1429 a range of builds defined by the first and the last buld number;
1430 - list plots to generate:
1435 - tests to be displayed in the plot defined by a filter;
1436 - list of parameters to extract from the data;
1437 - periods (daily = 1, weekly = 5, monthly = 30);
1446 title: "Continuous Performance Trending and Analysis"
1448 output-file-type: ".html"
1449 output-file: "{DIR[STATIC,VPP]}/cpta"
1450 data: "plot-performance-trending"
1452 - title: "VPP 1T1C L2 64B Packet Throughput - {period} Trending"
1453 output-file-name: "l2-1t1c-x520"
1454 data: "plot-performance-trending"
1455 filter: "'NIC_Intel-X520-DA2' and 'MRR' and '64B' and ('BASE' or 'SCALE') and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST' and not 'MEMIF'"
1465 - title: "VPP 2T2C L2 64B Packet Throughput - {period} Trending"
1466 output-file-name: "l2-2t2c-x520"
1467 data: "plot-performance-trending"
1468 filter: "'NIC_Intel-X520-DA2' and 'MRR' and '64B' and ('BASE' or 'SCALE') and '2T2C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST' and not 'MEMIF'"
1481 List of modules, classes, methods and functions
1482 ```````````````````````````````````````````````
1486 specification_parser.py
1509 input_data_parser.py
1550 Functions implementing algorithms to generate particular types of
1551 tables (called by the function "generate_tables"):
1553 table_performance_improvements
1561 Functions implementing algorithms to generate particular types of
1562 plots (called by the function "generate_plots"):
1563 plot_performance_box
1572 Functions implementing algorithms to generate particular types of
1573 files (called by the function "generate_files"):
1582 Functions implementing algorithms to generate particular types of
1583 report (called by the function "generate_report"):
1584 generate_html_report
1587 Other functions called by the function "generate_report":
1592 PAL functional diagram
1593 ``````````````````````
1601 \includesvg[width=0.90\textwidth]{../_tmp/src/csit_framework_documentation/pal_func_diagram}
1602 \label{fig:pal_func_diagram}
1607 .. figure:: pal_func_diagram.svg
1608 :alt: PAL functional diagram
1612 How to add an element
1613 `````````````````````
1615 Element can be added by adding it's model to the specification file. If
1616 the element is to be generated by an existing algorithm, only it's
1617 parameters must be set.
1619 If a brand new type of element needs to be added, also the algorithm
1620 must be implemented. Element generation algorithms are implemented in
1621 the files with names starting with "generator" prefix. The name of the
1622 function implementing the algorithm and the name of algorithm in the
1623 specification file have to be the same.