1 Presentation and Analytics Layer
2 ================================
7 The presentation and analytics layer (PAL) is the fourth layer of CSIT
8 hierarchy. The model of presentation and analytics layer consists of four
11 - sL1 - Data - input data to be processed:
13 - Static content - .rst text files, .svg static figures, and other files
14 stored in the CSIT git repository.
15 - Data to process - .xml files generated by Jenkins jobs executing tests,
16 stored as robot results files (output.xml).
17 - Specification - .yaml file with the models of report elements (tables,
18 plots, layout, ...) generated by this tool. There is also the configuration
19 of the tool and the specification of input data (jobs and builds).
21 - sL2 - Data processing
23 - The data are read from the specified input files (.xml) and stored as
24 multi-indexed `pandas.Series <https://pandas.pydata.org/pandas-docs/stable/
25 generated/pandas.Series.html>`_.
26 - This layer provides also interface to input data and filtering of the input
29 - sL3 - Data presentation - This layer generates the elements specified in the
32 - Tables: .csv files linked to static .rst files.
33 - Plots: .html files generated using plot.ly linked to static .rst files.
35 - sL4 - Report generation - Sphinx generates required formats and versions:
38 - versions: minimal, full (TODO: define the names and scope of versions)
46 \graphicspath{{../_tmp/src/csit_framework_documentation/}}
47 \includegraphics[width=0.90\textwidth]{pal_layers}
48 \label{fig:pal_layers}
53 .. figure:: pal_layers.svg
63 The report specification file defines which data is used and which outputs are
64 generated. It is human readable and structured. It is easy to add / remove /
65 change items. The specification includes:
67 - Specification of the environment.
68 - Configuration of debug mode (optional).
69 - Specification of input data (jobs, builds, files, ...).
70 - Specification of the output.
71 - What and how is generated:
72 - What: plots, tables.
73 - How: specification of all properties and parameters.
76 Structure of the specification file
77 '''''''''''''''''''''''''''''''''''
79 The specification file is organized as a list of dictionaries distinguished by
103 Each type represents a section. The sections "environment", "debug", "static",
104 "input" and "output" are listed only once in the specification; "table", "file"
105 and "plot" can be there multiple times.
107 Sections "debug", "table", "file" and "plot" are optional.
109 Table(s), files(s) and plot(s) are referred as "elements" in this text. It is
110 possible to define and implement other elements if needed.
116 This section has the following parts:
118 - type: "environment" - says that this is the section "environment".
119 - configuration - configuration of the PAL.
120 - paths - paths used by the PAL.
121 - urls - urls pointing to the data sources.
122 - make-dirs - a list of the directories to be created by the PAL while
123 preparing the environment.
124 - remove-dirs - a list of the directories to be removed while cleaning the
126 - build-dirs - a list of the directories where the results are stored.
128 The structure of the section "Environment" is as follows (example):
137 # - Download of input data files
139 # - Read data from given zip / xml files
140 # - Set the configuration as it is done in normal mode
141 # If the section "type: debug" is missing, CFG[DEBUG] is set to 0.
145 # Top level directories:
149 DIR[BUILD,HTML]: "_build"
150 DIR[BUILD,LATEX]: "_build_latex"
153 DIR[RST]: "../../../docs/report"
155 # Working directories
156 ## Input data files (.zip, .xml)
157 DIR[WORKING,DATA]: "{DIR[WORKING]}/data"
158 ## Static source files from git
159 DIR[WORKING,SRC]: "{DIR[WORKING]}/src"
160 DIR[WORKING,SRC,STATIC]: "{DIR[WORKING,SRC]}/_static"
162 # Static html content
163 DIR[STATIC]: "{DIR[BUILD,HTML]}/_static"
164 DIR[STATIC,VPP]: "{DIR[STATIC]}/vpp"
165 DIR[STATIC,DPDK]: "{DIR[STATIC]}/dpdk"
166 DIR[STATIC,ARCH]: "{DIR[STATIC]}/archive"
168 # Detailed test results
169 DIR[DTR]: "{DIR[WORKING,SRC]}/detailed_test_results"
170 DIR[DTR,PERF,DPDK]: "{DIR[DTR]}/dpdk_performance_results"
171 DIR[DTR,PERF,VPP]: "{DIR[DTR]}/vpp_performance_results"
172 DIR[DTR,PERF,HC]: "{DIR[DTR]}/honeycomb_performance_results"
173 DIR[DTR,FUNC,VPP]: "{DIR[DTR]}/vpp_functional_results"
174 DIR[DTR,FUNC,HC]: "{DIR[DTR]}/honeycomb_functional_results"
175 DIR[DTR,FUNC,NSHSFC]: "{DIR[DTR]}/nshsfc_functional_results"
176 DIR[DTR,PERF,VPP,IMPRV]: "{DIR[WORKING,SRC]}/vpp_performance_tests/performance_improvements"
178 # Detailed test configurations
179 DIR[DTC]: "{DIR[WORKING,SRC]}/test_configuration"
180 DIR[DTC,PERF,VPP]: "{DIR[DTC]}/vpp_performance_configuration"
181 DIR[DTC,FUNC,VPP]: "{DIR[DTC]}/vpp_functional_configuration"
183 # Detailed tests operational data
184 DIR[DTO]: "{DIR[WORKING,SRC]}/test_operational_data"
185 DIR[DTO,PERF,VPP]: "{DIR[DTO]}/vpp_performance_operational_data"
187 # .css patch file to fix tables generated by Sphinx
188 DIR[CSS_PATCH_FILE]: "{DIR[STATIC]}/theme_overrides.css"
189 DIR[CSS_PATCH_FILE2]: "{DIR[WORKING,SRC,STATIC]}/theme_overrides.css"
192 URL[JENKINS,CSIT]: "https://jenkins.fd.io/view/csit/job"
193 URL[JENKINS,HC]: "https://jenkins.fd.io/view/hc2vpp/job"
196 # List the directories which are created while preparing the environment.
197 # All directories MUST be defined in "paths" section.
198 - "DIR[WORKING,DATA]"
204 - "DIR[WORKING,SRC,STATIC]"
207 # List the directories which are deleted while cleaning the environment.
208 # All directories MUST be defined in "paths" section.
212 # List the directories where the results (build) is stored.
213 # All directories MUST be defined in "paths" section.
217 It is possible to use defined items in the definition of other items, e.g.:
221 DIR[WORKING,DATA]: "{DIR[WORKING]}/data"
223 will be automatically changed to
227 DIR[WORKING,DATA]: "_tmp/data"
230 Section: Configuration
231 ''''''''''''''''''''''
233 This section specifies the groups of parameters which are repeatedly used in the
234 elements defined later in the specification file. It has the following parts:
236 - data sets - Specification of data sets used later in element's specifications
237 to define the input data.
238 - plot layouts - Specification of plot layouts used later in plots'
239 specifications to define the plot layout.
241 The structure of the section "Configuration" is as follows (example):
246 type: "configuration"
248 plot-vpp-throughput-latency:
249 csit-vpp-perf-1710-all:
261 csit-vpp-perf-1710-all:
270 gridcolor: "rgb(238, 238, 238)"
271 linecolor: "rgb(238, 238, 238)"
276 tickcolor: "rgb(238, 238, 238)"
278 title: "Indexed Test Cases"
281 gridcolor: "rgb(238, 238, 238)'"
283 linecolor: "rgb(238, 238, 238)"
289 tickcolor: "rgb(238, 238, 238)"
290 title: "Packets Per Second [pps]"
306 The definitions from this sections are used in the elements, e.g.:
312 title: "VPP Performance 64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
313 algorithm: "plot_performance_box"
314 output-file-type: ".html"
315 output-file: "{DIR[STATIC,VPP]}/64B-1t1c-l2-sel1-ndrdisc"
317 "plot-vpp-throughput-latency"
318 filter: "'64B' and ('BASE' or 'SCALE') and 'NDRDISC' and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
324 boxpoints: "outliers"
327 title: "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
335 This section is optional as it configures the debug mode. It is used if one
336 does not want to download input data files and use local files instead.
338 If the debug mode is configured, the "input" section is ignored.
340 This section has the following parts:
342 - type: "debug" - says that this is the section "debug".
345 - input-format - xml or zip.
346 - extract - if "zip" is defined as the input format, this file is extracted
347 from the zip file, otherwise this parameter is ignored.
349 - builds - list of builds from which the data is used. Must include a job
350 name as a key and then a list of builds and their output files.
352 The structure of the section "Debug" is as follows (example):
359 input-format: "zip" # zip or xml
360 extract: "robot-plugin/output.xml" # Only for zip
362 # The files must be in the directory DIR[WORKING,DATA]
363 csit-dpdk-perf-1707-all:
366 file: "csit-dpdk-perf-1707-all__10.xml"
369 file: "csit-dpdk-perf-1707-all__9.xml"
370 csit-nsh_sfc-verify-func-1707-ubuntu1604-virl:
373 file: "csit-nsh_sfc-verify-func-1707-ubuntu1604-virl-2.xml"
374 csit-vpp-functional-1707-ubuntu1604-virl:
376 build: lastSuccessfulBuild
377 file: "csit-vpp-functional-1707-ubuntu1604-virl-lastSuccessfulBuild.xml"
378 hc2vpp-csit-integration-1707-ubuntu1604:
380 build: lastSuccessfulBuild
381 file: "hc2vpp-csit-integration-1707-ubuntu1604-lastSuccessfulBuild.xml"
382 csit-vpp-perf-1707-all:
385 file: "csit-vpp-perf-1707-all__16__output.xml"
388 file: "csit-vpp-perf-1707-all__17__output.xml"
394 This section defines the static content which is stored in git and will be used
395 as a source to generate the report.
397 This section has these parts:
399 - type: "static" - says that this section is the "static".
400 - src-path - path to the static content.
401 - dst-path - destination path where the static content is copied and then
407 src-path: "{DIR[RST]}"
408 dst-path: "{DIR[WORKING,SRC]}"
414 This section defines the data used to generate elements. It is mandatory
415 if the debug mode is not used.
417 This section has the following parts:
419 - type: "input" - says that this section is the "input".
420 - general - parameters common to all builds:
422 - file-name: file to be downloaded.
423 - file-format: format of the downloaded file, ".zip" or ".xml" are supported.
424 - download-path: path to be added to url pointing to the file, e.g.:
425 "{job}/{build}/robot/report/*zip*/{filename}"; {job}, {build} and
426 {filename} are replaced by proper values defined in this section.
427 - extract: file to be extracted from downloaded zip file, e.g.: "output.xml";
428 if xml file is downloaded, this parameter is ignored.
430 - builds - list of jobs (keys) and numbers of builds which output data will be
433 The structure of the section "Input" is as follows (example from 17.07 report):
438 type: "input" # Ignored in debug mode
440 file-name: "robot-plugin.zip"
442 download-path: "{job}/{build}/robot/report/*zip*/{filename}"
443 extract: "robot-plugin/output.xml"
445 csit-vpp-perf-1707-all:
457 csit-dpdk-perf-1707-all:
468 csit-vpp-functional-1707-ubuntu1604-virl:
469 - lastSuccessfulBuild
470 hc2vpp-csit-perf-master-ubuntu1604:
473 hc2vpp-csit-integration-1707-ubuntu1604:
474 - lastSuccessfulBuild
475 csit-nsh_sfc-verify-func-1707-ubuntu1604-virl:
482 This section specifies which format(s) will be generated (html, pdf) and which
483 versions will be generated for each format.
485 This section has the following parts:
487 - type: "output" - says that this section is the "output".
488 - format: html or pdf.
489 - version: defined for each format separately.
491 The structure of the section "Output" is as follows (example):
504 TODO: define the names of versions
507 Content of "minimal" version
508 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
510 TODO: define the name and content of this version
516 This section defines a table to be generated. There can be 0 or more "table"
519 This section has the following parts:
521 - type: "table" - says that this section defines a table.
522 - title: Title of the table.
523 - algorithm: Algorithm which is used to generate the table. The other
524 parameters in this section must provide all information needed by the used
526 - template: (optional) a .csv file used as a template while generating the
528 - output-file-ext: extension of the output file.
529 - output-file: file which the table will be written to.
530 - columns: specification of table columns:
532 - title: The title used in the table header.
533 - data: Specification of the data, it has two parts - command and arguments:
537 - template - take the data from template, arguments:
539 - number of column in the template.
541 - data - take the data from the input data, arguments:
543 - jobs and builds which data will be used.
545 - operation - performs an operation with the data already in the table,
548 - operation to be done, e.g.: mean, stdev, relative_change (compute
549 the relative change between two columns) and display number of data
550 samples ~= number of test jobs. The operations are implemented in the
552 TODO: Move from utils,py to e.g. operations.py
553 - numbers of columns which data will be used (optional).
555 - data: Specify the jobs and builds which data is used to generate the table.
556 - filter: filter based on tags applied on the input data, if "template" is
557 used, filtering is based on the template.
558 - parameters: Only these parameters will be put to the output data structure.
560 The structure of the section "Table" is as follows (example of
561 "table_performance_improvements"):
567 title: "Performance improvements"
568 algorithm: "table_performance_improvements"
569 template: "{DIR[DTR,PERF,VPP,IMPRV]}/tmpl_performance_improvements.csv"
570 output-file-ext: ".csv"
571 output-file: "{DIR[DTR,PERF,VPP,IMPRV]}/performance_improvements"
574 title: "VPP Functionality"
580 title: "VPP-16.09 mean [Mpps]"
583 title: "VPP-17.01 mean [Mpps]"
586 title: "VPP-17.04 mean [Mpps]"
589 title: "VPP-17.07 mean [Mpps]"
590 data: "data csit-vpp-perf-1707-all mean"
592 title: "VPP-17.07 stdev [Mpps]"
593 data: "data csit-vpp-perf-1707-all stdev"
595 title: "17.04 to 17.07 change [%]"
596 data: "operation relative_change 5 4"
598 csit-vpp-perf-1707-all:
613 Example of "table_details" which generates "Detailed Test Results - VPP
614 Performance Results":
620 title: "Detailed Test Results - VPP Performance Results"
621 algorithm: "table_details"
622 output-file-ext: ".csv"
623 output-file: "{DIR[WORKING]}/vpp_performance_results"
627 data: "data test_name"
629 title: "Documentation"
630 data: "data test_documentation"
633 data: "data test_msg"
635 csit-vpp-perf-1707-all:
643 Example of "table_details" which generates "Test configuration - VPP Performance
650 title: "Test configuration - VPP Performance Test Configs"
651 algorithm: "table_details"
652 output-file-ext: ".csv"
653 output-file: "{DIR[WORKING]}/vpp_test_configuration"
659 title: "VPP API Test (VAT) Commands History - Commands Used Per Test Case"
660 data: "data show-run"
662 csit-vpp-perf-1707-all:
674 This section defines a plot to be generated. There can be 0 or more "plot"
677 This section has these parts:
679 - type: "plot" - says that this section defines a plot.
680 - title: Plot title used in the logs. Title which is displayed is in the
682 - output-file-type: format of the output file.
683 - output-file: file which the plot will be written to.
684 - algorithm: Algorithm used to generate the plot. The other parameters in this
685 section must provide all information needed by plot.ly to generate the plot.
691 - These parameters are transparently passed to plot.ly.
693 - data: Specify the jobs and numbers of builds which data is used to generate
695 - filter: filter applied on the input data.
696 - parameters: Only these parameters will be put to the output data structure.
698 The structure of the section "Plot" is as follows (example of a plot showing
699 throughput in a chart box-with-whiskers):
705 title: "VPP Performance 64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
706 algorithm: "plot_performance_box"
707 output-file-type: ".html"
708 output-file: "{DIR[STATIC,VPP]}/64B-1t1c-l2-sel1-ndrdisc"
710 csit-vpp-perf-1707-all:
721 # Keep this formatting, the filter is enclosed with " (quotation mark) and
722 # each tag is enclosed with ' (apostrophe).
723 filter: "'64B' and 'BASE' and 'NDRDISC' and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
729 boxpoints: "outliers"
732 title: "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
737 gridcolor: "rgb(238, 238, 238)"
738 linecolor: "rgb(238, 238, 238)"
743 tickcolor: "rgb(238, 238, 238)"
745 title: "Indexed Test Cases"
748 gridcolor: "rgb(238, 238, 238)'"
750 linecolor: "rgb(238, 238, 238)"
756 tickcolor: "rgb(238, 238, 238)"
757 title: "Packets Per Second [pps]"
773 The structure of the section "Plot" is as follows (example of a plot showing
774 latency in a box chart):
780 title: "VPP Latency 64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
781 algorithm: "plot_latency_box"
782 output-file-type: ".html"
783 output-file: "{DIR[STATIC,VPP]}/64B-1t1c-l2-sel1-ndrdisc-lat50"
785 csit-vpp-perf-1707-all:
796 filter: "'64B' and 'BASE' and 'NDRDISC' and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
803 title: "64B-1t1c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
808 gridcolor: "rgb(238, 238, 238)"
809 linecolor: "rgb(238, 238, 238)"
814 tickcolor: "rgb(238, 238, 238)"
816 title: "Indexed Test Cases"
819 gridcolor: "rgb(238, 238, 238)'"
821 linecolor: "rgb(238, 238, 238)"
827 tickcolor: "rgb(238, 238, 238)"
828 title: "Latency min/avg/max [uSec]"
844 The structure of the section "Plot" is as follows (example of a plot showing
845 VPP HTTP server performance in a box chart with pre-defined data
846 "plot-vpp-httlp-server-performance" set and plot layout "plot-cps"):
852 title: "VPP HTTP Server Performance"
853 algorithm: "plot_http_server_performance_box"
854 output-file-type: ".html"
855 output-file: "{DIR[STATIC,VPP]}/http-server-performance-cps"
857 "plot-vpp-httlp-server-performance"
858 # Keep this formatting, the filter is enclosed with " (quotation mark) and
859 # each tag is enclosed with ' (apostrophe).
860 filter: "'HTTP' and 'TCP_CPS'"
866 boxpoints: "outliers"
869 title: "VPP HTTP Server Performance"
877 This section defines a file to be generated. There can be 0 or more "file"
880 This section has the following parts:
882 - type: "file" - says that this section defines a file.
883 - title: Title of the table.
884 - algorithm: Algorithm which is used to generate the file. The other
885 parameters in this section must provide all information needed by the used
887 - output-file-ext: extension of the output file.
888 - output-file: file which the file will be written to.
889 - file-header: The header of the generated .rst file.
890 - dir-tables: The directory with the tables.
891 - data: Specify the jobs and builds which data is used to generate the table.
892 - filter: filter based on tags applied on the input data, if "all" is
893 used, no filtering is done.
894 - parameters: Only these parameters will be put to the output data structure.
895 - chapters: the hierarchy of chapters in the generated file.
896 - start-level: the level of the the top-level chapter.
898 The structure of the section "file" is as follows (example):
904 title: "VPP Performance Results"
905 algorithm: "file_test_results"
906 output-file-ext: ".rst"
907 output-file: "{DIR[DTR,PERF,VPP]}/vpp_performance_results"
908 file-header: "\n.. |br| raw:: html\n\n <br />\n\n\n.. |prein| raw:: html\n\n <pre>\n\n\n.. |preout| raw:: html\n\n </pre>\n\n"
909 dir-tables: "{DIR[DTR,PERF,VPP]}"
911 csit-vpp-perf-1707-all:
918 data-start-level: 2 # 0, 1, 2, ...
919 chapters-start-level: 2 # 0, 1, 2, ...
925 - Manually created / edited files.
926 - .rst files, static .csv files, static pictures (.svg), ...
927 - Stored in CSIT git repository.
929 No more details about the static content in this document.
935 The PAL processes tests results and other information produced by Jenkins jobs.
936 The data are now stored as robot results in Jenkins (TODO: store the data in
937 nexus) either as .zip and / or .xml files.
943 As the first step, the data are downloaded and stored locally (typically on a
944 Jenkins slave). If .zip files are used, the given .xml files are extracted for
947 Parsing of the .xml files is performed by a class derived from
948 "robot.api.ResultVisitor", only necessary methods are overridden. All and only
949 necessary data is extracted from .xml file and stored in a structured form.
951 The parsed data are stored as the multi-indexed pandas.Series data type. Its
952 structure is as follows:
962 "job name", "build", "metadata", "suites", "tests" are indexes to access the
991 Using indexes data["job 1 name"]["build 1"]["tests"] (e.g.:
992 data["csit-vpp-perf-1704-all"]["17"]["tests"]) we get a list of all tests with
995 Data will not be accessible directly using indexes, but using getters and
998 **Structure of metadata:**
1003 "version": "VPP version",
1004 "job": "Jenkins job name"
1005 "build": "Information about the build"
1008 **Structure of suites:**
1014 "doc": "Suite 1 documentation"
1015 "parent": "Suite 1 parent"
1018 "doc": "Suite N documentation"
1019 "parent": "Suite N parent"
1022 **Structure of tests:**
1030 "name": "Test name",
1031 "parent": "Name of the parent of the test",
1032 "doc": "Test documentation"
1033 "msg": "Test message"
1034 "tags": ["tag 1", "tag 2", "tag n"],
1035 "type": "PDR" | "NDR",
1038 "unit": "pps" | "bps" | "percentage"
1047 "50": { # Only for NDR
1052 "10": { # Only for NDR
1064 "50": { # Only for NDR
1069 "10": { # Only for NDR
1076 "lossTolerance": "lossTolerance" # Only for PDR
1077 "vat-history": "DUT1 and DUT2 VAT History"
1079 "show-run": "Show Run"
1091 "name": "Test name",
1092 "parent": "Name of the parent of the test",
1093 "doc": "Test documentation"
1094 "msg": "Test message"
1095 "tags": ["tag 1", "tag 2", "tag n"],
1096 "vat-history": "DUT1 and DUT2 VAT History"
1097 "show-run": "Show Run"
1098 "status": "PASS" | "FAIL"
1105 Note: ID is the lowercase full path to the test.
1111 The first step when generating an element is getting the data needed to
1112 construct the element. The data are filtered from the processed input data.
1114 The data filtering is based on:
1119 - required data - only this data is included in the output.
1121 WARNING: The filtering is based on tags, so be careful with tagging.
1123 For example, the element which specification includes:
1128 csit-vpp-perf-1707-all:
1140 - "'64B' and 'BASE' and 'NDRDISC' and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
1142 will be constructed using data from the job "csit-vpp-perf-1707-all", for all
1143 listed builds and the tests with the list of tags matching the filter
1146 The output data structure for filtered test data is:
1169 Data analytics part implements:
1171 - methods to compute statistical data from the filtered input data.
1174 Throughput Speedup Analysis - Multi-Core with Multi-Threading
1175 '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
1177 Throughput Speedup Analysis (TSA) calculates throughput speedup ratios
1178 for tested 1-, 2- and 4-core multi-threaded VPP configurations using the
1184 N_core_throughput_speedup = -----------------
1187 Multi-core throughput speedup ratios are plotted in grouped bar graphs
1188 for throughput tests with 64B/78B frame size, with number of cores on
1189 X-axis and speedup ratio on Y-axis.
1191 For better comparison multiple test results' data sets are plotted per
1194 - graph type: grouped bars;
1195 - graph X-axis: (testcase index, number of cores);
1196 - graph Y-axis: speedup factor.
1198 Subset of existing performance tests is covered by TSA graphs.
1206 title: "TSA: 64B-*-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
1207 algorithm: "plot_throughput_speedup_analysis"
1208 output-file-type: ".html"
1209 output-file: "{DIR[STATIC,VPP]}/10ge2p1x520-64B-l2-tsa-ndrdisc"
1211 "plot-throughput-speedup-analysis"
1212 filter: "'NIC_Intel-X520-DA2' and '64B' and 'BASE' and 'NDRDISC' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
1218 title: "64B-*-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
1220 "plot-throughput-speedup-analysis"
1223 Comparison of results from two sets of the same test executions
1224 '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
1226 This algorithm enables comparison of results coming from two sets of the
1227 same test executions. It is used to quantify performance changes across
1228 all tests after test environment changes e.g. Operating System
1229 upgrades/patches, Hardware changes.
1231 It is assumed that each set of test executions includes multiple runs
1232 of the same tests, 10 or more, to verify test results repeatibility and
1233 to yield statistically meaningful results data.
1235 Comparison results are presented in a table with a specified number of
1236 the best and the worst relative changes between the two sets. Following table
1237 columns are defined:
1240 - throughput mean values of the reference set;
1241 - throughput standard deviation of the reference set;
1242 - throughput mean values of the set to compare;
1243 - throughput standard deviation of the set to compare;
1244 - relative change of the mean values.
1248 The model specifies:
1250 - type: "table" - means this section defines a table.
1251 - title: Title of the table.
1252 - algorithm: Algorithm which is used to generate the table. The other
1253 parameters in this section must provide all information needed by the used
1255 - output-file-ext: Extension of the output file.
1256 - output-file: File which the table will be written to.
1257 - reference - the builds which are used as the reference for comparison.
1258 - compare - the builds which are compared to the reference.
1259 - data: Specify the sources, jobs and builds, providing data for generating
1261 - filter: Filter based on tags applied on the input data, if "template" is
1262 used, filtering is based on the template.
1263 - parameters: Only these parameters will be put to the output data
1265 - nr-of-tests-shown: Number of the best and the worst tests presented in the
1266 table. Use 0 (zero) to present all tests.
1274 title: "Performance comparison"
1275 algorithm: "table_performance_comparison"
1276 output-file-ext: ".csv"
1277 output-file: "{DIR[DTR,PERF,VPP,IMPRV]}/vpp_performance_comparison"
1279 title: "csit-vpp-perf-1801-all - 1"
1281 csit-vpp-perf-1801-all:
1285 title: "csit-vpp-perf-1801-all - 2"
1287 csit-vpp-perf-1801-all:
1291 "vpp-perf-comparison"
1297 nr-of-tests-shown: 20
1300 Advanced data analytics
1301 ```````````````````````
1303 In the future advanced data analytics (ADA) will be added to analyze the
1304 telemetry data collected from SUT telemetry sources and correlate it to
1305 performance test results.
1309 - describe the concept of ADA.
1310 - add specification.
1316 Generates the plots and tables according to the report models per
1317 specification file. The elements are generated using algorithms and data
1318 specified in their models.
1324 - tables are generated by algorithms implemented in PAL, the model includes the
1325 algorithm and all necessary information.
1326 - output format: csv
1327 - generated tables are stored in specified directories and linked to .rst
1334 - `plot.ly <https://plot.ly/>`_ is currently used to generate plots, the model
1335 includes the type of plot and all the necessary information to render it.
1336 - output format: html.
1337 - generated plots are stored in specified directories and linked to .rst files.
1343 Report is generated using Sphinx and Read_the_Docs template. PAL generates html
1344 and pdf formats. It is possible to define the content of the report by
1345 specifying the version (TODO: define the names and content of versions).
1351 1. Read the specification.
1352 2. Read the input data.
1353 3. Process the input data.
1354 4. For element (plot, table, file) defined in specification:
1356 a. Get the data needed to construct the element using a filter.
1357 b. Generate the element.
1358 c. Store the element.
1360 5. Generate the report.
1361 6. Store the report (Nexus).
1363 The process is model driven. The elements' models (tables, plots, files
1364 and report itself) are defined in the specification file. Script reads
1365 the elements' models from specification file and generates the elements.
1367 It is easy to add elements to be generated in the report. If a new type
1368 of an element is required, only a new algorithm needs to be implemented
1372 Continuous Performance Measurements and Trending
1373 ------------------------------------------------
1375 Performance analysis and trending execution sequence:
1376 `````````````````````````````````````````````````````
1378 CSIT PA runs performance analysis, change detection and trending using specified
1379 trend analysis metrics over the rolling window of last <N> sets of historical
1380 measurement data. PA is defined as follows:
1384 #. By PT job at its completion.
1385 #. Manually from Jenkins UI.
1387 #. Download and parse archived historical data and the new data:
1389 #. New data from latest PT job is evaluated against the rolling window
1390 of <N> sets of historical data.
1391 #. Download RF output.xml files and compressed archived data.
1392 #. Parse out the data filtering test cases listed in PA specification
1393 (part of CSIT PAL specification file).
1395 #. Calculate trend metrics for the rolling window of <N> sets of historical
1398 #. Calculate quartiles Q1, Q2, Q3.
1399 #. Trim outliers using IQR.
1400 #. Calculate TMA and TMSD.
1401 #. Calculate normal trending range per test case based on TMA and TMSD.
1403 #. Evaluate new test data against trend metrics:
1405 #. If within the range of (TMA +/- 3*TMSD) => Result = Pass,
1407 #. If below the range => Result = Fail, Reason = Regression.
1408 #. If above the range => Result = Pass, Reason = Progression.
1410 #. Generate and publish results
1412 #. Relay evaluation result to job result.
1413 #. Generate a new set of trend analysis summary graphs and drill-down
1416 #. Summary graphs to include measured values with Normal,
1417 Progression and Regression markers. MM shown in the background if
1419 #. Drill-down graphs to include MM, TMA and TMSD.
1421 #. Publish trend analysis graphs in html format on
1422 https://docs.fd.io/csit/master/trending/.
1425 Parameters to specify:
1426 ``````````````````````
1428 *General section - parameters common to all plots:*
1431 - title: The title of this section;
1432 - output-file-type: only ".html" is supported;
1433 - output-file: path where the generated files will be stored.
1439 - input data for plots;
1441 - job to be monitored - the Jenkins job which results are used as input
1443 - builds used for trending plot(s) - specified by a list of build
1444 numbers or by a range of builds defined by the first and the last
1447 - tests to be displayed in the plot defined by a filter;
1448 - list of parameters to extract from the data;
1457 title: "Continuous Performance Trending and Analysis"
1458 output-file-type: ".html"
1459 output-file: "{DIR[STATIC,VPP]}/cpta"
1462 - title: "VPP 1T1C L2 64B Packet Throughput - Trending"
1463 output-file-name: "l2-1t1c-x520"
1464 data: "plot-performance-trending-vpp"
1465 filter: "'NIC_Intel-X520-DA2' and 'MRR' and '64B' and ('BASE' or 'SCALE') and '1T1C' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST' and not 'MEMIF'"
1468 layout: "plot-cpta-vpp"
1470 - title: "DPDK 4T4C IMIX MRR Trending"
1471 output-file-name: "dpdk-imix-4t4c-xl710"
1472 data: "plot-performance-trending-dpdk"
1473 filter: "'NIC_Intel-XL710' and 'IMIX' and 'MRR' and '4T4C' and 'DPDK'"
1476 layout: "plot-cpta-dpdk"
1481 Performance dashboard tables provide the latest VPP throughput trend, trend
1482 compliance and detected anomalies, all on a per VPP test case basis.
1483 The Dashboard is generated as three tables for 1t1c, 2t2c and 4t4c MRR tests.
1485 At first, the .csv tables are generated (only the table for 1t1c is shown):
1491 title: "Performance trending dashboard"
1492 algorithm: "table_performance_trending_dashboard"
1493 output-file-ext: ".csv"
1494 output-file: "{DIR[STATIC,VPP]}/performance-trending-dashboard-1t1c"
1495 data: "plot-performance-trending-all"
1496 filter: "'MRR' and '1T1C'"
1502 - "tests.vpp.perf.l2.10ge2p1x520-eth-l2bdscale1mmaclrn-mrr.tc01-64b-1t1c-eth-l2bdscale1mmaclrn-ndrdisc"
1505 evaluated-window: 14
1506 long-trend-window: 180
1508 Then, html tables stored inside .rst files are generated:
1514 title: "HTML performance trending dashboard 1t1c"
1515 algorithm: "table_performance_trending_dashboard_html"
1516 input-file: "{DIR[STATIC,VPP]}/performance-trending-dashboard-1t1c.csv"
1517 output-file: "{DIR[STATIC,VPP]}/performance-trending-dashboard-1t1c.rst"
1522 Root Cause Analysis (RCA) by analysing archived performance results – re-analyse
1523 available data for specified:
1525 - range of jobs builds,
1526 - set of specific tests and
1527 - PASS/FAIL criteria to detect performance change.
1529 In addition, PAL generates trending plots to show performance over the specified
1532 Root Cause Analysis - Option 1: Analysing Archived VPP Results
1533 ``````````````````````````````````````````````````````````````
1535 It can be used to speed-up the process, or when the existing data is sufficient.
1536 In this case, PAL uses existing data saved in Nexus, searches for performance
1537 degradations and generates plots to show performance over the specified time
1538 interval for the selected tests.
1543 #. Download and parse archived historical data and the new data.
1544 #. Calculate trend metrics.
1545 #. Find regression / progression.
1546 #. Generate and publish results:
1548 #. Summary graphs to include measured values with Progression and
1550 #. List the DUT build(s) where the anomalies were detected.
1552 CSIT PAL Specification
1553 ''''''''''''''''''''''
1557 - first build (Good); specified by the Jenkins job name and the build
1559 - last build (Bad); specified by the Jenkins job name and the build
1565 - tests of interest; list of tests (full name is used) which results are
1578 List of modules, classes, methods and functions
1579 ```````````````````````````````````````````````
1583 specification_parser.py
1606 input_data_parser.py
1647 Functions implementing algorithms to generate particular types of
1648 tables (called by the function "generate_tables"):
1650 table_performance_improvements
1658 Functions implementing algorithms to generate particular types of
1659 plots (called by the function "generate_plots"):
1660 plot_performance_box
1669 Functions implementing algorithms to generate particular types of
1670 files (called by the function "generate_files"):
1679 Functions implementing algorithms to generate particular types of
1680 report (called by the function "generate_report"):
1681 generate_html_report
1684 Other functions called by the function "generate_report":
1689 PAL functional diagram
1690 ``````````````````````
1698 \graphicspath{{../_tmp/src/csit_framework_documentation/}}
1699 \includegraphics[width=0.90\textwidth]{pal_func_diagram}
1700 \label{fig:pal_func_diagram}
1705 .. figure:: pal_func_diagram.svg
1706 :alt: PAL functional diagram
1710 How to add an element
1711 `````````````````````
1713 Element can be added by adding it's model to the specification file. If
1714 the element is to be generated by an existing algorithm, only it's
1715 parameters must be set.
1717 If a brand new type of element needs to be added, also the algorithm
1718 must be implemented. Element generation algorithms are implemented in
1719 the files with names starting with "generator" prefix. The name of the
1720 function implementing the algorithm and the name of algorithm in the
1721 specification file have to be the same.