+
+Throughput Speedup Analysis - Multi-Core with Multi-Threading
+'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+
+Throughput Speedup Analysis (TSA) calculates throughput speedup ratios
+for tested 1-, 2- and 4-core multi-threaded VPP configurations using the
+following formula:
+
+::
+
+ N_core_throughput
+ N_core_throughput_speedup = -----------------
+ 1_core_throughput
+
+Multi-core throughput speedup ratios are plotted in grouped bar graphs
+for throughput tests with 64B/78B frame size, with number of cores on
+X-axis and speedup ratio on Y-axis.
+
+For better comparison multiple test results' data sets are plotted per
+each graph:
+
+ - graph type: grouped bars;
+ - graph X-axis: (testcase index, number of cores);
+ - graph Y-axis: speedup factor.
+
+Subset of existing performance tests is covered by TSA graphs.
+
+**Model for TSA:**
+
+::
+
+ -
+ type: "plot"
+ title: "TSA: 64B-*-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
+ algorithm: "plot_throughput_speedup_analysis"
+ output-file-type: ".html"
+ output-file: "{DIR[STATIC,VPP]}/10ge2p1x520-64B-l2-tsa-ndrdisc"
+ data:
+ "plot-throughput-speedup-analysis"
+ filter: "'NIC_Intel-X520-DA2' and '64B' and 'BASE' and 'NDRDISC' and ('L2BDMACSTAT' or 'L2BDMACLRN' or 'L2XCFWD') and not 'VHOST'"
+ parameters:
+ - "throughput"
+ - "parent"
+ - "tags"
+ layout:
+ title: "64B-*-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc"
+ layout:
+ "plot-throughput-speedup-analysis"
+
+
+Comparison of results from two sets of the same test executions
+'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+
+This algorithm enables comparison of results coming from two sets of the
+same test executions. It is used to quantify performance changes across
+all tests after test environment changes e.g. Operating System
+upgrades/patches, Hardware changes.
+
+It is assumed that each set of test executions includes multiple runs
+of the same tests, 10 or more, to verify test results repeatibility and
+to yield statistically meaningful results data.
+
+Comparison results are presented in a table with a specified number of
+the best and the worst relative changes between the two sets. Following table
+columns are defined:
+
+ - name of the test;
+ - throughput mean values of the reference set;
+ - throughput standard deviation of the reference set;
+ - throughput mean values of the set to compare;
+ - throughput standard deviation of the set to compare;
+ - relative change of the mean values.
+
+**The model**
+
+The model specifies:
+
+ - type: "table" - means this section defines a table.
+ - title: Title of the table.
+ - algorithm: Algorithm which is used to generate the table. The other
+ parameters in this section must provide all information needed by the used
+ algorithm.
+ - output-file-ext: Extension of the output file.
+ - output-file: File which the table will be written to.
+ - reference - the builds which are used as the reference for comparison.
+ - compare - the builds which are compared to the reference.
+ - data: Specify the sources, jobs and builds, providing data for generating
+ the table.
+ - filter: Filter based on tags applied on the input data, if "template" is
+ used, filtering is based on the template.
+ - parameters: Only these parameters will be put to the output data
+ structure.
+ - nr-of-tests-shown: Number of the best and the worst tests presented in the
+ table. Use 0 (zero) to present all tests.
+
+*Example:*
+
+::
+
+ -
+ type: "table"
+ title: "Performance comparison"
+ algorithm: "table_performance_comparison"
+ output-file-ext: ".csv"
+ output-file: "{DIR[DTR,PERF,VPP,IMPRV]}/vpp_performance_comparison"
+ reference:
+ title: "csit-vpp-perf-1801-all - 1"
+ data:
+ csit-vpp-perf-1801-all:
+ - 1
+ - 2
+ compare:
+ title: "csit-vpp-perf-1801-all - 2"
+ data:
+ csit-vpp-perf-1801-all:
+ - 1
+ - 2
+ data:
+ "vpp-perf-comparison"
+ filter: "all"
+ parameters:
+ - "name"
+ - "parent"
+ - "throughput"
+ nr-of-tests-shown: 20