Report 2005: Fix comparison table hdr, add data

Change-Id: I3ef6dfb1b3da4f91c77841536a840a439816e61f
Signed-off-by: Tibor Frank <tifrank@cisco.com>

diff --git a/docs/report/dpdk_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst b/docs/report/dpdk_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst
index 6e14312..ec7a9f3 100644 (file)
--- a/docs/report/dpdk_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst
+++ b/docs/report/dpdk_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst
@@ -1,5 +1,5 @@
-2n-Skx vs. 2n-Clx Testbeds
---------------------------
+2n-Skx vs 2n-Clx Testbeds
+-------------------------

 
 Relative comparison of |dpdk-release| Testpmd and L3fwd packet
 throughput (NDR, PDR and MRR) is calculated for the same tests executed
 
 Relative comparison of |dpdk-release| Testpmd and L3fwd packet
 throughput (NDR, PDR and MRR) is calculated for the same tests executed

diff --git a/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst b/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst
index bafbb38..2eb1e7b 100644 (file)
--- a/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst
+++ b/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst
@@ -1,8 +1,8 @@
 
 .. _dpdk_compare_topologies_3n-Skx_vs_2n-Skx:
 
 
 .. _dpdk_compare_topologies_3n-Skx_vs_2n-Skx:
 

-3n-Skx vs. 2n-Skx Testbeds
---------------------------
+3n-Skx vs 2n-Skx Testbeds
+-------------------------

 
 Relative comparison of |dpdk-release| Testpmd and L3fwd packet
 throughput (NDR, PDR and MRR) is calculated for the same tests executed
 
 Relative comparison of |dpdk-release| Testpmd and L3fwd packet
 throughput (NDR, PDR and MRR) is calculated for the same tests executed

diff --git a/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst b/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst
index e60606a..8b581f9 100644 (file)
--- a/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst
+++ b/docs/report/dpdk_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst
@@ -1,5 +1,5 @@
-3n-Skx vs. 3n-Hsw Testbeds
---------------------------
+3n-Skx vs 3n-Hsw Testbeds
+-------------------------

 
 Relative comparison of |dpdk-release| Testpmd and L3fwd packet
 throughput (NDR, PDR and MRR) is calculated for the same tests executed
 
 Relative comparison of |dpdk-release| Testpmd and L3fwd packet
 throughput (NDR, PDR and MRR) is calculated for the same tests executed

diff --git a/docs/report/dpdk_performance_tests/comparisons/current_vs_previous_release.rst b/docs/report/dpdk_performance_tests/comparisons/current_vs_previous_release.rst
index 4557c43..6358185 100644 (file)
--- a/docs/report/dpdk_performance_tests/comparisons/current_vs_previous_release.rst
+++ b/docs/report/dpdk_performance_tests/comparisons/current_vs_previous_release.rst
@@ -1,8 +1,8 @@
 
 .. _dpdk_compare_current_vs_previous_release:
 
 
 .. _dpdk_compare_current_vs_previous_release:
 

-Current vs. Previous Release
-----------------------------
+Current vs Previous Release
+---------------------------

 
 Relative comparison of DPDK Testpmd and L3fwd packet throughput (NDR,
 PDR and MRR) between |dpdk-release| and |dpdk-release-1| (measured for
 
 Relative comparison of DPDK Testpmd and L3fwd packet throughput (NDR,
 PDR and MRR) between |dpdk-release| and |dpdk-release-1| (measured for

diff --git a/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst b/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst
index 764e198..3df54b3 100644 (file)
--- a/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst
+++ b/docs/report/introduction/methodology_data_plane_throughput/methodology_data_plane_throughput.rst
@@ -75,7 +75,7 @@ trending jobs enabling detection of performance anomalies (regressions,
 progressions) resulting from data plane code changes.
 
 MRR tests are also used for VPP per patch performance jobs verifying
 progressions) resulting from data plane code changes.
 
 MRR tests are also used for VPP per patch performance jobs verifying

-patch performance vs. parent. CSIT reports include MRR throughput
+patch performance vs parent. CSIT reports include MRR throughput

 comparisons between releases and test environments. Small frame sizes
 only (64b/78B, IMIX).
 
 comparisons between releases and test environments. Small frame sizes
 only (64b/78B, IMIX).
 

diff --git a/docs/report/vpp_performance_tests/comparisons/2n-clx_vs_3n-hsw_testbeds.rst b/docs/report/vpp_performance_tests/comparisons/2n-clx_vs_3n-hsw_testbeds.rst
index 5dccf34..4837b64 100644 (file)
--- a/docs/report/vpp_performance_tests/comparisons/2n-clx_vs_3n-hsw_testbeds.rst
+++ b/docs/report/vpp_performance_tests/comparisons/2n-clx_vs_3n-hsw_testbeds.rst
@@ -1,5 +1,5 @@
-2n-Clx vs. 3n-Hsw Testbeds
---------------------------
+2n-Clx vs 3n-Hsw Testbeds
+-------------------------

 
 Relative comparison of |vpp-release| packet throughput (NDR, PDR and
 MRR) is calculated for the same tests executed on 3-Node Skylake (3n-
 
 Relative comparison of |vpp-release| packet throughput (NDR, PDR and
 MRR) is calculated for the same tests executed on 3-Node Skylake (3n-

diff --git a/docs/report/vpp_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst b/docs/report/vpp_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst
index 328fa19..3b9abb8 100644 (file)
--- a/docs/report/vpp_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst
+++ b/docs/report/vpp_performance_tests/comparisons/2n-skx_vs_2n-clx_testbeds.rst
@@ -1,5 +1,5 @@
-2n-Skx vs. 2n-Clx Testbeds
---------------------------
+2n-Skx vs 2n-Clx Testbeds
+-------------------------

 
 Relative comparison of |vpp-release| packet throughput (NDR, PDR and
 MRR) is calculated for the same tests executed on 2-Node Skylake (2n-
 
 Relative comparison of |vpp-release| packet throughput (NDR, PDR and
 MRR) is calculated for the same tests executed on 2-Node Skylake (2n-

diff --git a/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst b/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst
index b71e2bb..bb4a23e 100644 (file)
--- a/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst
+++ b/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_2n-skx_testbeds.rst
@@ -1,8 +1,8 @@
 
 .. _vpp_compare_topologies_3n-Skx_vs_2n-Skx:
 
 
 .. _vpp_compare_topologies_3n-Skx_vs_2n-Skx:
 

-3n-Skx vs. 2n-Skx Testbeds
---------------------------
+3n-Skx vs 2n-Skx Testbeds
+-------------------------

 
 Relative comparison of |vpp-release| packet throughput (NDR, PDR and
 MRR) is calculated for the same tests executed on 3-Node Skylake (3n-
 
 Relative comparison of |vpp-release| packet throughput (NDR, PDR and
 MRR) is calculated for the same tests executed on 3-Node Skylake (3n-

diff --git a/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst b/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst
index dd7cb03..003f92c 100644 (file)
--- a/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst
+++ b/docs/report/vpp_performance_tests/comparisons/3n-skx_vs_3n-hsw_testbeds.rst
@@ -1,5 +1,5 @@
-3n-Skx vs. 3n-Hsw Testbeds
---------------------------
+3n-Skx vs 3n-Hsw Testbeds
+-------------------------

 
 Relative comparison of |vpp-release| packet throughput (NDR, PDR and
 MRR) is calculated for the same tests executed on 3-Node Skylake (3n-
 
 Relative comparison of |vpp-release| packet throughput (NDR, PDR and
 MRR) is calculated for the same tests executed on 3-Node Skylake (3n-

diff --git a/docs/report/vpp_performance_tests/comparisons/current_vs_previous_release.rst b/docs/report/vpp_performance_tests/comparisons/current_vs_previous_release.rst
index 50c4710..a65e5d1 100644 (file)
--- a/docs/report/vpp_performance_tests/comparisons/current_vs_previous_release.rst
+++ b/docs/report/vpp_performance_tests/comparisons/current_vs_previous_release.rst
@@ -1,8 +1,8 @@
 
 .. _vpp_compare_current_vs_previous_release:
 
 
 .. _vpp_compare_current_vs_previous_release:
 

-Current vs. Previous Release
-----------------------------
+Current vs Previous Release
+---------------------------

 
 Relative comparison of VPP packet throughput (NDR, PDR and MRR) between
 |vpp-release| and |vpp-release-1| (measured for |csit-release| and
 
 Relative comparison of VPP packet throughput (NDR, PDR and MRR) between
 |vpp-release| and |vpp-release-1| (measured for |csit-release| and

diff --git a/docs/report/vpp_performance_tests/comparisons/soak_vs_ndr.rst b/docs/report/vpp_performance_tests/comparisons/soak_vs_ndr.rst
index c219ffd..b4de50b 100644 (file)
--- a/docs/report/vpp_performance_tests/comparisons/soak_vs_ndr.rst
+++ b/docs/report/vpp_performance_tests/comparisons/soak_vs_ndr.rst
@@ -1,9 +1,9 @@
 .. _`soak vs ndr comparison`:
 
 .. _`soak vs ndr comparison`:
 

-Soak Tests vs. NDR Tests
-------------------------
+Soak Tests vs NDR Tests
+-----------------------

 
 

-Relative comparison of |vpp-release| Soak PLRSearch vs. NDR packet throughput
+Relative comparison of |vpp-release| Soak PLRSearch vs NDR packet throughput

 is calculated for the tests executed on 2-Node Skylake physical testbed types,
 in 1-core configurations.
 
 is calculated for the tests executed on 2-Node Skylake physical testbed types,
 in 1-core configurations.
 


@@ -20,11 +20,11 @@ Comparison tables in ASCII and CSV formats:
 2n-skx
 ~~~~~~
 
 2n-skx
 ~~~~~~
 

-  - `ASCII Soak vs. NDR comparison <../../_static/vpp/soak-vs-ndr-2n-skx.txt>`_
-  - `CSV Soak vs. NDR comparison <../../_static/vpp/soak-vs-ndr-2n-skx.csv>`_
+  - `ASCII Soak vs NDR comparison <../../_static/vpp/soak-vs-ndr-2n-skx.txt>`_
+  - `CSV Soak vs NDR comparison <../../_static/vpp/soak-vs-ndr-2n-skx.csv>`_

 
 2n-clx
 ~~~~~~
 
 
 2n-clx
 ~~~~~~
 

-  - `ASCII Soak vs. NDR comparison <../../_static/vpp/soak-vs-ndr-2n-clx.txt>`_
-  - `CSV Soak vs. NDR comparison <../../_static/vpp/soak-vs-ndr-2n-clx.csv>`_
+  - `ASCII Soak vs NDR comparison <../../_static/vpp/soak-vs-ndr-2n-clx.txt>`_
+  - `CSV Soak vs NDR comparison <../../_static/vpp/soak-vs-ndr-2n-clx.csv>`_

diff --git a/resources/tools/presentation/specification.yaml b/resources/tools/presentation/specification.yaml
index a6ff3b0..1772b7c 100644 (file)
--- a/resources/tools/presentation/specification.yaml
+++ b/resources/tools/presentation/specification.yaml
@@ -811,9 +811,9 @@
         - 41  # NDRPDR sel
     vpp-performance-changes-3n-skx-cmp-2:
       csit-vpp-perf-report-iterative-2005-3n-skx:
         - 41  # NDRPDR sel
     vpp-performance-changes-3n-skx-cmp-2:
       csit-vpp-perf-report-iterative-2005-3n-skx:

-        # TODO: Add data

         - 42  # NDRPDR sel env 4
         - 43  # NDRPDR sel env 4
         - 42  # NDRPDR sel env 4
         - 43  # NDRPDR sel env 4

+        - 45  # NDRPDR sel env 4

 
     vpp-performance-changes-2n-skx-h1:
       csit-vpp-perf-verify-1908_1-2n-skx:
 
     vpp-performance-changes-2n-skx-h1:
       csit-vpp-perf-verify-1908_1-2n-skx:


@@ -853,7 +853,6 @@
         - 73  # NDRPDR sel
     vpp-performance-changes-2n-skx-cmp-2:
       csit-vpp-perf-report-iterative-2005-2n-skx:
         - 73  # NDRPDR sel
     vpp-performance-changes-2n-skx-cmp-2:
       csit-vpp-perf-report-iterative-2005-2n-skx:

-        # TODO: Add data

         - 88  # NDRPDR sel env 4
         - 89  # NDRPDR sel env 4
         - 90  # NDRPDR sel env 4
         - 88  # NDRPDR sel env 4
         - 89  # NDRPDR sel env 4
         - 90  # NDRPDR sel env 4


@@ -1013,7 +1012,6 @@
         - 14  # NDRPDR sel
         - 21  # NDRPDR sel
     vpp-performance-changes-3n-tsh-cmp-2:
         - 14  # NDRPDR sel
         - 21  # NDRPDR sel
     vpp-performance-changes-3n-tsh-cmp-2:

-      # TODO: Add data:

       csit-vpp-perf-report-iterative-2005-3n-tsh:
         - 23  # NDRPDR sel env 4
 
       csit-vpp-perf-report-iterative-2005-3n-tsh:
         - 23  # NDRPDR sel env 4
 


@@ -2272,6 +2270,7 @@
       - 41  # NDRPDR sel
       - 42  # NDRPDR sel env 4
       - 43  # NDRPDR sel env 4
       - 41  # NDRPDR sel
       - 42  # NDRPDR sel env 4
       - 43  # NDRPDR sel env 4

+      - 45  # NDRPDR sel env 4

       - 6   # MRR sel
     csit-vpp-perf-report-coverage-2005-3n-skx:
       - 10  # perf-full-3n-skx-crypto_I
       - 6   # MRR sel
     csit-vpp-perf-report-coverage-2005-3n-skx:
       - 10  # perf-full-3n-skx-crypto_I


@@ -3196,10 +3195,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-hsw-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-hsw-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -3208,12 +3207,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-hsw-1t1c-pdr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-hsw-1t1c-pdr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: PDR throughput on 3n-hsw with xl710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 3n-hsw 2t2c pdr
 - type: "table"
 
 # VPP Performance Changes 3n-hsw 2t2c pdr
 - type: "table"


@@ -3232,10 +3233,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-hsw-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-hsw-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -3244,12 +3245,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-hsw-2t2c-pdr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-hsw-2t2c-pdr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: PDR throughput on 3n-hsw with xl710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 3n-hsw 1t1c ndr
 - type: "table"
 
 # VPP Performance Changes 3n-hsw 1t1c ndr
 - type: "table"


@@ -3268,11 +3271,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-hsw-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-hsw-cmp-2"
   comparisons:

-    - title: "Diff1"
-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -3281,12 +3283,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-hsw-1t1c-ndr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-hsw-1t1c-ndr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: NDR throughput on 3n-hsw with xl710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 3n-hsw 2t2c ndr
 - type: "table"
 
 # VPP Performance Changes 3n-hsw 2t2c ndr
 - type: "table"


@@ -3305,10 +3309,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-hsw-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-hsw-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -3317,12 +3321,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-hsw-2t2c-ndr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-hsw-2t2c-ndr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: NDR throughput on 3n-hsw with xl710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 3n-skx 2t1c pdr
 - type: "table"
 
 # VPP Performance Changes 3n-skx 2t1c pdr
 - type: "table"


@@ -3341,10 +3347,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-skx-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-skx-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -3353,12 +3359,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-skx-2t1c-pdr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-skx-2t1c-pdr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: PDR throughput on 3n-skx with xxv710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 3n-skx 4t2c pdr
 - type: "table"
 
 # VPP Performance Changes 3n-skx 4t2c pdr
 - type: "table"


@@ -3377,10 +3385,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-skx-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-skx-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -3389,12 +3397,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-skx-4t2c-pdr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-skx-4t2c-pdr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: PDR throughput on 3n-skx with xxv710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 3n-skx 2t1c ndr
 - type: "table"
 
 # VPP Performance Changes 3n-skx 2t1c ndr
 - type: "table"


@@ -3413,10 +3423,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-skx-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-skx-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -3425,12 +3435,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-skx-2t1c-ndr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-skx-2t1c-ndr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: NDR throughput on 3n-skx with xxv710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 3n-skx 4t2c ndr
 - type: "table"
 
 # VPP Performance Changes 3n-skx 4t2c ndr
 - type: "table"


@@ -3449,10 +3461,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-skx-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-skx-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -3461,12 +3473,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-skx-4t2c-ndr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-skx-4t2c-ndr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: NDR throughput on 3n-skx with xxv710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 2n-skx 2t1c pdr
 - type: "table"
 
 # VPP Performance Changes 2n-skx 2t1c pdr
 - type: "table"


@@ -3485,10 +3499,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-skx-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-skx-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -3497,12 +3511,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-skx-2t1c-pdr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-skx-2t1c-pdr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: PDR throughput on 2n-skx with xxv710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 2n-skx 4t2c pdr
 - type: "table"
 
 # VPP Performance Changes 2n-skx 4t2c pdr
 - type: "table"


@@ -3521,10 +3537,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-skx-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-skx-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -3533,12 +3549,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-skx-4t2c-pdr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-skx-4t2c-pdr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: PDR throughput on 2n-skx with xxv710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 2n-skx 2t1c ndr
 - type: "table"
 
 # VPP Performance Changes 2n-skx 2t1c ndr
 - type: "table"


@@ -3557,10 +3575,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-skx-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-skx-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -3569,12 +3587,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-skx-2t1c-ndr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-skx-2t1c-ndr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: NDR throughput on 2n-skx with xxv710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 2n-skx 4t2c ndr
 - type: "table"
 
 # VPP Performance Changes 2n-skx 4t2c ndr
 - type: "table"


@@ -3593,10 +3613,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-skx-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-skx-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -3605,12 +3625,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-skx-4t2c-ndr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-skx-4t2c-ndr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: NDR throughput on 2n-skx with xxv710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 2n-clx-xxv710 2t1c pdr
 - type: "table"
 
 # VPP Performance Changes 2n-clx-xxv710 2t1c pdr
 - type: "table"


@@ -3627,10 +3649,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-clx-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-clx-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 3
-      compare: 2
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 2
+      compare: 3
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 1
       compare: 3
 #  rca:
       reference: 1
       compare: 3
 #  rca:


@@ -3639,12 +3661,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-clx-2t1c-pdr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-clx-2t1c-pdr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: PDR throughput on 2n-clx with xxv710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 2n-clx-xxv710 4t2c pdr
 - type: "table"
 
 # VPP Performance Changes 2n-clx-xxv710 4t2c pdr
 - type: "table"


@@ -3661,10 +3685,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-clx-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-clx-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 3
-      compare: 2
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 2
+      compare: 3
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 1
       compare: 3
 #  rca:
       reference: 1
       compare: 3
 #  rca:


@@ -3673,12 +3697,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-clx-4t2c-pdr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-clx-4t2c-pdr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: PDR throughput on 2n-clx with xxv710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 2n-clx-xxv710 2t1c ndr
 - type: "table"
 
 # VPP Performance Changes 2n-clx-xxv710 2t1c ndr
 - type: "table"


@@ -3695,10 +3721,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-clx-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-clx-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 3
-      compare: 2
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 2
+      compare: 3
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 1
       compare: 3
 #  rca:
       reference: 1
       compare: 3
 #  rca:


@@ -3707,12 +3733,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-clx-2t1c-ndr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-clx-2t1c-ndr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: NDR throughput on 2n-clx with xxv710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 2n-clx-xxv710 4t2c ndr
 - type: "table"
 
 # VPP Performance Changes 2n-clx-xxv710 4t2c ndr
 - type: "table"


@@ -3729,10 +3757,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-clx-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-2n-clx-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 3
-      compare: 2
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 2
+      compare: 3
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 1
       compare: 3
 #  rca:
       reference: 1
       compare: 3
 #  rca:


@@ -3741,12 +3769,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-clx-4t2c-ndr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-2n-clx-4t2c-ndr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: NDR throughput on 2n-clx with xxv710.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 2n-clx-x556 2t1c pdr
 - type: "table"
 
 # VPP Performance Changes 2n-clx-x556 2t1c pdr
 - type: "table"


@@ -4033,10 +4063,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-tsh-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-tsh-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -4045,12 +4075,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-tsh-1t1c-pdr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-tsh-1t1c-pdr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: PDR throughput on 3n-tsh with x520.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 3n-tsh 2t2c pdr
 - type: "table"
 
 # VPP Performance Changes 3n-tsh 2t2c pdr
 - type: "table"


@@ -4069,10 +4101,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-tsh-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-tsh-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -4081,12 +4113,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-tsh-2t2c-pdr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-tsh-2t2c-pdr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: PDR throughput on 3n-tsh with x520.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 3n-tsh 1t1c ndr
 - type: "table"
 
 # VPP Performance Changes 3n-tsh 1t1c ndr
 - type: "table"


@@ -4105,10 +4139,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-tsh-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-tsh-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -4117,12 +4151,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-tsh-1t1c-ndr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-tsh-1t1c-ndr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: NDR throughput on 3n-tsh with x520.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Performance Changes 3n-tsh 2t2c ndr
 - type: "table"
 
 # VPP Performance Changes 3n-tsh 2t2c ndr
 - type: "table"


@@ -4141,10 +4177,10 @@
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-tsh-cmp-2"
   comparisons:
     - title: "2005.0-4"
       data-set: "vpp-performance-changes-3n-tsh-cmp-2"
   comparisons:

-    - title: "Diff1"
-      reference: 4
-      compare: 3
-    - title: "Diff2"
+    - title: "2005.0-4 vs 2005.0-3"
+      reference: 3
+      compare: 4
+    - title: "2005.0-4 vs 2001.0-2"

       reference: 2
       compare: 4
 #  rca:
       reference: 2
       compare: 4
 #  rca:


@@ -4153,12 +4189,14 @@
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-tsh-2t2c-ndr-rca2.yaml"
   legend:
 #    - title: "RCA2"
 #      data: "rca/rca-2005-3n-tsh-2t2c-ndr-rca2.yaml"
   legend:

-    - "Legend to Table"
-    - 'YYMM.V-E - tested VPP version (YY year, MM month, V minor version) and CSIT test environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
-    - 'Diff1 - difference between environment versions "3" and "4" for the same VPP version, 2005.0-3 relative to 2005.0-4. Version "3" was found to be incompatible with "2" due to different number of TRex workers used.'
-    - 'Diff2 - difference between the current and the previous VPP version in the compatible environment versions "2" and "4", 2005.0-4 relative to 2001.0-2.'
-#    - 'RCA1 - root cause analysis reference for Diff1.'
-#    - 'RCA2 - root cause analysis reference for Diff2.'
+    - 'Legend to Table:'
+    - ''
+    - '           Test Case: Test cases executed for: NDR throughput on 3n-tsh with x520.'
+    - '            YYMM.V-E: [Mpps +- StDev] results for VPP version (YY year, MM month, V minor version) tested in CSIT environment version (E environment version, "2" for 2020-03-27, "3" for 2020-06-11, "4" for 2020-06-24).'
+    - '2005.0-4 vs 2005.0-3: [% +- StDev] Relative difference, latter is the base. Environment version "3" was found to be incompatible with "2" due to higher number of TRex workers used.'
+    - '2005.0-4 vs 2001.0-2: [% +- StDev] Relative difference, latter is the base. Used as a reference to compare benchmark results of VPP current release v20.05.0 vs previous release v20.01.0.'
+#    - '                RCA1: root cause analysis reference for 2005.0-4 vs 2005.0-3.'
+#    - '                RCA2: root cause analysis reference for 2005.0-4 vs 2001.0-2.'

 
 # VPP Comparison Across Testbeds 3n-hsw to 3n-skx 1c ndr
 - type: "table"
 
 # VPP Comparison Across Testbeds 3n-hsw to 3n-skx 1c ndr
 - type: "table"