More edits in trending docs: methodology, dashboard.

Change-Id: I008fd39c57810dcf4cb84e5a9fc4f9adf6923a4f
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>

diff --git a/docs/cpta/introduction/index.rst b/docs/cpta/introduction/index.rst
index 8b3c170..c724c30 100644 (file)
--- a/docs/cpta/introduction/index.rst
+++ b/docs/cpta/introduction/index.rst
@@ -4,18 +4,28 @@ VPP Performance Dashboard
 Description
 -----------
 
 Description
 -----------
 

-Dashboard tables list a summary of latest per test-case VPP Maximum
-Receive Rate (MRR) performance trend, trend compliance metrics and
-detected number of anomalies. Data samples come from the CSIT VPP
-performance trending jobs  executed twice a day, every 12 hrs (02:00,
-14:00 UTC). All trend and anomaly evaluation is based on a rolling
-window of <N=14> data samples, covering last 7 days.
+Performance dashboard tables provide the latest VPP throughput trend,
+trend compliance and detected anomalies, all on a per VPP test case
+basis.  Linked trendline graphs enable further drill-down into the
+trendline compliance, sequence and nature of anomalies, as well as
+pointers to performance test builds/logs and VPP builds.
+
+Performance trending is currently based on the Maximum Receive Rate
+(MRR) tests. MRR tests measure the packet forwarding rate under the
+maximum load offered by traffic generator over a set trial duration,
+regardless of packet loss. See :ref:`trending_methodology` section for
+more detail including trend and anomaly calculations.
+
+Data samples are generated by the CSIT VPP performance trending jobs
+executed twice a day (target start: every 12 hrs, 02:00, 14:00 UTC). All
+trend and anomaly evaluation is based on a rolling window of <N=14> data
+samples, covering last 7 days.

 
 Legend to table:
 
 
 Legend to table:
 

-    - **Test Case** : name of CSIT test case, naming convention in
-      `CSIT wiki <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
-    - **Trend [Mpps]** : last value of trend.
+    - **Test Case** : name of FD.io CSIT test case, naming convention
+      `here <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
+    - **Trend [Mpps]** : last value of performance trend.

     - **Short-Term Change [%]** : Relative change of last trend value
       vs. last week trend value.
     - **Long-Term Change [%]** : Relative change of last trend value vs.
     - **Short-Term Change [%]** : Relative change of last trend value
       vs. last week trend value.
     - **Long-Term Change [%]** : Relative change of last trend value vs.


@@ -24,17 +34,10 @@ Legend to table:
     - **Progressions [#]** : Number of progressions detected.
     - **Outliers [#]** : Number of outliers detected.
 
     - **Progressions [#]** : Number of progressions detected.
     - **Outliers [#]** : Number of outliers detected.
 

-MRR tests measure the packet forwarding rate under the maximum load
-offered by traffic generator over a set trial duration, regardless of
-packet loss.
-
-For more detail about MRR tests, trend and anomaly calculations please
-refer to :ref:`trending_methodology` section.
-

 Tested VPP worker-thread-core combinations (1t1c, 2t2c, 4t4c) are listed
 in separate tables in section 1.x. Followed by trending methodology in
 section 2. and daily trending graphs in sections 3.x. Daily trending
 Tested VPP worker-thread-core combinations (1t1c, 2t2c, 4t4c) are listed
 in separate tables in section 1.x. Followed by trending methodology in
 section 2. and daily trending graphs in sections 3.x. Daily trending

-data used is provided in sections 4.x.
+data used for graphs is provided in sections 4.x.

 
 VPP worker on 1t1c
 ------------------
 
 VPP worker on 1t1c
 ------------------

diff --git a/docs/cpta/methodology/index.rst b/docs/cpta/methodology/index.rst
index 29dcae2..5efdfaa 100644 (file)
--- a/docs/cpta/methodology/index.rst
+++ b/docs/cpta/methodology/index.rst
@@ -1,10 +1,10 @@
-Performance Trending Methodology
-================================
-

 .. _trending_methodology:
 
 .. _trending_methodology:
 

-Continuous Trending and Analysis
---------------------------------
+Trending Methodology
+====================
+
+Overview
+--------

 
 This document describes a high-level design of a system for continuous
 performance measuring, trending and change detection for FD.io VPP SW
 
 This document describes a high-level design of a system for continuous
 performance measuring, trending and change detection for FD.io VPP SW


@@ -22,8 +22,8 @@ trending dashboard and graphs with summary and drill-down views across
 all specified tests that can be reviewed and inspected regularly by
 FD.io developers and users community.
 
 all specified tests that can be reviewed and inspected regularly by
 FD.io developers and users community.
 

-Performance Trending Tests
---------------------------
+Performance Tests
+-----------------

 
 Performance trending is currently relying on the Maximum Receive Rate
 (MRR) tests. MRR tests measure the packet forwarding rate under the
 
 Performance trending is currently relying on the Maximum Receive Rate
 (MRR) tests. MRR tests measure the packet forwarding rate under the


@@ -51,13 +51,14 @@ Current parameters for performance trending MRR tests:
 - Trial duration: 10sec.
 - Execution frequency: twice a day, every 12 hrs (02:00, 14:00 UTC).
 
 - Trial duration: 10sec.
 - Execution frequency: twice a day, every 12 hrs (02:00, 14:00 UTC).
 

-In the future if tested VPP configuration can handle the packet rate
-higher than bi-directional 10GE link rate, e.g. all IMIX tests and
-64B/78B multi-core tests, a higher maximum load will be offered
-(25GE|40GE|100GE).
+Note: MRR tests should be reporting bi-directional link rate (or NIC
+rate, if lower) if tested VPP configuration can handle the packet rate
+higher than bi-directional link rate, e.g. large packet tests and/or
+multi-core tests. In other words MRR = min(VPP rate, bi-dir link rate,
+NIC rate).

 
 

-Performance Trend Analysis
---------------------------
+Trend Analysis
+--------------

 
 All measured performance trend data is treated as time-series data that
 can be modelled using normal distribution. After trimming the outliers,
 
 All measured performance trend data is treated as time-series data that
 can be modelled using normal distribution. After trimming the outliers,


@@ -65,12 +66,11 @@ the median and deviations from median are used for detecting performance
 change anomalies following the three-sigma rule of thumb (a.k.a.
 68-95-99.7 rule).
 
 change anomalies following the three-sigma rule of thumb (a.k.a.
 68-95-99.7 rule).
 

-Analysis Metrics
+Metrics

 ````````````````
 
 ````````````````
 

-Following statistical metrics are proposed as performance trend
-indicators over the rolling window of last <N> sets of historical
-measurement data:
+Following statistical metrics are used as performance trend indicators
+over the rolling window of last <N> sets of historical measurement data:

 
 - Q1, Q2, Q3 : Quartiles, three points dividing a ranked data set
   of <N> values into four equal parts, Q2 is the median of the data.
 
 - Q1, Q2, Q3 : Quartiles, three points dividing a ranked data set
   of <N> values into four equal parts, Q2 is the median of the data.


@@ -135,8 +135,8 @@ respectively. This results in following trend compliance calculations:
   Short-Term Change     ((V - R) / R)     TMM[last]    TMM[last - 1week]
   Long-Term Change      ((V - R) / R)     TMM[last]    max(TMM[(last - 3mths)..(last - 1week)])
 
   Short-Term Change     ((V - R) / R)     TMM[last]    TMM[last - 1week]
   Long-Term Change      ((V - R) / R)     TMM[last]    max(TMM[(last - 3mths)..(last - 1week)])
 

-Performance Trend Presentation
-------------------------------
+Trend Presentation
+------------------

 
 Performance Dashboard
 `````````````````````
 
 Performance Dashboard
 `````````````````````


@@ -168,8 +168,8 @@ data points, representing (trend job build Id, MRR value) and the actual
 vpp build number (b<XXX>) tested.
 
 
 vpp build number (b<XXX>) tested.
 
 

-Jenkins Jobs Description
-------------------------
+Jenkins Jobs
+------------

 
 Performance Trending (PT)
 `````````````````````````
 
 Performance Trending (PT)
 `````````````````````````


@@ -231,13 +231,14 @@ PA is defined as follows:
 #. Evaluate new test data against trend metrics:
 
   #. If within the range of (TMA +/- 3*TMSD) => Result = Pass,
 #. Evaluate new test data against trend metrics:
 
   #. If within the range of (TMA +/- 3*TMSD) => Result = Pass,

-     Reason = Normal. (to be updated base on final Jenkins code)
+     Reason = Normal. (to be updated base on the final Jenkins code).

   #. If below the range => Result = Fail, Reason = Regression.
   #. If above the range => Result = Pass, Reason = Progression.
 
 #. Generate and publish results
 
   #. If below the range => Result = Fail, Reason = Regression.
   #. If above the range => Result = Pass, Reason = Progression.
 
 #. Generate and publish results
 

-  #. Relay evaluation result to job result. (to be updated base on final
-     Jenkins code)
+  #. Relay evaluation result to job result. (to be updated base on the
+     final Jenkins code).

   #. Generate a new set of trend summary dashboard and graphs.
   #. Generate a new set of trend summary dashboard and graphs.

-  #. Publish trend dashboard and graphs in html format on https://docs.fd.io/.
+  #. Publish trend dashboard and graphs in html format on
+     https://docs.fd.io/.