-Performance Trending Methodology
-================================
-
.. _trending_methodology:
-Continuous Trending and Analysis
---------------------------------
+Trending Methodology
+====================
+
+Overview
+--------
This document describes a high-level design of a system for continuous
performance measuring, trending and change detection for FD.io VPP SW
all specified tests that can be reviewed and inspected regularly by
FD.io developers and users community.
-Performance Trending Tests
---------------------------
+Performance Tests
+-----------------
Performance trending is currently relying on the Maximum Receive Rate
(MRR) tests. MRR tests measure the packet forwarding rate under the
- Trial duration: 10sec.
- Execution frequency: twice a day, every 12 hrs (02:00, 14:00 UTC).
-In the future if tested VPP configuration can handle the packet rate
-higher than bi-directional 10GE link rate, e.g. all IMIX tests and
-64B/78B multi-core tests, a higher maximum load will be offered
-(25GE|40GE|100GE).
+Note: MRR tests should be reporting bi-directional link rate (or NIC
+rate, if lower) if tested VPP configuration can handle the packet rate
+higher than bi-directional link rate, e.g. large packet tests and/or
+multi-core tests. In other words MRR = min(VPP rate, bi-dir link rate,
+NIC rate).
-Performance Trend Analysis
---------------------------
+Trend Analysis
+--------------
All measured performance trend data is treated as time-series data that
can be modelled using normal distribution. After trimming the outliers,
change anomalies following the three-sigma rule of thumb (a.k.a.
68-95-99.7 rule).
-Analysis Metrics
+Metrics
````````````````
-Following statistical metrics are proposed as performance trend
-indicators over the rolling window of last <N> sets of historical
-measurement data:
+Following statistical metrics are used as performance trend indicators
+over the rolling window of last <N> sets of historical measurement data:
- Q1, Q2, Q3 : Quartiles, three points dividing a ranked data set
of <N> values into four equal parts, Q2 is the median of the data.
Outlier evaluation of test result of value <X> follows the definition
from previous section:
+ ::
+
Outlier Evaluation Formula Evaluation Result
====================================================
X < (Q1 - 1.5 * IQR) Outlier
trend metrics and detect anomalies, three simple evaluation formulas are
used:
+ ::
+
Anomaly Compliance Evaluation
Evaluation Formula Confidence Level Result
=============================================================================
quarter except last week, max(TMM[(last - 3mths)..(last - 1week)]),
respectively. This results in following trend compliance calculations:
- Trend
+ ::
+
+ Trend
Compliance Metric Change Formula V(alue) R(eference)
=============================================================================================
Short-Term Change ((V - R) / R) TMM[last] TMM[last - 1week]
Long-Term Change ((V - R) / R) TMM[last] max(TMM[(last - 3mths)..(last - 1week)])
-Performance Trend Presentation
-------------------------------
+Trend Presentation
+------------------
Performance Dashboard
`````````````````````
vpp build number (b<XXX>) tested.
-Jenkins Jobs Description
-------------------------
+Jenkins Jobs
+------------
Performance Trending (PT)
`````````````````````````
#. PT job triggers:
- #. Periodic e.g. daily.
- #. On-demand gerrit triggered.
+ - Periodic e.g. daily.
+ - On-demand gerrit triggered.
#. Measurements and data calculations per test case:
- #. MRR Max Received Rate
+ - MRR Max Received Rate
- #. Measured: Unlimited tolerance of packet loss.
- #. Send packets at link rate, count total received packets, divide
+ - Measured: Unlimited tolerance of packet loss.
+ - Send packets at link rate, count total received packets, divide
by test trial period.
#. Archive MRR per test case.
#. PA job triggers:
- #. By PT job at its completion.
- #. On-demand gerrit triggered.
+ - By PT job at its completion.
+ - On-demand gerrit triggered.
#. Download and parse archived historical data and the new data:
- #. Download RF output.xml files from latest PT job and compressed
+ - Download RF output.xml files from latest PT job and compressed
archived data.
- #. Parse out the data filtering test cases listed in PA specification
+ - Parse out the data filtering test cases listed in PA specification
(part of CSIT PAL specification file).
- #. Evalute new data from latest PT job against the rolling window of
+ - Evalute new data from latest PT job against the rolling window of
<N> sets of historical data for trendline calculation, anomaly
detection and short-term trend compliance. And against long-term
trendline metrics for long-term trend compliance.
#. Calculate trend metrics for the rolling window of <N> sets of
historical data:
- #. Calculate quartiles Q1, Q2, Q3.
- #. Trim outliers using IQR.
- #. Calculate TMA and TMSD.
- #. Calculate normal trending range per test case based on TMM and
+ - Calculate quartiles Q1, Q2, Q3.
+ - Trim outliers using IQR.
+ - Calculate TMA and TMSD.
+ - Calculate normal trending range per test case based on TMM and
TMSD.
#. Evaluate new test data against trend metrics:
- #. If within the range of (TMA +/- 3*TMSD) => Result = Pass,
- Reason = Normal. (to be updated base on final Jenkins code)
- #. If below the range => Result = Fail, Reason = Regression.
- #. If above the range => Result = Pass, Reason = Progression.
+ - If within the range of (TMA +/- 3*TMSD) => Result = Pass,
+ Reason = Normal. (to be updated base on the final Jenkins code).
+ - If below the range => Result = Fail, Reason = Regression.
+ - If above the range => Result = Pass, Reason = Progression.
#. Generate and publish results
- #. Relay evaluation result to job result. (to be updated base on final
- Jenkins code)
- #. Generate a new set of trend summary dashboard and graphs.
- #. Publish trend dashboard and graphs in html format on https://docs.fd.io/.
+ - Relay evaluation result to job result. (to be updated base on the
+ final Jenkins code).
+ - Generate a new set of trend summary dashboard and graphs.
+ - Publish trend dashboard and graphs in html format on
+ https://docs.fd.io/.
Code Review / csit.git / blobdiff