-Performance Trending Methodology
-================================
-
.. _trending_methodology:
-Continuous Trending and Analysis
---------------------------------
+Trending Methodology
+====================
+
+Overview
+--------
This document describes a high-level design of a system for continuous
performance measuring, trending and change detection for FD.io VPP SW
all specified tests that can be reviewed and inspected regularly by
FD.io developers and users community.
-Performance Trending Tests
---------------------------
+Performance Tests
+-----------------
Performance trending is currently relying on the Maximum Receive Rate
(MRR) tests. MRR tests measure the packet forwarding rate under the
Current parameters for performance trending MRR tests:
-- Ethernet frame sizes: 64B (78B for IPv6 tests) for all tests, IMIX for
+- **Ethernet frame sizes**: 64B (78B for IPv6 tests) for all tests, IMIX for
selected tests (vhost, memif); all quoted sizes include frame CRC, but
exclude per frame transmission overhead of 20B (preamble, inter frame
gap).
-
-- Maximum load offered: 10GE and 40GE link (sub-)rates depending on NIC
+- **Maximum load offered**: 10GE and 40GE link (sub-)rates depending on NIC
tested, with the actual packet rate depending on frame size,
transmission overhead and traffic generator NIC forwarding capacity.
a 40GE bi-directional link sub-rate limited by TG 40GE NIC used,
XL710.
-- Trial duration: 10sec.
-- Execution frequency: twice a day, every 12 hrs (02:00, 14:00 UTC).
+- **Trial duration**: 10sec.
+- **Execution frequency**: twice a day, every 12 hrs (02:00, 14:00 UTC).
-In the future if tested VPP configuration can handle the packet rate
-higher than bi-directional 10GE link rate, e.g. all IMIX tests and
-64B/78B multi-core tests, a higher maximum load will be offered
-(25GE|40GE|100GE).
+Note: MRR tests should be reporting bi-directional link rate (or NIC
+rate, if lower) if tested VPP configuration can handle the packet rate
+higher than bi-directional link rate, e.g. large packet tests and/or
+multi-core tests. In other words MRR = min(VPP rate, bi-dir link rate,
+NIC rate).
-Performance Trend Analysis
---------------------------
+Trend Analysis
+--------------
All measured performance trend data is treated as time-series data that
can be modelled using normal distribution. After trimming the outliers,
change anomalies following the three-sigma rule of thumb (a.k.a.
68-95-99.7 rule).
-Analysis Metrics
+Metrics
````````````````
-Following statistical metrics are proposed as performance trend
-indicators over the rolling window of last <N> sets of historical
-measurement data:
+Following statistical metrics are used as performance trend indicators
+over the rolling window of last <N> sets of historical measurement data:
-- Q1, Q2, Q3 : Quartiles, three points dividing a ranked data set
- of <N> values into four equal parts, Q2 is the median of the data.
-- IQR = Q3 - Q1 : Inter Quartile Range, measure of variability, used
- here to calculate and eliminate outliers.
-- Outliers : extreme values that are at least (1.5 * IQR) below Q1.
+- **Q1**, **Q2**, **Q3** : **Quartiles**, three points dividing a ranked
+ data set of <N> values into four equal parts, Q2 is the median of the
+ data.
+- **IQR** = Q3 - Q1 : **Inter Quartile Range**, measure of variability,
+ used here to calculate and eliminate outliers.
+- **Outliers** : extreme values that are at least (1.5 * IQR) below Q1.
- Note: extreme values that are at least (1.5 * IQR) above Q3 are not
considered outliers, and are likely to be classified as
progressions.
-- TMA : Trimmed Moving Average, average across the data set of <N>
- values without the outliers. Used here to calculate TMSD.
-- TMSD : Trimmed Moving Standard Deviation, standard deviation over the
- data set of <N> values without the outliers,
- requires calculating TMA. Used for anomaly detection.
-- TMM : Trimmed Moving Median, median across the data set of <N> values
- excluding the outliers. Used as a trending value and as a reference
- for anomaly detection.
+- **TMA** : **Trimmed Moving Average**, average across the data set of
+ <N> values without the outliers. Used here to calculate TMSD.
+- **TMSD** : **Trimmed Moving Standard Deviation**, standard deviation
+ over the data set of <N> values without the outliers, requires
+ calculating TMA. Used for anomaly detection.
+- **TMM** : **Trimmed Moving Median**, median across the data set of <N>
+ values excluding the outliers. Used as a trending value and as a
+ reference for anomaly detection.
Outlier Detection
`````````````````
-Outlier evaluation of test result of value <X> follows the definition
+Outlier evaluation of test result of value *X* follows the definition
from previous section:
- Outlier Evaluation Formula Evaluation Result
- ====================================================
- X < (Q1 - 1.5 * IQR) Outlier
- X >= (Q1 - 1.5 * IQR) Valid (For Trending)
++----------------------------+----------------------+
+| Outlier Evaluation Formula | Evaluation Result |
++============================+======================+
+| X < (Q1 - 1.5 * IQR) | Outlier |
++----------------------------+----------------------+
+| X >= (Q1 - 1.5 * IQR) | Valid (For Trending) |
++----------------------------+----------------------+
Anomaly Detection
`````````````````
trend metrics and detect anomalies, three simple evaluation formulas are
used:
- Anomaly Compliance Evaluation
- Evaluation Formula Confidence Level Result
- =============================================================================
- (TMM - 3 * TMSD) <= X <= (TMM + 3 * TMSD) 99.73% Normal
- X < (TMM - 3 * TMSD) Anomaly Regression
- X > (TMM + 3 * TMSD) Anomaly Progression
++-------------------------------------------+-----------------------------+-------------------+
+| Anomaly Evaluation Formula | Compliance Confidence Level | Evaluation Result |
++===========================================+=============================+===================+
+| (TMM - 3 * TMSD) <= X <= (TMM + 3 * TMSD) | 99.73% | Normal |
++-------------------------------------------+-----------------------------+-------------------+
+| X < (TMM - 3 * TMSD) | Anomaly | Regression |
++-------------------------------------------+-----------------------------+-------------------+
+| X > (TMM + 3 * TMSD) | Anomaly | Progression |
++-------------------------------------------+-----------------------------+-------------------+
TMM is used for the central trend reference point instead of TMA as it
is more robust to anomalies.
quarter except last week, max(TMM[(last - 3mths)..(last - 1week)]),
respectively. This results in following trend compliance calculations:
- Trend
- Compliance Metric Change Formula V(alue) R(eference)
- =============================================================================================
- Short-Term Change ((V - R) / R) TMM[last] TMM[last - 1week]
- Long-Term Change ((V - R) / R) TMM[last] max(TMM[(last - 3mths)..(last - 1week)])
++-------------------------+---------------------------------+-----------+------------------------------------------+
+| Trend Compliance Metric | Trend Change Formula | Value | Reference |
++=========================+=================================+===========+==========================================+
+| Short-Term Change | (Value - Reference) / Reference | TMM[last] | TMM[last - 1week] |
++-------------------------+---------------------------------+-----------+------------------------------------------+
+| Long-Term Change | (Value - Reference) / Reference | TMM[last] | max(TMM[(last - 3mths)..(last - 1week)]) |
++-------------------------+---------------------------------+-----------+------------------------------------------+
-Performance Trend Presentation
-------------------------------
+Trend Presentation
+------------------
Performance Dashboard
`````````````````````
vpp build number (b<XXX>) tested.
-Jenkins Jobs Description
-------------------------
+Jenkins Jobs
+------------
Performance Trending (PT)
`````````````````````````
CSIT PT runs regular performance test jobs measuring and collecting MRR
data per test case. PT is designed as follows:
-#. PT job triggers:
+1. PT job triggers:
- #. Periodic e.g. daily.
- #. On-demand gerrit triggered.
+ a) Periodic e.g. daily.
+ b) On-demand gerrit triggered.
-#. Measurements and data calculations per test case:
+2. Measurements and data calculations per test case:
- #. MRR Max Received Rate
+ a) Max Received Rate (MRR) - send packets at link rate over a trial
+ period, count total received packets, divide by trial period.
- #. Measured: Unlimited tolerance of packet loss.
- #. Send packets at link rate, count total received packets, divide
- by test trial period.
-
-#. Archive MRR per test case.
-#. Archive all counters collected at MRR.
+3. Archive MRR per test case.
+4. Archive all counters collected at MRR.
Performance Analysis (PA)
`````````````````````````
over the rolling window of last <N> sets of historical measurement data.
PA is defined as follows:
-#. PA job triggers:
+1. PA job triggers:
- #. By PT job at its completion.
- #. On-demand gerrit triggered.
+ a) By PT job at its completion.
+ b) On-demand gerrit triggered.
-#. Download and parse archived historical data and the new data:
+2. Download and parse archived historical data and the new data:
- #. Download RF output.xml files from latest PT job and compressed
- archived data.
+ a) Download RF output.xml files from latest PT job and compressed
+ archived data.
+ b) Parse out the data filtering test cases listed in PA specification
+ (part of CSIT PAL specification file).
+ c) Evalute new data from latest PT job against the rolling window of
+ <N> sets of historical data for trendline calculation, anomaly
+ detection and short-term trend compliance. And against long-term
+ trendline metrics for long-term trend compliance.
- #. Parse out the data filtering test cases listed in PA specification
- (part of CSIT PAL specification file).
+3. Calculate trend metrics for the rolling window of <N> sets of
+ historical data:
- #. Evalute new data from latest PT job against the rolling window of
- <N> sets of historical data for trendline calculation, anomaly
- detection and short-term trend compliance. And against long-term
- trendline metrics for long-term trend compliance.
+ a) Calculate quartiles Q1, Q2, Q3.
+ b) Trim outliers using IQR.
+ c) Calculate TMA and TMSD.
+ d) Calculate normal trending range per test case based on TMM and
+ TMSD.
-#. Calculate trend metrics for the rolling window of <N> sets of
- historical data:
+4. Evaluate new test data against trend metrics:
- #. Calculate quartiles Q1, Q2, Q3.
- #. Trim outliers using IQR.
- #. Calculate TMA and TMSD.
- #. Calculate normal trending range per test case based on TMM and
- TMSD.
+ a) If within the range of (TMA +/- 3*TMSD) => Result = Pass,
+ Reason = Normal. (to be updated base on the final Jenkins code).
+ b) If below the range => Result = Fail, Reason = Regression.
+ c) If above the range => Result = Pass, Reason = Progression.
-#. Evaluate new test data against trend metrics:
+5. Generate and publish results
- #. If within the range of (TMA +/- 3*TMSD) => Result = Pass,
- Reason = Normal. (to be updated base on final Jenkins code)
- #. If below the range => Result = Fail, Reason = Regression.
- #. If above the range => Result = Pass, Reason = Progression.
+ a) Relay evaluation result to job result. (to be updated base on the
+ final Jenkins code).
+ b) Generate a new set of trend summary dashboard and graphs.
+ c) Publish trend dashboard and graphs in html format on
+ https://docs.fd.io/.
-#. Generate and publish results
+Testbed HW configuration
+------------------------
- #. Relay evaluation result to job result. (to be updated base on final
- Jenkins code)
- #. Generate a new set of trend summary dashboard and graphs.
- #. Publish trend dashboard and graphs in html format on https://docs.fd.io/.
+The testbed HW configuration is described on
+`this FD.IO wiki page <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed#FD.IO_CSIT_testbed_-_Server_HW_Configuration>`_.
Code Review / csit.git / blobdiff