-Continuous Performance Trending and Analysis
-============================================
+Continuous Performance Trending
+===============================
-This auto-generated document contains VPP performance trending graphs and data.
-It is generated using CSIT continuous trending test and analysis jobs and is
-updated daily. More detail is available on
-`CSIT Performance Trending and Analysis <https://wiki.fd.io/view/CSIT/PerformanceTrendingAnalysis>`_
-wiki page.
+This document contains VPP performance dashboard, trendline graphs and
+performance test data. It is generated using FD.io CSIT continuous
+performance trending and analysis jobs and is updated daily. See
+:ref:`trending_methodology` section for more details including trend
+analysis and anomaly detection methodology.
Contents
--------
trend compliance and detected anomalies, all on a per VPP test case
basis. Linked trendline graphs enable further drill-down into the
trendline compliance, sequence and nature of anomalies, as well as
-pointers to performance test builds/logs and VPP builds.
-
-Performance trending is currently based on the Maximum Receive Rate
-(MRR) tests. MRR tests measure the packet forwarding rate under the
-maximum load offered by traffic generator over a set trial duration,
-regardless of packet loss. See :ref:`trending_methodology` section for
-more detail including trend and anomaly calculations.
+pointers to performance test builds/logs and VPP builds. Performance
+trending is currently based on the Maximum Receive Rate (MRR) tests. MRR
+tests measure the packet forwarding rate under the maximum load offered
+by traffic generator over a set trial duration, regardless of packet
+loss. See :ref:`trending_methodology` section for more detail including
+trend and anomaly calculations.
Data samples are generated by the CSIT VPP performance trending jobs
executed twice a day (target start: every 12 hrs, 02:00, 14:00 UTC). All
Tested VPP worker-thread-core combinations (1t1c, 2t2c, 4t4c) are listed
in separate tables in section 1.x. Followed by trending methodology in
-section 2. and daily trending graphs in sections 3.x. Daily trending
-data used for graphs is provided in sections 4.x.
+section 2. and trendline graphs in sections 3.x. Performance test data
+used for trendline graphs is provided in sections 4.x.
VPP worker on 1t1c
------------------
Outlier evaluation of test result of value <X> follows the definition
from previous section:
+ ::
+
Outlier Evaluation Formula Evaluation Result
====================================================
X < (Q1 - 1.5 * IQR) Outlier
trend metrics and detect anomalies, three simple evaluation formulas are
used:
+ ::
+
Anomaly Compliance Evaluation
Evaluation Formula Confidence Level Result
=============================================================================
quarter except last week, max(TMM[(last - 3mths)..(last - 1week)]),
respectively. This results in following trend compliance calculations:
- Trend
+ ::
+
+ Trend
Compliance Metric Change Formula V(alue) R(eference)
=============================================================================================
Short-Term Change ((V - R) / R) TMM[last] TMM[last - 1week]
#. PT job triggers:
- #. Periodic e.g. daily.
- #. On-demand gerrit triggered.
+ - Periodic e.g. daily.
+ - On-demand gerrit triggered.
#. Measurements and data calculations per test case:
- #. MRR Max Received Rate
+ - MRR Max Received Rate
- #. Measured: Unlimited tolerance of packet loss.
- #. Send packets at link rate, count total received packets, divide
+ - Measured: Unlimited tolerance of packet loss.
+ - Send packets at link rate, count total received packets, divide
by test trial period.
#. Archive MRR per test case.
#. PA job triggers:
- #. By PT job at its completion.
- #. On-demand gerrit triggered.
+ - By PT job at its completion.
+ - On-demand gerrit triggered.
#. Download and parse archived historical data and the new data:
- #. Download RF output.xml files from latest PT job and compressed
+ - Download RF output.xml files from latest PT job and compressed
archived data.
- #. Parse out the data filtering test cases listed in PA specification
+ - Parse out the data filtering test cases listed in PA specification
(part of CSIT PAL specification file).
- #. Evalute new data from latest PT job against the rolling window of
+ - Evalute new data from latest PT job against the rolling window of
<N> sets of historical data for trendline calculation, anomaly
detection and short-term trend compliance. And against long-term
trendline metrics for long-term trend compliance.
#. Calculate trend metrics for the rolling window of <N> sets of
historical data:
- #. Calculate quartiles Q1, Q2, Q3.
- #. Trim outliers using IQR.
- #. Calculate TMA and TMSD.
- #. Calculate normal trending range per test case based on TMM and
+ - Calculate quartiles Q1, Q2, Q3.
+ - Trim outliers using IQR.
+ - Calculate TMA and TMSD.
+ - Calculate normal trending range per test case based on TMM and
TMSD.
#. Evaluate new test data against trend metrics:
- #. If within the range of (TMA +/- 3*TMSD) => Result = Pass,
+ - If within the range of (TMA +/- 3*TMSD) => Result = Pass,
Reason = Normal. (to be updated base on the final Jenkins code).
- #. If below the range => Result = Fail, Reason = Regression.
- #. If above the range => Result = Pass, Reason = Progression.
+ - If below the range => Result = Fail, Reason = Regression.
+ - If above the range => Result = Pass, Reason = Progression.
#. Generate and publish results
- #. Relay evaluation result to job result. (to be updated base on the
+ - Relay evaluation result to job result. (to be updated base on the
final Jenkins code).
- #. Generate a new set of trend summary dashboard and graphs.
- #. Publish trend dashboard and graphs in html format on
+ - Generate a new set of trend summary dashboard and graphs.
+ - Publish trend dashboard and graphs in html format on
https://docs.fd.io/.
Code Review / csit.git / commitdiff