X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Fcpta%2Fmethodology%2Findex.rst;h=227dfbcb026bc591c8ea097d4062655a3f66e9f6;hp=29dcae2e7fe1d59a3447978936e686671d42e565;hb=c6aea4422456d455efd0c7ffce94aa0bc0a4dcbf;hpb=cadab3a8d0f465cc5ac8b005d9670b80c102170b diff --git a/docs/cpta/methodology/index.rst b/docs/cpta/methodology/index.rst index 29dcae2e7f..227dfbcb02 100644 --- a/docs/cpta/methodology/index.rst +++ b/docs/cpta/methodology/index.rst @@ -1,10 +1,10 @@ -Performance Trending Methodology -================================ - .. _trending_methodology: -Continuous Trending and Analysis --------------------------------- +Trending Methodology +==================== + +Overview +-------- This document describes a high-level design of a system for continuous performance measuring, trending and change detection for FD.io VPP SW @@ -22,8 +22,8 @@ trending dashboard and graphs with summary and drill-down views across all specified tests that can be reviewed and inspected regularly by FD.io developers and users community. -Performance Trending Tests --------------------------- +Performance Tests +----------------- Performance trending is currently relying on the Maximum Receive Rate (MRR) tests. MRR tests measure the packet forwarding rate under the @@ -51,13 +51,14 @@ Current parameters for performance trending MRR tests: - Trial duration: 10sec. - Execution frequency: twice a day, every 12 hrs (02:00, 14:00 UTC). -In the future if tested VPP configuration can handle the packet rate -higher than bi-directional 10GE link rate, e.g. all IMIX tests and -64B/78B multi-core tests, a higher maximum load will be offered -(25GE|40GE|100GE). +Note: MRR tests should be reporting bi-directional link rate (or NIC +rate, if lower) if tested VPP configuration can handle the packet rate +higher than bi-directional link rate, e.g. large packet tests and/or +multi-core tests. In other words MRR = min(VPP rate, bi-dir link rate, +NIC rate). -Performance Trend Analysis --------------------------- +Trend Analysis +-------------- All measured performance trend data is treated as time-series data that can be modelled using normal distribution. After trimming the outliers, @@ -65,12 +66,11 @@ the median and deviations from median are used for detecting performance change anomalies following the three-sigma rule of thumb (a.k.a. 68-95-99.7 rule). -Analysis Metrics +Metrics ```````````````` -Following statistical metrics are proposed as performance trend -indicators over the rolling window of last sets of historical -measurement data: +Following statistical metrics are used as performance trend indicators +over the rolling window of last sets of historical measurement data: - Q1, Q2, Q3 : Quartiles, three points dividing a ranked data set of values into four equal parts, Q2 is the median of the data. @@ -97,6 +97,8 @@ Outlier Detection Outlier evaluation of test result of value follows the definition from previous section: + :: + Outlier Evaluation Formula Evaluation Result ==================================================== X < (Q1 - 1.5 * IQR) Outlier @@ -109,6 +111,8 @@ To verify compliance of test result of valid value against defined trend metrics and detect anomalies, three simple evaluation formulas are used: + :: + Anomaly Compliance Evaluation Evaluation Formula Confidence Level Result ============================================================================= @@ -129,14 +133,16 @@ ago, TMM[last - 1week] and to the maximum of trend values over last quarter except last week, max(TMM[(last - 3mths)..(last - 1week)]), respectively. This results in following trend compliance calculations: - Trend + :: + + Trend Compliance Metric Change Formula V(alue) R(eference) ============================================================================================= Short-Term Change ((V - R) / R) TMM[last] TMM[last - 1week] Long-Term Change ((V - R) / R) TMM[last] max(TMM[(last - 3mths)..(last - 1week)]) -Performance Trend Presentation ------------------------------- +Trend Presentation +------------------ Performance Dashboard ````````````````````` @@ -168,8 +174,8 @@ data points, representing (trend job build Id, MRR value) and the actual vpp build number (b) tested. -Jenkins Jobs Description ------------------------- +Jenkins Jobs +------------ Performance Trending (PT) ````````````````````````` @@ -179,15 +185,15 @@ data per test case. PT is designed as follows: #. PT job triggers: - #. Periodic e.g. daily. - #. On-demand gerrit triggered. + - Periodic e.g. daily. + - On-demand gerrit triggered. #. Measurements and data calculations per test case: - #. MRR Max Received Rate + - MRR Max Received Rate - #. Measured: Unlimited tolerance of packet loss. - #. Send packets at link rate, count total received packets, divide + - Measured: Unlimited tolerance of packet loss. + - Send packets at link rate, count total received packets, divide by test trial period. #. Archive MRR per test case. @@ -203,18 +209,18 @@ PA is defined as follows: #. PA job triggers: - #. By PT job at its completion. - #. On-demand gerrit triggered. + - By PT job at its completion. + - On-demand gerrit triggered. #. Download and parse archived historical data and the new data: - #. Download RF output.xml files from latest PT job and compressed + - Download RF output.xml files from latest PT job and compressed archived data. - #. Parse out the data filtering test cases listed in PA specification + - Parse out the data filtering test cases listed in PA specification (part of CSIT PAL specification file). - #. Evalute new data from latest PT job against the rolling window of + - Evalute new data from latest PT job against the rolling window of sets of historical data for trendline calculation, anomaly detection and short-term trend compliance. And against long-term trendline metrics for long-term trend compliance. @@ -222,22 +228,23 @@ PA is defined as follows: #. Calculate trend metrics for the rolling window of sets of historical data: - #. Calculate quartiles Q1, Q2, Q3. - #. Trim outliers using IQR. - #. Calculate TMA and TMSD. - #. Calculate normal trending range per test case based on TMM and + - Calculate quartiles Q1, Q2, Q3. + - Trim outliers using IQR. + - Calculate TMA and TMSD. + - Calculate normal trending range per test case based on TMM and TMSD. #. Evaluate new test data against trend metrics: - #. If within the range of (TMA +/- 3*TMSD) => Result = Pass, - Reason = Normal. (to be updated base on final Jenkins code) - #. If below the range => Result = Fail, Reason = Regression. - #. If above the range => Result = Pass, Reason = Progression. + - If within the range of (TMA +/- 3*TMSD) => Result = Pass, + Reason = Normal. (to be updated base on the final Jenkins code). + - If below the range => Result = Fail, Reason = Regression. + - If above the range => Result = Pass, Reason = Progression. #. Generate and publish results - #. Relay evaluation result to job result. (to be updated base on final - Jenkins code) - #. Generate a new set of trend summary dashboard and graphs. - #. Publish trend dashboard and graphs in html format on https://docs.fd.io/. + - Relay evaluation result to job result. (to be updated base on the + final Jenkins code). + - Generate a new set of trend summary dashboard and graphs. + - Publish trend dashboard and graphs in html format on + https://docs.fd.io/.