X-Git-Url: https://gerrit.fd.io/r/gitweb?p=csit.git;a=blobdiff_plain;f=docs%2Fcpta%2Fmethodology%2Findex.rst;h=349778999ebdd7cb14f75db5f094aa27b053970c;hp=5efdfaae32ee7332af562a5303694a642205d7c6;hb=3dcef45002a1b82c4503ec590d680950930fa193;hpb=1a72adeb35bfd540f882a107ed1007e4a8545dec diff --git a/docs/cpta/methodology/index.rst b/docs/cpta/methodology/index.rst index 5efdfaae32..349778999e 100644 --- a/docs/cpta/methodology/index.rst +++ b/docs/cpta/methodology/index.rst @@ -33,12 +33,11 @@ size is set to the bi-directional link rate. Current parameters for performance trending MRR tests: -- Ethernet frame sizes: 64B (78B for IPv6 tests) for all tests, IMIX for +- **Ethernet frame sizes**: 64B (78B for IPv6 tests) for all tests, IMIX for selected tests (vhost, memif); all quoted sizes include frame CRC, but exclude per frame transmission overhead of 20B (preamble, inter frame gap). - -- Maximum load offered: 10GE and 40GE link (sub-)rates depending on NIC +- **Maximum load offered**: 10GE and 40GE link (sub-)rates depending on NIC tested, with the actual packet rate depending on frame size, transmission overhead and traffic generator NIC forwarding capacity. @@ -48,8 +47,8 @@ Current parameters for performance trending MRR tests: a 40GE bi-directional link sub-rate limited by TG 40GE NIC used, XL710. -- Trial duration: 10sec. -- Execution frequency: twice a day, every 12 hrs (02:00, 14:00 UTC). +- **Trial duration**: 10sec. +- **Execution frequency**: twice a day, every 12 hrs (02:00, 14:00 UTC). Note: MRR tests should be reporting bi-directional link rate (or NIC rate, if lower) if tested VPP configuration can handle the packet rate @@ -72,35 +71,39 @@ Metrics Following statistical metrics are used as performance trend indicators over the rolling window of last sets of historical measurement data: -- Q1, Q2, Q3 : Quartiles, three points dividing a ranked data set - of values into four equal parts, Q2 is the median of the data. -- IQR = Q3 - Q1 : Inter Quartile Range, measure of variability, used - here to calculate and eliminate outliers. -- Outliers : extreme values that are at least (1.5 * IQR) below Q1. +- **Q1**, **Q2**, **Q3** : **Quartiles**, three points dividing a ranked + data set of values into four equal parts, Q2 is the median of the + data. +- **IQR** = Q3 - Q1 : **Inter Quartile Range**, measure of variability, + used here to calculate and eliminate outliers. +- **Outliers** : extreme values that are at least (1.5 * IQR) below Q1. - Note: extreme values that are at least (1.5 * IQR) above Q3 are not considered outliers, and are likely to be classified as progressions. -- TMA : Trimmed Moving Average, average across the data set of - values without the outliers. Used here to calculate TMSD. -- TMSD : Trimmed Moving Standard Deviation, standard deviation over the - data set of values without the outliers, - requires calculating TMA. Used for anomaly detection. -- TMM : Trimmed Moving Median, median across the data set of values - excluding the outliers. Used as a trending value and as a reference - for anomaly detection. +- **TMA** : **Trimmed Moving Average**, average across the data set of + values without the outliers. Used here to calculate TMSD. +- **TMSD** : **Trimmed Moving Standard Deviation**, standard deviation + over the data set of values without the outliers, requires + calculating TMA. Used for anomaly detection. +- **TMM** : **Trimmed Moving Median**, median across the data set of + values excluding the outliers. Used as a trending value and as a + reference for anomaly detection. Outlier Detection ````````````````` -Outlier evaluation of test result of value follows the definition +Outlier evaluation of test result of value *X* follows the definition from previous section: - Outlier Evaluation Formula Evaluation Result - ==================================================== - X < (Q1 - 1.5 * IQR) Outlier - X >= (Q1 - 1.5 * IQR) Valid (For Trending) ++----------------------------+----------------------+ +| Outlier Evaluation Formula | Evaluation Result | ++============================+======================+ +| X < (Q1 - 1.5 * IQR) | Outlier | ++----------------------------+----------------------+ +| X >= (Q1 - 1.5 * IQR) | Valid (For Trending) | ++----------------------------+----------------------+ Anomaly Detection ````````````````` @@ -109,12 +112,15 @@ To verify compliance of test result of valid value against defined trend metrics and detect anomalies, three simple evaluation formulas are used: - Anomaly Compliance Evaluation - Evaluation Formula Confidence Level Result - ============================================================================= - (TMM - 3 * TMSD) <= X <= (TMM + 3 * TMSD) 99.73% Normal - X < (TMM - 3 * TMSD) Anomaly Regression - X > (TMM + 3 * TMSD) Anomaly Progression ++-------------------------------------------+-----------------------------+-------------------+ +| Anomaly Evaluation Formula | Compliance Confidence Level | Evaluation Result | ++===========================================+=============================+===================+ +| (TMM - 3 * TMSD) <= X <= (TMM + 3 * TMSD) | 99.73% | Normal | ++-------------------------------------------+-----------------------------+-------------------+ +| X < (TMM - 3 * TMSD) | Anomaly | Regression | ++-------------------------------------------+-----------------------------+-------------------+ +| X > (TMM + 3 * TMSD) | Anomaly | Progression | ++-------------------------------------------+-----------------------------+-------------------+ TMM is used for the central trend reference point instead of TMA as it is more robust to anomalies. @@ -129,11 +135,13 @@ ago, TMM[last - 1week] and to the maximum of trend values over last quarter except last week, max(TMM[(last - 3mths)..(last - 1week)]), respectively. This results in following trend compliance calculations: - Trend - Compliance Metric Change Formula V(alue) R(eference) - ============================================================================================= - Short-Term Change ((V - R) / R) TMM[last] TMM[last - 1week] - Long-Term Change ((V - R) / R) TMM[last] max(TMM[(last - 3mths)..(last - 1week)]) ++-------------------------+---------------------------------+-----------+------------------------------------------+ +| Trend Compliance Metric | Trend Change Formula | Value | Reference | ++=========================+=================================+===========+==========================================+ +| Short-Term Change | (Value - Reference) / Reference | TMM[last] | TMM[last - 1week] | ++-------------------------+---------------------------------+-----------+------------------------------------------+ +| Long-Term Change | (Value - Reference) / Reference | TMM[last] | max(TMM[(last - 3mths)..(last - 1week)]) | ++-------------------------+---------------------------------+-----------+------------------------------------------+ Trend Presentation ------------------ @@ -177,21 +185,18 @@ Performance Trending (PT) CSIT PT runs regular performance test jobs measuring and collecting MRR data per test case. PT is designed as follows: -#. PT job triggers: - - #. Periodic e.g. daily. - #. On-demand gerrit triggered. +1. PT job triggers: -#. Measurements and data calculations per test case: + a) Periodic e.g. daily. + b) On-demand gerrit triggered. - #. MRR Max Received Rate +2. Measurements and data calculations per test case: - #. Measured: Unlimited tolerance of packet loss. - #. Send packets at link rate, count total received packets, divide - by test trial period. + a) Max Received Rate (MRR) - send packets at link rate over a trial + period, count total received packets, divide by trial period. -#. Archive MRR per test case. -#. Archive all counters collected at MRR. +3. Archive MRR per test case. +4. Archive all counters collected at MRR. Performance Analysis (PA) ````````````````````````` @@ -201,44 +206,48 @@ compliance and anomaly detection using specified trend analysis metrics over the rolling window of last sets of historical measurement data. PA is defined as follows: -#. PA job triggers: +1. PA job triggers: - #. By PT job at its completion. - #. On-demand gerrit triggered. + a) By PT job at its completion. + b) On-demand gerrit triggered. -#. Download and parse archived historical data and the new data: +2. Download and parse archived historical data and the new data: - #. Download RF output.xml files from latest PT job and compressed - archived data. + a) Download RF output.xml files from latest PT job and compressed + archived data. + b) Parse out the data filtering test cases listed in PA specification + (part of CSIT PAL specification file). + c) Evalute new data from latest PT job against the rolling window of + sets of historical data for trendline calculation, anomaly + detection and short-term trend compliance. And against long-term + trendline metrics for long-term trend compliance. - #. Parse out the data filtering test cases listed in PA specification - (part of CSIT PAL specification file). +3. Calculate trend metrics for the rolling window of sets of + historical data: - #. Evalute new data from latest PT job against the rolling window of - sets of historical data for trendline calculation, anomaly - detection and short-term trend compliance. And against long-term - trendline metrics for long-term trend compliance. + a) Calculate quartiles Q1, Q2, Q3. + b) Trim outliers using IQR. + c) Calculate TMA and TMSD. + d) Calculate normal trending range per test case based on TMM and + TMSD. -#. Calculate trend metrics for the rolling window of sets of - historical data: +4. Evaluate new test data against trend metrics: - #. Calculate quartiles Q1, Q2, Q3. - #. Trim outliers using IQR. - #. Calculate TMA and TMSD. - #. Calculate normal trending range per test case based on TMM and - TMSD. + a) If within the range of (TMA +/- 3*TMSD) => Result = Pass, + Reason = Normal. (to be updated base on the final Jenkins code). + b) If below the range => Result = Fail, Reason = Regression. + c) If above the range => Result = Pass, Reason = Progression. -#. Evaluate new test data against trend metrics: +5. Generate and publish results - #. If within the range of (TMA +/- 3*TMSD) => Result = Pass, - Reason = Normal. (to be updated base on the final Jenkins code). - #. If below the range => Result = Fail, Reason = Regression. - #. If above the range => Result = Pass, Reason = Progression. + a) Relay evaluation result to job result. (to be updated base on the + final Jenkins code). + b) Generate a new set of trend summary dashboard and graphs. + c) Publish trend dashboard and graphs in html format on + https://docs.fd.io/. -#. Generate and publish results +Testbed HW configuration +------------------------ - #. Relay evaluation result to job result. (to be updated base on the - final Jenkins code). - #. Generate a new set of trend summary dashboard and graphs. - #. Publish trend dashboard and graphs in html format on - https://docs.fd.io/. +The testbed HW configuration is described on +`this FD.IO wiki page `_.