Doc: Update anomaly methodology documentation

+ Define currently used compliance metrics (e.g. no short term change),
+ Clarify that (and why) column names in Dashboard and Alerting
  are somewhat different from the ones in the methodology document.

Change-Id: Ie091b1981d7578245ca3e700820fab9ac93bbd17
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

diff --git a/docs/cpta/methodology/trend_analysis.rst b/docs/cpta/methodology/trend_analysis.rst
index b98dbe3..5b9ebd3 100644 (file)
--- a/docs/cpta/methodology/trend_analysis.rst
+++ b/docs/cpta/methodology/trend_analysis.rst
@@ -16,35 +16,47 @@ Trend Compliance
 
 .. _Trend_Compliance:
 
-In the text below, "trend at time <t>", shorthand "Trend[t]"
+Trend compliance metrics are targeted to provide an indication of trend
+changes, and hint at their reliability.
+
+There is a difference between compliance metric names used in this document,
+and column names used in :ref:`Dashboard` tables and Alerting emails.
+In cases of low user confusion risk, column names are shortened,
+e.g. Trend instead of Last Trend.
+In cases of high user confusion risk, column names are prolonged,
+e.g. Long-Term Change instead of Trend Change.
+(This document refers to a generic "trend",
+so the compliance metric name is prolonged to Last Trend to avoid confusion.)
+
+The definition of Reference for Trend Change is perhaps surprising.
+It was chosen to allow both positive difference on progression
+(if within last week), but also negative difference on progression
+(if performance was even better somewhere between 3 months and 1 week ago).
+
+In the table below, "trend at time <t>", shorthand "trend[t]"
 means "the group average of the group the sample at time <t> belongs to".
 Here, time is usually given as "last" or last with an offset,
 e.g. "last - 1week".
+Also, "runs[t]" is a shorthand for "number of samples in the group
+the sample at time <t> belongs to".
 
-Trend compliance metrics are targeted to provide an indication of trend
-changes over a short-term (i.e. weekly) and a long-term (i.e.
-quarterly), comparing the last group average Trend[last], to the one from week
-ago, Trend[last - 1week] and to the maximum of trend values over last
-quarter except last week, max(Trend[last - 3mths]..Trend[last - 1week]),
-respectively.
-
-This results in following trend compliance calculations:
-
-+-------------------------+---------------------------------+-------------+-----------------------------------------------+
-| Trend Compliance Metric | Trend Change Formula            | Value       | Reference                                     |
-+=========================+=================================+=============+===============================================+
-| Short-Term Change       | (Value - Reference) / Reference | Trend[last] | Trend[last - 1week]                           |
-+-------------------------+---------------------------------+-------------+-----------------------------------------------+
-| Long-Term Change        | (Value - Reference) / Reference | Trend[last] | max(Trend[last - 3mths]..Trend[last - 1week]) |
-+-------------------------+---------------------------------+-------------+-----------------------------------------------+
+The definitions of compliance metrics:
 
-These metrics are displayed in the Dashboard table.
++-------------------+-------------------+---------------------------------+-------------+-----------------------------------------------+
+| Compliance Metric | Legend Short Name | Formula                         | Value       | Reference                                     |
++===================+===================+=================================+=============+===============================================+
+| Last Trend        | Trend             | trend[last]                     |             |                                               |
++-------------------+-------------------+---------------------------------+-------------+-----------------------------------------------+
+| Number of runs    | Runs              | runs[last]                      |             |                                               |
++-------------------+-------------------+---------------------------------+-------------+-----------------------------------------------+
+| Trend Change      | Long-Term Change  | (Value - Reference) / Reference | trend[last] | max(trend[last - 3mths]..trend[last - 1week]) |
++-------------------+-------------------+---------------------------------+-------------+-----------------------------------------------+
 
 Caveats
 -------
 
-Obviously, is result history is too short, the true Trend[t] value
-may not by available, we use the earliest Trend available instead.
+Obviously, if the result history is too short, the true Trend[t] value
+may not by available. We use the earliest Trend available instead.
 
 The current implementaton does not track time of the samples,
 it counts runs instead.