Skip to content

Add mad_over_time function #5514

@hadjiski

Description

@hadjiski

Proposal

Use case. Why is this important?
MAD using median
MAD using mean
Mean/Median absolute deviation around a central point (mean, median) is a very popular mechanism (MAD), which next to the stddev_over_time can be used to detect anomalies.

Now stddev_over_time is prone to huge outliers, since the standard deviation will increase, the bigger the ourliers are, leading to not being able to detect any anomalies if there are huge outliers.

That's why using one of the MADs would be really beneficial. The median absolute deviation around a median would resist the most huge outliers, whereas the mean absolute deviation around a mean or median would still be more resistant to huge outliers than the standard deviation, but they still will be affected.

I see two options here:
Either only one single mad_over_time is implemented, which implicitly calculates the median absolute deviation around a median, because it is the more robust out of the described above, which according to MAD using median is:

MAD is defined as the median of the absolute deviations from the data's median

Xm = median(X1, X2, ..., Xn)
For each X from (X1, X2, ..., Xn) -> build new set of abs(X - Xm)
MAD = median(new set of absolute deviations)

For example: Consider the data (1, 1, 2, 2, 4, 6, 999). It has a median value of 2.
The absolute deviations from 2 are (1, 1, 0, 0, 2, 4, 997)
which in turn have a median value of 1
(because the sorted absolute deviations are (0, 0, 1, 1, 2, 4, 997)).
So the MAD for this data is 1.

Or we introduce three MAD over time promQL functions, which calculate the:
median absolute deviation around median
mean absolute deviation around median
mean absolute deviation around mean

From the two options I think just one mad_over_time should be enough

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions