Skip to content
Method note

outlier detection, robust denoisingRobust denoising: understand the breakdown point before you choose an estimator

An outlier is a value that does not fit the pattern you are studying. Outlier detection is the task of naming that mismatch, and robust denoising is the choice of rule used to do it. Every automatic rule encodes an assumption about the pattern. This page compares the three scale estimators in the workspace, their efficiency and breakdown points, and the situations where each one lies to you.

Reproducible procedure

This is the procedure the workspace executes for this method, in order. Each step has an action, a rule that can stop the step, and a defined output.

  1. 01

    Select the rule: IQR fences, the modified Z-score, MAD or Qn, or the STL residual rule.

    Rule
    A rule must publish its efficiency and breakdown point before it is applied to any data.
    Output
    The active rule and its threshold.
  2. 02

    Flag the observations that fail the stated threshold.

    Rule
    An observation is removed only if it fails the rule that was declared, not because it looks unusual.
    Output
    The flagged index set.
  3. 03

    Record the rule, the threshold and the row index for every removed observation.

    Rule
    A removed observation without provenance cannot be reported in a conclusion.
    Output
    The removed-point table.
  4. 04

    Recompute the conclusion with and without the removed observations.

    Rule
    If the conclusion changes, the change is published as a caveat rather than dropped.
    Output
    The before and after comparison and the caveat.

IQR fences

The fences are Q1 − k·IQR and Q3 + k·IQR, with k = 1.5 by default. Quartiles use linear interpolation (R type 7, NumPy default). The rule is transparent and works well for roughly symmetric data. In a strongly skewed distribution it will flag the long tail as outliers, even when those values are genuine.

IQR fences
Q1 − k·IQR and Q3 + k·IQR
What this computes:
Line the data up smallest to largest, take the middle half (25th–75th percentile); anything too far below or above the fences is an outlier.
Example:
Q1 = 10, Q3 = 20, IQR = 10, k = 1.5 → below −5 or above 35 is an outlier.
Caution:
In skewed data it may flag a legitimately long tail; raise k or switch to MAD.

Modified Z-score

The modified Z-score measures distance from the median in units of MAD. It resists extreme values, but when MAD is zero every point looks infinitely far away. The workspace refuses to remove anything in that case and tells you why.

Modified Z-score
0.6745 · (x − median) / MAD
What this computes:
Distance from the median in units of MAD, more resistant to extremes than the usual Z-score.
Example:
median = 50, MAD = 4, x = 70 → 0.6745 × 20 / 4 ≈ 3.37, near the 3.5 threshold.
Caution:
MAD near 0 inflates the score; impose a floor.

MAD versus Qn

Both have a 50% breakdown point: almost half the data can be contaminated before the estimate becomes meaningless. Their efficiency on clean Gaussian data differs sharply. MAD keeps 37% of the efficiency of the standard deviation; Qn keeps 82%. When you care about power and the data are not badly contaminated, Qn is the better scale estimator. When you need the simplest possible rule, MAD is easier to explain and audit.

EstimatorGaussian efficiencyBreakdown pointBest use
Standard deviation100%0%clean, roughly normal data
MAD37%50%simple robust rule, heavy contamination
Qn82%50%robust but power-sensitive analysis

STL residual denoising

For time series, an outlier can be a point that deviates from a locally valid pattern. The STL residual rule compares each residual with the median residual in units of MAD. It needs a decomposition first, and it inherits every problem of a wrong period. Use it after diagnostics, not instead of them.

Robust denoising pipelines and auditability

Each step receives the previous step’s surviving indexes. The final removed set is the union of removals, and every removed point keeps the step id that removed it. If removing outliers changes the trend conclusion, the result says so; the tool does not delete points and then present the cleaned conclusion as if nothing happened.

Common questions

Should I use IQR or MAD for outlier detection?

Use IQR for roughly symmetric data where a transparent fence is enough. Use MAD or the modified Z-score when extreme values would distort the quartiles or when the distribution is heavy-tailed. In strongly skewed data, IQR may flag a legitimate long tail.

What is the difference between MAD and Qn?

Both have a 50% breakdown point. On clean Gaussian data MAD keeps 37% of the efficiency of the standard deviation, while Qn keeps 82%. Qn is the better choice when power matters and contamination is not severe; MAD is simpler to explain and audit.

What modified Z-score threshold should I use?

The conventional threshold is 3.5, using 0.6745·(x − median)/MAD. Lower thresholds remove more points and raise the false-positive rate; higher thresholds keep more extreme values. If MAD is zero, the score is undefined, so this workspace removes nothing and says why.

Can STL residuals detect outliers in a time series?

Yes, after a decomposition: a residual that is far from the median residual in MAD units is a candidate. The rule inherits every problem of a wrong period, so run it after diagnostics and confirm the cycle length first.

Related method notes

Cite this page: Detrend, “Robust denoising: understand the breakdown point before you choose an estimator”, https://detrend.datacorp.top/methods/robust-denoising. Last updated 2026-11-15.

Verification matrixFAQAll method notesRSSllms.txt

Run this procedure on your own workbook

The workspace executes exactly these steps locally, selects the method from the diagnostics and keeps every alternative visible.

Open the workspace