outlier detection, robust denoisingRobust denoising: understand the breakdown point before you choose an estimator
An outlier is a value that does not fit the pattern you are studying. Outlier detection is the task of naming that mismatch, and robust denoising is the choice of rule used to do it. Every automatic rule encodes an assumption about the pattern. This page compares the three scale estimators in the workspace, their efficiency and breakdown points, and the situations where each one lies to you.
Reproducible procedure
This is the procedure the workspace executes for this method, in order. Each step has an action, a rule that can stop the step, and a defined output.
- 01
Select the rule: IQR fences, the modified Z-score, MAD or Qn, or the STL residual rule.
- Rule
- A rule must publish its efficiency and breakdown point before it is applied to any data.
- Output
- The active rule and its threshold.
- 02
Flag the observations that fail the stated threshold.
- Rule
- An observation is removed only if it fails the rule that was declared, not because it looks unusual.
- Output
- The flagged index set.
- 03
Record the rule, the threshold and the row index for every removed observation.
- Rule
- A removed observation without provenance cannot be reported in a conclusion.
- Output
- The removed-point table.
- 04
Recompute the conclusion with and without the removed observations.
- Rule
- If the conclusion changes, the change is published as a caveat rather than dropped.
- Output
- The before and after comparison and the caveat.
IQR fences
The fences are Q1 − k·IQR and Q3 + k·IQR, with k = 1.5 by default. Quartiles use linear interpolation (R type 7, NumPy default). The rule is transparent and works well for roughly symmetric data. In a strongly skewed distribution it will flag the long tail as outliers, even when those values are genuine.
- What this computes:
- Line the data up smallest to largest, take the middle half (25th–75th percentile); anything too far below or above the fences is an outlier.
- Example:
- Q1 = 10, Q3 = 20, IQR = 10, k = 1.5 → below −5 or above 35 is an outlier.
- Caution:
- In skewed data it may flag a legitimately long tail; raise k or switch to MAD.
Modified Z-score
The modified Z-score measures distance from the median in units of MAD. It resists extreme values, but when MAD is zero every point looks infinitely far away. The workspace refuses to remove anything in that case and tells you why.
- What this computes:
- Distance from the median in units of MAD, more resistant to extremes than the usual Z-score.
- Example:
- median = 50, MAD = 4, x = 70 → 0.6745 × 20 / 4 ≈ 3.37, near the 3.5 threshold.
- Caution:
- MAD near 0 inflates the score; impose a floor.
MAD versus Qn
Both have a 50% breakdown point: almost half the data can be contaminated before the estimate becomes meaningless. Their efficiency on clean Gaussian data differs sharply. MAD keeps 37% of the efficiency of the standard deviation; Qn keeps 82%. When you care about power and the data are not badly contaminated, Qn is the better scale estimator. When you need the simplest possible rule, MAD is easier to explain and audit.
| Estimator | Gaussian efficiency | Breakdown point | Best use |
|---|---|---|---|
| Standard deviation | 100% | 0% | clean, roughly normal data |
| MAD | 37% | 50% | simple robust rule, heavy contamination |
| Qn | 82% | 50% | robust but power-sensitive analysis |
STL residual denoising
For time series, an outlier can be a point that deviates from a locally valid pattern. The STL residual rule compares each residual with the median residual in units of MAD. It needs a decomposition first, and it inherits every problem of a wrong period. Use it after diagnostics, not instead of them.
Robust denoising pipelines and auditability
Each step receives the previous step’s surviving indexes. The final removed set is the union of removals, and every removed point keeps the step id that removed it. If removing outliers changes the trend conclusion, the result says so; the tool does not delete points and then present the cleaned conclusion as if nothing happened.
Common questions
Should I use IQR or MAD for outlier detection?
Use IQR for roughly symmetric data where a transparent fence is enough. Use MAD or the modified Z-score when extreme values would distort the quartiles or when the distribution is heavy-tailed. In strongly skewed data, IQR may flag a legitimate long tail.
What is the difference between MAD and Qn?
Both have a 50% breakdown point. On clean Gaussian data MAD keeps 37% of the efficiency of the standard deviation, while Qn keeps 82%. Qn is the better choice when power matters and contamination is not severe; MAD is simpler to explain and audit.
What modified Z-score threshold should I use?
The conventional threshold is 3.5, using 0.6745·(x − median)/MAD. Lower thresholds remove more points and raise the false-positive rate; higher thresholds keep more extreme values. If MAD is zero, the score is undefined, so this workspace removes nothing and says why.
Can STL residuals detect outliers in a time series?
Yes, after a decomposition: a residual that is far from the median residual in MAD units is a candidate. The rule inherits every problem of a wrong period, so run it after diagnostics and confirm the cycle length first.
Related method notes
Cite this page: Detrend, “Robust denoising: understand the breakdown point before you choose an estimator”, https://detrend.datacorp.top/methods/robust-denoising. Last updated 2026-11-15.
Run this procedure on your own workbook
The workspace executes exactly these steps locally, selects the method from the diagnostics and keeps every alternative visible.
Open the workspace