Cleaning noisy labels with a local median

Labels that come from another model, from sensors or from a pipeline are often a series: a value per time step, per position, per frame of some sequence. Such series pick up a particular kind of error: a single sample that collapses to a fraction of its true value, while its neighbours are fine. Before training on them, it is tempting to filter those samples out with a simple rule. I wanted to compare the simple rules on a series that also contains a real, large change, because that is where a rule that is too simple does its damage.

The series

Each simulated series has 300 samples. The true value drifts slowly, by about 20%, and somewhere in the series it dips to a third of its usual level for a stretch of a few dozen samples. That dip is real and must survive the cleaning. Every sample carries 5% noise, and 3% of the samples collapse to between 10% and 35% of their true value. I generated 400 series and applied three rules for flagging samples:

1. a global threshold: flag anything below 0.55 times the series' overall median;
2. a local threshold: flag anything below 0.55 times the median of the 15 samples around it;
3. the Hampel filter: flag anything more than three scaled median absolute deviations from the median of the 15 samples around it.

The scaled median absolute deviation, $1.4826 \cdot \mathrm{median}(|x_i - \tilde{x}|)$, estimates the standard deviation of normal data while ignoring a minority of outliers. It is the robust spread estimate behind Hampel's (1974)¹ approach to robust statistics. Leys et al. (2013)² make the case for preferring it to the standard deviation for exactly this reason: outliers inflate the standard deviation and so hide themselves.

Why the global rule fails

The global threshold caught every collapse and then went on to remove most of the real dip. Inside it, the genuine values sit at about a third of the series' median, below the 0.55 cut, so 69% of them were flagged as bad. A rule like this does more than clean the data. It deletes exactly the rare, large real variations that a model most needs to see, and the cleaned training set no longer contains them.

The local rule compares each sample with its own neighbourhood. A collapse is a single low sample among normal ones, so it falls far below its local median. The genuine dip is a stretch of low samples, so the local median follows it down and none of them stands out. Fifteen neighbours is enough for the median to ignore one collapsed sample, and short enough to follow the dip.

The Hampel filter is a local rule too, but symmetric: it flags anything unusually far from the median in either direction. It caught fewer collapses, because some fall within three deviations in a noisy stretch. It also flagged some good samples, the noisy high ones. When the errors have a known direction, as collapses do, a one-sided relative rule is both simpler and stricter.

Why not just smooth?

Smoothing is the other reflex, and it does something worse than missing a collapse: it spreads it. With a moving average over five samples, the clean neighbours of each collapse were pulled away from their true values by four times their own noise level. A median filter would resist this better, but any filter replaces every sample, the good ones too, and leaves the collapse partly in place. Flagging and then dropping or masking the bad samples changes only the samples that are bad.

What I would check before filtering labels

From this simulation, the checks I would make:

• whether the series contains real large changes, and how the rule treats them: count the flags inside known genuine features, not only overall;
• whether errors have a direction, so that a one-sided rule can be used;
• a window long enough to outvote a single bad sample and short enough to follow real changes;
• masking flagged samples out of the loss, so that nothing is invented in their place.

Whether to mask a doubtful label or count it as negative changes what a model learns, in unlabelled is not negative. A per-sequence median is also the basis of the test-time repair in a loss cannot fix what the training data never shows.

What I still don't know

My collapses are isolated single samples. Errors that come in short runs, two or three bad samples in a row, would pull a small window's median towards them, and the window would need to grow, which costs some of its ability to follow real changes. I haven't measured where that breaks. I also chose the 0.55 threshold by hand for this noise level. In practice it should be set from the noise of the clean samples, and seeing how sensitive the results are to it would be the next step.

References

1. The Influence Curve and its Role in Robust Estimation — Frank R. Hampel, Journal of the American Statistical Association 1974
2. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median — Christophe Leys, Christophe Ley, Olivier Klein, Philippe Bernard and Laurent Licata, Journal of Experimental Social Psychology 2013