# Score perfect inputs before scoring a model

> An evaluation pipeline has an error floor of its own, which no model can beat. In my simulation, at sparse sampling, perfect inputs already produced 99% of the error that a good model's inputs did, so almost none of the measured error was the model's.

- Author: Ajish Pradeep (https://ajishpradeep.com/about/)
- Published: 2026-10-07
- Area: evaluation
- Canonical: https://ajishpradeep.com/blog/score-perfect-inputs-before-scoring-a-model/
- Cite as: Pradeep, A. (2026). Score perfect inputs before scoring a model. https://ajishpradeep.com/blog/score-perfect-inputs-before-scoring-a-model/

When a model is evaluated through a pipeline, the error that comes out at the end is usually attributed to the model. The model produces values, the pipeline turns them into the quantity of interest, and the difference from the truth becomes the model's score. But the pipeline makes choices too: where it samples, how it picks, how it rounds. I wanted a way to separate the two, and the simplest one I know is to run the pipeline on perfect inputs first.

## A pipeline with a floor

The task: estimate when a smooth signal reaches its maximum. The signal is a main bump with a smaller, wider bump before it, placed at a random time in an interval of 1,000 time units, about 3 to 8 units wide. The pipeline only sees the signal at sparse, irregular sample times, and it reports the sample time with the largest value. A model supplies those values, here with a 5% error on each.

I ran it three ways, with 3,000 random signals each: the model's values at the sampled times, the true values at the same sampled times, and the true values on a dense grid. The last is the reference, and its error is zero by construction.

*(An interactive figure is on the web page; every number it shows is stated in the text.)*

| Mean spacing between samples | Median error, model values | Median error, perfect values at the same times | 90th percentile, model | 90th percentile, perfect |
|---:|---:|---:|---:|---:|
| 1 | 0.70 | 0.20 | 1.7 | 0.4 |
| 4 | 1.5 | 1.2 | 4.0 | 3.9 |
| 12 | 4.05 | 4.0 | 16.8 | 15.4 |

With dense sampling, the floor is small: perfect values miss by 0.2 units, and the model's noise accounts for most of its 0.7. At a mean spacing of 4, perfect values already miss by 1.2 of the model's 1.5, so 80% of the measured error belongs to the pipeline. At a spacing of 12, perfect values miss by 4.0 against the model's 4.05. A model with no error at all would have scored essentially the same. Every effort spent improving the model would have been invisible in this metric.

## Why the floor appears

The maximum of the true signal falls between samples. Picking the largest sampled value can only return a sample time, so its error is at least the distance from the true maximum to the nearest sample, which grows with the spacing. When samples are irregular, a gap can open exactly around the maximum, and the 90th-percentile errors show it. The model's 5% noise matters only when two samples near the top have nearly equal values, which happens more often at fine spacing, where samples crowd the maximum.

None of this is about the model. The floor is a property of the sampling and the picking rule, and it is the same for every model run through them.

## Running the pipeline on its own reference

The general version of this check is to feed the pipeline the best inputs it can receive, and score them exactly as a model would be scored:

- the true values where they exist, or the reference measurement in place of the model's;
- the same sampling, alignment, picking and rounding as in the real evaluation;
- reported next to every model's score as the floor, with the model's error read as the difference.

A further step is to swap in the perfect version of one stage at a time. In a pipeline with several stages, the stage whose replacement removes most of the error is the one worth fixing, and it may not be the model.

## What I would change in this pipeline

The floor here comes from picking among samples. Fitting a smooth curve to the samples near the top and taking its maximum would place the estimate between samples, lowering the floor. Sampling more densely would too, if it is possible. Either change improves every model's score without touching any model, which is the strongest sign that the floor was the real limit.

A metric can hide a model's real behaviour in other ways too: by averaging over thresholds nobody uses, in [average precision scores thresholds you will never use](https://ajishpradeep.com/blog/average-precision-scores-thresholds-you-will-never-use/), or by being scored against another model's labels, in [a pseudo-labelled test set](https://ajishpradeep.com/blog/a-pseudo-labelled-test-set-measures-agreement-with-the-labeller/).

## What I still don't know

The signal here is smooth and the model's errors are independent from sample to sample. A model whose errors are correlated over time, smooth but shifted, would interact with the picking rule differently, and could move the estimate coherently in one direction. I'd like to know how much of a real model's timing error is that kind of shift, which a floor measured with perfect values cannot show.