# Unlabelled is not negative, and the usual metric hides it

> Treating unlabelled objects as background barely moved average precision in my experiments. It lowered the scores of the objects annotators tend to miss, cutting their recall at a 0.5 threshold from 61% to 26%, and a test set with the same omissions ranked that model first.

- Author: Ajish Pradeep (https://ajishpradeep.com/about/)
- Published: 2026-10-07
- Area: evaluation
- Canonical: https://ajishpradeep.com/blog/unlabelled-is-not-negative-and-the-usual-metric-hides-it/
- Cite as: Pradeep, A. (2026). Unlabelled is not negative, and the usual metric hides it. https://ajishpradeep.com/blog/unlabelled-is-not-negative-and-the-usual-metric-hides-it/

Dense prediction losses usually make an assumption without saying so: every location without a label is background. A detector, a segmentation head or a per-cell classifier computes its loss over the whole grid, and every cell that no annotation covers gets a target of zero. Real annotations are rarely complete. Annotators miss small objects, faint ones and the fifth object in a crowd, and datasets merged from several sources often label different subsets of what is visible.

I wanted to know what that assumption costs, and where the cost shows up. My expectation was a weaker detector. What I found was more specific than that, and harder to see.

## What the loss learns from missing labels, on paper

A per-cell score trained with binary cross-entropy converges to the expected value of its target given the input. That is the property that makes a confidence head report whatever its target means. If an object with appearance $x$ is labelled with probability $c(x)$ and otherwise counted as background, the expected target on such an object is $c(x)$ times the probability that it is an object at all:

$$s^*(x) = c(x)\; P(\text{object} \mid x)$$

Elkan and Noto (2008) derived this for learning from positive and unlabelled data. When $c$ is the same constant everywhere, the scores of all objects shrink by the same factor. Their order doesn't change, so a ranking metric such as average precision barely notices, but every fixed threshold now cuts in a different place. When $c(x)$ depends on the appearance, for example because faint objects are missed more often, the shrinkage is selective: the objects that are hardest to label get the lowest scores.

## Missing at random: the scores shrink, the ranking survives

The task: 64×64 images, each with six crosses to detect and six squares as distractors on a 16×16 grid, objects of random brightness, and Gaussian noise. A small fully convolutional network predicts an objectness score per cell. In the first version I dropped 40% of the cross labels at random in training and treated them as background.

The average precision against complete labels was 0.9998, the same as with full labels. The mean score on crosses was 0.589. That is the label rate, $1 - 0.4 = 0.6$, as the derivation predicts. The model learnt to say "60% likely" about every cross, and ranked them all correctly above the squares. Wu et al. (2019) report the same robustness on a real benchmark: dropping 30% of annotations and labelling them as background cost Faster R-CNN only about 5% on PASCAL VOC.

## Missing by appearance: the hard objects disappear

Real omissions aren't random, so in the second version I made faint crosses lose their label 80% of the time and bright ones 10%, and raised the image noise so that faint crosses are hard to see. I trained four ways, three seeds each:

1. unlabelled crosses counted as background;
2. unlabelled crosses removed from the loss, an oracle that knows where the missing labels are;
3. a teacher model, trained separately on a small fully labelled set, relabels the unlabelled cells: a score of at least 0.8 becomes a positive, 0.3 to 0.8 is ignored, and below 0.3 stays background;
4. full labels, as a reference.

*(An interactive figure is on the web page; every number it shows is stated in the text.)*

| Training | AP, complete labels | Recall at 0.5, faint crosses | Recall at 0.5, bright crosses | Mean score, faint crosses | AP on a test set with the same omissions |
|---|---:|---:|---:|---:|---:|
| Unlabelled as background | 0.944 | 26% | 89% | 0.30 | 0.841 |
| Unlabelled ignored (oracle) | 0.945 | 52% | 96% | 0.52 | 0.836 |
| Teacher: positive, ignore, background | 0.944 | 63% | 98% | 0.64 | 0.824 |
| Full labels | 0.946 | 61% | 97% | 0.59 | 0.834 |

The average precision against complete labels is the same in every row, 0.944 to 0.946. By the headline metric, the missing labels did nothing. At an operating threshold of 0.5, the model trained with unlabelled objects as background found 26% of the faint crosses, against 61% with full labels. Its mean score on them was 0.30. It had learnt, accurately, that a faint cross in its training data was usually labelled background, and it ranked them low enough to fall under any reasonable threshold. Faint crosses still ranked above the squares, which is why the average precision survived.

*(An interactive figure is on the web page; every number it shows is stated in the text.)*

## The evaluation that rewards the omission

The last column is the part I found most uncomfortable. If the test set is annotated the same way as the training set, with the same faint objects missing, then every faint cross the detector finds counts as a false positive. The model that learnt to suppress faint crosses gives up fewer of those, and it scored the highest average precision of the four, 0.841 against 0.834 for the model trained on full labels. It did so in each of the three seeds. A benchmark built with the same blind spot as the training data rewards a model for sharing it.

## What fixed it

Removing the unlabelled crosses from the loss, the oracle row, recovered most of the faint recall, 52%. It needs to know where the missing labels are, which is the information that is missing. The teacher-based relabelling doesn't, and did slightly better than full labels on faint crosses, 63% against 61%. The three-way split matters. Promoting only confident detections to positives avoids teaching the model a teacher's mistakes, and ignoring the uncertain middle band keeps those cells from being forced either way. The cost is a teacher, which here had the advantage of being trained on complete labels from the same kind of images.

## What I would check in a detector trained on incomplete labels

These are the checks I would add:

- recall at the threshold actually deployed, broken down by how hard the objects are to see or label, not only a ranking metric;
- the mean score per difficulty level, since a whole class of objects sitting at the label rate is the signature of systematic omission;
- an evaluation subset annotated more completely than the training data, even a small one, because a test set with the same omissions cannot see the problem.

The proper-scoring argument behind the 0.59 is worked through in [a confidence head learns the target you give it](https://ajishpradeep.com/blog/a-confidence-head-learns-the-target-you-give-it/), and the evaluation trap has a sibling: scoring against a model's labels instead of the truth, in [a pseudo-labelled test set measures agreement with the labeller](https://ajishpradeep.com/blog/a-pseudo-labelled-test-set-measures-agreement-with-the-labeller/).

## What I still don't know

My teacher was trained on clean, complete labels of the same kind of image, which real teachers rarely are. A teacher with its own blind spots would pass them on, and the ignore band would hide some of them. I chose the 0.3 and 0.8 thresholds by hand and only tried one pair. I also made omission depend on a single property, brightness. Real annotators miss objects for several reasons at once, such as size, crowding and occlusion, and I don't know how well a single teacher threshold handles a mixture of those.

## References

- Learning classifiers from only positive and unlabeled data — Charles Elkan and Keith Noto, KDD 2008 — https://doi.org/10.1145/1401890.1401920
- Soft Sampling for Robust Object Detection — Zhe Wu, Navaneeth Bodla, Bharat Singh, Mahyar Najibi, Rama Chellappa and Larry S. Davis, BMVC 2019 — https://arxiv.org/abs/1806.06986