Average precision scores thresholds you will never use

Average precision is the default summary for detection and retrieval, and for good reasons: it needs no threshold, it is a single number, and benchmarks such as PASCAL VOC (Everingham et al., 2010¹) and COCO (Lin et al., 2014²) made it the common currency of the field. A model that ships, though, runs at one threshold, and someone chose it. I wanted to see how far apart "best by average precision" and "best at the threshold I will use" can be, and which choices inside the metric move it.

What the number averages

Sort all detections by score. At each true positive in that list, record the precision so far; average precision is the mean of those values, over all the positives:

It is the area under the precision-recall curve, and every recall level counts equally. The last 10% of recall, found at scores no deployed system would accept, weighs as much as the first 10%. Two more choices hide inside it: which detections are pooled into the ranking, and how close a prediction must be to count as a hit. Davis and Goadrich (2006)³ show that the precision-recall curve contains the same information as the ROC curve but weighs it differently, and the same is true of any one number taken from it.

Two detectors whose curves cross

I simulated scores for 2,000 positives and 20,000 negatives from two detectors. A is sharp: confident on 85% of the positives, rarely confident on a negative, but it misses the other 15% outright, giving them very low scores. B finds nearly every positive, but at lower scores, and is confident on more negatives.

Average precision prefers B. At a threshold of 0.5, A's detections are right 96% of the time and B's 75%, for a small difference in recall. At 0.3, B's precision is about a half. Which detector is better depends on what the threshold is and what a false alarm costs, and the single number has averaged that question away. B earns its lead at the far end of the curve: it reaches full recall only by accepting scores down to about 0.05, where its precision has fallen to 17%.

Pooling in a source that has none

Evaluation sets are often pooled across sources, and some sources may not contain the class at all. Every detection the model makes there is a false positive, usually at a low score. Those candidates sit below the deployed threshold, but they still enter the ranking and interleave with the positives that the model scored low.

My first attempt at this didn't show what I expected. With detector A, whose missed positives score lower than everything else, adding 40,000 candidates from an absent source moved average precision only from 0.865 to 0.857. The candidates also reached the 0.3 threshold often enough to cut precision there from 0.83 to 0.76. The effect I was after needs positives with a broad low-score tail, so I made a third detector, E, whose true positives spread from confident to doubtful, and added a growing absent source whose candidates almost never reach 0.3.

Average precision falls steadily with the size of the source that has nothing to find, 2.3 points at 160,000 candidates. At the operating threshold, recall is untouched and precision moves by less than a point. The pooled number blends a model's ranking of genuine low-confidence detections with how many irrelevant candidates happened to be in the evaluation set. Both versions of the experiment are real: when an absent source's candidates do reach the threshold, the precision there falls too, which is worth knowing per source.

The tolerance decides the winner

A detection counts as a hit when it lies within some tolerance of the truth: an overlap threshold for boxes, a distance or a similarity kernel for points. I simulated two point detectors. C finds 90% of the objects with a localisation error of standard deviation 1 pixel per axis. D finds every object, with an error of standard deviation 2.

At tolerances up to 3 pixels C is far ahead. At 5 they are level, and at 8 D wins, because every one of its detections now counts and C's missing 10% cannot be recovered. Nothing about either detector changed between columns. COCO averages over several overlap thresholds partly to avoid committing to one, and that average is itself a choice of weights over the columns.

What I would report for a detector that ships

From these simulations, the numbers I would put next to average precision:

• precision and recall at the deployed threshold, per source, and the threshold itself;
• the tolerance used for matching, and the result at a tighter and a looser one, so that a localisation gap can be told apart from a detection gap;
• whether the pool includes sources without the class, and average precision with and without them.

Before comparing detectors at all, it is worth running the evaluation on perfect inputs, in score perfect inputs before scoring a model. When the labels themselves come from a model, the ranking can invert completely, in a pseudo-labelled test set measures agreement with the labeller.

What I still don't know

These are simulated score distributions chosen to make each effect visible; I haven't measured how often real detectors' curves cross in the region that matters. The tolerance experiment also assumes errors that are the same in every direction and independent of the score, while real detectors are usually more accurate when they are more confident. That correlation would soften the flip in the last table, and I'd like to know by how much.

References

1. The Pascal Visual Object Classes (VOC) Challenge — Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn and Andrew Zisserman, International Journal of Computer Vision 2010
2. Microsoft COCO: Common Objects in Context — Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick and Piotr Dollár, ECCV 2014
3. The relationship between Precision-Recall and ROC curves — Jesse Davis and Mark Goadrich, ICML 2006