A pseudo-labelled test set measures agreement with the labeller

When ground truth is expensive, a strong existing model is often used to label data: for training, and sometimes for evaluation as well. Training on pseudo-labels is a known trade-off. Evaluating on them is subtler, because the number that comes out looks like accuracy and is reported like accuracy. I wanted to know exactly what it measures, and whether it can rank models correctly at all.

What the score is, on paper

Let a model's prediction be $m$, the teacher's label $t$ and the truth $y$. The pseudo-labelled score is $P(m = t)$, not $P(m = y)$. Suppose the model is perfect, $m = y$ always. Then its score is $P(y = t)$, which is exactly the teacher's accuracy. A perfect model can score no higher than the teacher, and every point the teacher gets wrong is charged to the model as an error.

A model that copies the teacher's mistakes does better. A model that makes the teacher's mistakes when the teacher is wrong, and is right when the teacher is right, scores 100% on this ruler whatever its true accuracy. The teacher itself is that model. Pseudo-labels reward imitation, and the better a model is than its labeller, the more it is penalised for it.

The experiment

A 20-class image task built from noisy random templates. A teacher network was trained on 2,000 labelled images and then used to label a pool of 30,000 images and a test set of 5,000. Two more networks of the same size were trained on the pool: an imitator on the teacher's labels, and a third model on the pool's true labels. I scored all three on the test set against the truth and against the teacher's labels, three seeds.

The two rulers rank the models in opposite orders. Against the truth, the model trained on true labels is far ahead. Against the teacher's labels it comes last, below the imitator, which is worse than the teacher in reality. Its score of 64.9% sits right where the arithmetic puts a nearly perfect model, at the teacher's own accuracy, 66%. The imitator's 71.9% is the share of the teacher's decisions, right and wrong, that it learnt to reproduce.

Where this shows up

It shows up whenever the labeller is also a contestant, directly or indirectly:

• a benchmark whose labels were produced by one of the models being compared, which then scores that model as perfect;
• a student distilled from a teacher and evaluated on the same teacher's labels, where improvement towards the truth reads as regression;
• two numbers for the same model on two label sets, one human and one model-made, quoted side by side as if they measured the same thing.

Even human labels aren't immune. Northcutt et al. (2021)¹ estimated that at least 3.3% of the test labels across ten widely used benchmarks are wrong, and found that judging models on corrected labels can change which model looks best. Pseudo-labels are the extreme case of the same problem: the label errors are systematic, and they are shared by any model that learnt from the same source.

What I would do with a pseudo-labelled evaluation set

From this experiment, these are the rules I would follow:

• Report it as agreement with the named labeller, never as accuracy.
• Keep a small evaluation set with independent labels, even a few hundred items. With a gap of 94% against 66%, a few hundred true labels are far more than enough to show the right order.
• Never compare against the model that produced the labels, and be wary of any model trained on its outputs.
• Treat a score close to the labeller's estimated accuracy as a model that may be better than the labeller, not as a ceiling reached.

A test set that shares the training data's omissions fails in the same way, in unlabelled is not negative. And when the teacher's outputs are used for training instead of evaluation, the same question of who knows more decides whether they help, in a student gains from a teacher only when the teacher knows more.

What I still don't know

My teacher's mistakes come from limited training data, so they are spread across classes in a fairly ordinary way. A teacher with concentrated blind spots, wrong on one kind of input and nearly perfect elsewhere, would distort the ranking differently: it would only punish models that are better on exactly those inputs. I'd like to see whether a small independent set can be chosen to target such blind spots, so that it corrects the ranking with as few true labels as possible.

References

1. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks — Curtis G. Northcutt, Anish Athalye and Jonas Mueller, NeurIPS 2021 Datasets and Benchmarks