# How far a pretrained backbone moves decides what it keeps

> Three fine-tuning recipes reached the same 100% accuracy on a new task in my experiment, and differed by up to 13 points on noisier inputs the pretraining had covered. The recipe that let the backbone move furthest kept the least of its robustness.

- Author: Ajish Pradeep (https://ajishpradeep.com/about/)
- Published: 2026-10-07
- Area: training
- Canonical: https://ajishpradeep.com/blog/how-far-a-pretrained-backbone-moves-decides-what-it-keeps/
- Cite as: Pradeep, A. (2026). How far a pretrained backbone moves decides what it keeps. https://ajishpradeep.com/blog/how-far-a-pretrained-backbone-moves-decides-what-it-keeps/

Fine-tuning a pretrained network has two moving parts: the backbone, which arrives with useful features, and a new head, which arrives with nothing. With one learning rate for both, the head starts from random weights and sends large, uninformed gradients back into the backbone during the first steps. I wanted to see how far that pushes the backbone, and what it costs, measured on something the new task's own validation set can't show.

## The setup

Two tasks share the same building blocks. I made 40 smooth random patterns and built each class of task A and task B as a sparse combination of them, 20 classes per task, so features that recognise the patterns transfer from one task to the other. A small convolutional network with batch normalisation was pretrained on task A with noise levels varying from 0.5 to 2.5. It ended with features that work across that whole range: 97% accuracy on task A even at a noise level of 1.8.

Then I fine-tuned it on task B with low noise only, 0.5, for 1,500 steps of SGD, three seeds and three recipes:

1. one learning rate, 0.05, for the backbone and the new head;
2. a ten times smaller rate for the backbone;
3. the head trained first on the frozen backbone for 300 steps, then everything at one rate, the LP-FT recipe of Kumar et al. (2022).

I measured task B accuracy at the training noise and at noise 1.8, which the pretraining had seen and the fine-tuning had not. I also measured how far the backbone's weights moved, relative to their size.

*(An interactive figure is on the web page; every number it shows is stated in the text.)*

| Recipe | Task B at training noise | Task B at noise 1.8 | Relative backbone change |
|---|---:|---:|---:|
| One rate | 100% | 72.1% | 0.19 |
| Backbone at a tenth of the rate | 100% | 82.6% | 0.06 |
| Head first, then everything | 100% | 85.2% | 0.07 |

All three recipes solved the new task completely. The validation set built like the training data says they are equal. At the higher noise level they are not: one rate kept 72% accuracy, the other two 83% and 85%. In every seed the order was the same. The backbone moved about three times as far with one rate as with either of the other recipes, and that change is what cost the robustness the pretraining had built.

## Why the head matters so much

At the first step, the new head's weights are random, so the gradient it sends back into the backbone points in directions that have nothing to do with the new task's structure yet. With one learning rate, the backbone takes those steps at full size. It reshapes its features around whatever the random head happened to need, and only then do the two settle together. The low-noise fine-tuning data never shows the features being used at high noise, so nothing in training protects that ability.

The two other recipes remove the problem in different ways. A smaller backbone rate lets the head learn quickly while the backbone barely moves. Training the head first means that by the time the backbone is unfrozen, the head is already sensible: after its 300 frozen steps it scored 100% on task B and 77% at the higher noise. Its gradient into the backbone then asks for small, relevant changes, and the head itself moved only 5% further during the joint phase. Kumar et al. (2022) report the same trade-off on real distribution shifts: across ten datasets, full fine-tuning averaged 2% higher accuracy in-distribution and 7% lower out of distribution than training the head alone, and the head-first recipe got the best of both. Howard and Ruder (2018) made per-layer learning rates a standard part of fine-tuning for the same reason: the parts that already work should move least.

## What I would measure in a fine-tuning run

From this experiment, the measurements I would add:

- the relative change of the backbone and of the head, logged during training, because with one rate the backbone can move far while the in-distribution metric stays perfect;
- an evaluation on inputs the pretraining covered and the fine-tuning data doesn't, since that is where lost features show;
- the head-first recipe or a lower backbone rate as the default, with full one-rate fine-tuning kept for when the new data really is different from anything the backbone saw.

Freezing the backbone completely removes the trade-off and replaces it with another, which [adding an output without moving the old ones](https://ajishpradeep.com/blog/adding-an-output-to-a-trained-network-without-moving-the-old-ones/) measures. A frozen backbone with batch normalisation has one more catch, in [freezing a layer does not freeze its batch norm](https://ajishpradeep.com/blog/freezing-a-layer-does-not-freeze-its-batch-norm/).

## What I still don't know

My tasks are built from the same building blocks, which makes the pretrained features unusually well suited to the new task. When they are less suited, the backbone has to move to do well in-distribution, and the trade-off becomes real: some of the change is needed. I don't know where that crossover sits, and finding it would take tasks with a controllable amount of shared structure. I also measured the change as a single relative norm for the whole backbone. Early and late layers probably move very differently, and per-layer change would say more about which features are lost.

## References

- Universal Language Model Fine-tuning for Text Classification — Jeremy Howard and Sebastian Ruder, ACL 2018 — https://arxiv.org/abs/1801.06146
- Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution — Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma and Percy Liang, ICLR 2022 — https://arxiv.org/abs/2202.10054