# Batch norm's running statistics describe a network that has already moved

> With batch normalisation, only the direction of a convolution's weights matters. In my experiments Adam kept rotating that direction after the loss had flattened, and evaluation-mode accuracy fell with the rate of rotation, not with the growth of the weights.

- Author: Ajish Pradeep (https://ajishpradeep.com/about/)
- Published: 2026-10-07
- Area: training
- Canonical: https://ajishpradeep.com/blog/batch-norm-running-statistics-describe-a-network-that-has-moved/
- Cite as: Pradeep, A. (2026). Batch norm's running statistics describe a network that has already moved. https://ajishpradeep.com/blog/batch-norm-running-statistics-describe-a-network-that-has-moved/

Batch normalisation has two modes, and most of us learn only one of them properly. Training mode gets the paper, the diagrams and the intuition about smoother loss surfaces. Evaluation mode gets a line in the documentation: at inference the layer uses running averages instead of batch statistics. That line hides a question I wanted to answer properly. A familiar forum complaint goes like this: someone puts the same validation images through a network in training mode and in evaluation mode, gets perfect accuracy from one and a fraction of it from the other, and asks which one is lying.

The standard answers did not satisfy me. Overfitting cannot produce a gap on identical inputs. Tiny or correlated batches can, and Ioffe (2017) designed batch renormalisation for exactly that case, but they are not the only route. I wanted to know whether the optimizer could matter for a layer whose evaluation mode never sees a gradient.

This is the path I took, including a first explanation that turned out to be wrong.

## What batch norm does to the geometry

I started on paper. In training mode, as Ioffe and Szegedy (2015) defined it, a convolution's output $z$ is normalised per channel with the mean and standard deviation of the current batch:

$$y = \gamma \, \frac{z - \mu_B}{\sigma_B} + \beta$$

Multiply that channel's weights by any $\alpha > 0$. The outputs become $\alpha z$, the batch mean $\alpha \mu_B$, the batch standard deviation $\alpha \sigma_B$, and the factor cancels. Up to the small $\epsilon$ under the square root, the loss cannot see the scale of the weights at all:

$$L(\alpha w) = L(w) \quad \text{for every } \alpha > 0$$

Two consequences fall out of differentiating that identity. With respect to $\alpha$ at $\alpha = 1$ it gives $\langle \nabla L(w), w \rangle = 0$, so the gradient is always orthogonal to the weights. With respect to $w$ it gives $\nabla L(\alpha w) = \nabla L(w)/\alpha$, so the gradient gets smaller as the weights get larger. The layer's function depends only on the direction $w/\|w\|$. This is well-trodden ground: van Laarhoven (2017) and Hoffer et al. (2018) use it to show that weight decay behind a normalisation layer controls an effective learning rate of roughly $\eta/\|w\|^2$ and does not regularise anything.

Orthogonality has a direct effect on the norm. An SGD step $w - \eta g$ with $g \perp w$ can only lengthen the weights, by Pythagoras:

$$\|w_{t+1}\|^2 = \|w_t\|^2 + \eta^2 \|g_t\|^2$$

Adam's step is a different story. It divides each coordinate of the averaged gradient by the root of its averaged square,

$$u_i = \eta \, \frac{\hat m_i}{\sqrt{\hat v_i} + \epsilon}$$

and Kingma and Ba (2015) list among its design properties that it is "invariant to diagonal rescaling of the gradients". A gradient that shrinks as the weights grow therefore does not shrink Adam's step. The per-coordinate division also breaks orthogonality, because a diagonal preconditioner $D$ gives $\langle D g, w \rangle \neq 0$ in general even when $\langle g, w \rangle = 0$. On paper, Adam's steps should lengthen the weights faster than SGD's.

## My first guess, and a run that seemed to confirm it

That gave me a tidy hypothesis. Under Adam the weights inflate, the activations inflate with them, and the running averages, which remember older and smaller activations, fall behind. Bigger weights, staler statistics, worse evaluation.

To test it I built a small convolutional network: three convolutions, each followed by batch normalisation and a ReLU, then a linear classifier. The task is deliberately boring, ten classes made of random 3×32×32 templates buried in heavy noise, so that nothing about the data could be blamed. Batch-norm momentum is 0.03 and the batch size 128. I trained it for 6,000 steps with SGD, with Adam and with AdamW, three seeds each, and checked evaluation-mode accuracy on 4,000 held-out images every 100 steps from step 2,000.

The first result looked like a confirmation. Under SGD the last convolution's weight norm grew by 1.16× and evaluation-mode accuracy averaged 99%. Under Adam the norm grew sixfold and evaluation accuracy averaged 60%, dipping to 10%, which is chance for ten classes. AdamW landed in between. In training mode every run scored 100% on the same held-out images the whole time. I also measured the cosine between each update and the weight vector, expecting Adam's to lean outward, and it did: about +0.04 under Adam against +0.005 under SGD, while the raw gradient's cosine with the weights was zero to four decimal places. Everything fit.

## The test that broke it

A hypothesis that says "the growing norm causes the failure" makes a prediction: stop the norm growing and the failure should go away. Heo et al. (2021) had already built the tool for that. Their AdamP optimizer projects the outward, radial component out of every update, because they found that with momentum optimizers this growth shrinks the effective step size too early. I added the same projection to Adam, per output channel, and reran the three seeds.

The norm behaved as intended and grew only 2× instead of 6×. Evaluation-mode accuracy got worse, falling from 60% to 31%. My explanation had predicted the opposite, so it had to go.

## Measuring the thing the loss actually sees

If only the direction of the weights matters to the network, then the direction is what I should have been measuring. I logged the angle each output channel of the last convolution turned per 100 steps, and added two more Adam runs to vary that angle on purpose, one with a ten times smaller learning rate and one with cosine decay to zero.

*(An interactive figure is on the web page; every number it shows is stated in the text.)*

| Optimizer | Rotation per 100 steps | Norm growth | Evaluation-mode accuracy | Checks below 90% |
|---|---:|---:|---:|---:|
| SGD, lr 0.05, momentum 0.9 | 0.46° | 1.16× | 99.2% | 1% |
| Adam, lr 3e-4 | 1.29° | 1.11× | 99.9% | 1% |
| Adam, lr 3e-3, cosine decay to 0 | 1.78° | 2.16× | 96.7% | 11% |
| AdamW, lr 3e-3, decay 0.05 | 5.65° | 3.35× | 82.6% | 38% |
| Adam, lr 3e-3 | 7.73° | 5.96× | 60.0% | 71% |
| Adam, lr 3e-3, radial part projected out | 24.29° | 2.08× | 30.6% | 96% |

Sorted by rotation, the evaluation accuracy falls in order. The norm column doesn't line up with it at all. The projected run has a small norm and the worst accuracy, because it also turns fastest.

That also explained why the norm had looked guilty. For a step of roughly fixed length, the angle turned is about that length divided by $\|w\|$, so a longer weight vector turns more slowly. For SGD the angle per step is about $\eta\|g_t\|/\|w_t\|$, and both factors work in its favour once the model fits: the gradient falls as the loss flattens, and the norm grows. Adam's step stays near $\eta$ per coordinate however small the gradient gets, so the only brake left is the growing norm. Removing it, which is what the projection did, let the weights spin faster. The growth I had blamed was slowing the damage down. As far as I can tell from their abstract, this does not contradict Heo et al.: they are after faster optimisation, where a larger effective step helps, while here a small step late in training is what keeps the running statistics valid.

The question left was how a fraction of a degree per step could wreck a layer.

## From rotation to a wrong mean

In evaluation mode the layer normalises with running averages, updated after every training batch with momentum $m$:

$$\hat\mu_t = (1 - m)\,\hat\mu_{t-1} + m\,\mu_{B,t}$$

The batch from $k$ steps ago carries weight $m(1-m)^k$, so the average age of the estimate is $(1-m)/m$ steps: about 32 steps at $m = 0.03$, and 9 at PyTorch's default of 0.1. If a channel's true mean drifts at a steady $r$ standard deviations per step, the running estimate trails it by

$$\text{lag} = r \, \frac{1 - m}{m}$$

A channel's pre-activation mean is $\mu_c = w_c^\top \bar p$, where $\bar p$ is the average input patch. After a ReLU every input is non-negative, so $\bar p$ is large and the mean is of the same order as the spread. In my network the median of $|\mu_c|/\sigma_c$ in the middle layer was about 1.7, which means a small rotation of $w_c$ moves $\mu_c$ by a visible fraction of $\sigma_c$. I measured the drift directly on a fixed probe batch. The median was 0.0074 standard deviations per step under Adam and 0.0010 under SGD. Put through the formula with $m = 0.03$, a typical channel's running mean sits about 0.24 standard deviations behind under Adam and 0.03 under SGD, and the fastest channels sit several times further back.

To see whether the lag was really the culprit, I took one of the failing Adam checkpoints and compared each layer's stored statistics with fresh ones computed from the current weights. One middle-layer channel's stored mean was 2 standard deviations off. Then I swapped in fresh statistics for one layer at a time, leaving every weight alone. Replacing the middle layer's statistics on its own lifted evaluation accuracy at the failing checkpoints from as low as 10% to between 82% and 100%. Every mis-normalised layer hands a shifted input to the next one, so a single stale layer can take the whole network down.

## The window test, twice

The lag formula has a second factor, the age $(1-m)/m$, and it makes a sharp prediction. Raising the batch-norm momentum from 0.03 to 0.3 cuts the average age from about 32 steps to about 2. At the same rotation, the failure should mostly go away.

My first run of this test showed no improvement at all, 62% against 60%, which looked like a clean negative result. Then I checked the script and found that it had patched the momentum in a file the experiment never loaded, so it had simply run the old setting again. With the change actually applied, Adam at momentum 0.3 rotated at 6.94° per 100 steps, almost the same as before, and averaged 93.2% in evaluation mode, with 18% of checks below 90% instead of 71%. A negative result is only as good as the check that the change was really applied.

## Fixes, by the factor each one changes

Each fix acts on one factor of the lag, or removes the running statistics altogether. I tested the first three on the toy network. The others follow from the same reasoning, but I have not tested them here.

### Re-estimate the statistics before evaluating

Reset every running mean and variance, run a few hundred training batches forward with no gradient step, and keep their plain average. This makes the lag zero for the weights being evaluated. It brought every final checkpoint in my runs back to 100%. Li et al. (2016) and Schneider et al. (2020) use the same operation to adapt a trained network to shifted inputs. It is cheap enough to run before evaluating, selecting or exporting any checkpoint.

### Let the learning rate decay

A decaying rate drives Adam's rotation, and with it the drift, towards zero. Cosine decay took Adam from 60% to 97%, and a ten times smaller constant rate matched SGD. The flip side is that checkpoints taken while the rate is still high stay exposed, which matters if the best checkpoint is picked from the middle of a run.

### Shorten the window

A larger momentum makes the average younger, as the window test showed. It also makes the average noisier, since fewer batches contribute to it. I only tried one value.

### Untested here: SGD, or no running statistics at all

SGD brakes itself once the gradient vanishes, which the first row of the table reflects. Weight decay on top of Adam, as in AdamW (Loshchilov and Hutter, 2019), helped only partly, which fits the picture: it acts on the norm, and the norm was never the failing quantity. Layer normalisation (Ba et al., 2016) and group normalisation (Wu and He, 2018) compute their statistics from each sample at inference exactly as in training. The weights stay scale-invariant and Adam would still rotate them, but there would be nothing to go stale. I expect the gap to disappear with them; I haven't run it.

## The code

The core of the experiment fits in a page. This version prints both accuracies as it trains. Swapping the optimizer or the momentum reproduces each row.

```python
import torch, torch.nn as nn, torch.nn.functional as F

dev = "cuda" if torch.cuda.is_available() else "cpu"
torch.manual_seed(0)
K = 10
templates = torch.randn(K, 3, 32, 32, device=dev)

def batch(n):
    y = torch.randint(0, K, (n,), device=dev)
    return 0.6 * templates[y] + torch.randn(n, 3, 32, 32, device=dev), y

def block(cin, cout):
    return nn.Sequential(nn.Conv2d(cin, cout, 3, padding=1, bias=False),
                         nn.BatchNorm2d(cout, momentum=0.03), nn.ReLU())

net = nn.Sequential(block(3, 32), nn.MaxPool2d(2), block(32, 64), nn.MaxPool2d(2),
                    block(64, 64), nn.AdaptiveAvgPool2d(1), nn.Flatten(),
                    nn.Linear(64, K)).to(dev)
opt = torch.optim.Adam(net.parameters(), lr=3e-3)
x_val, y_val = batch(4000)

def accuracy(train_mode):
    net.train(train_mode)
    with torch.no_grad():
        acc = (net(x_val).argmax(1) == y_val).float().mean().item()
    net.train()
    return acc

for step in range(6000):
    x, y = batch(128)
    loss = F.cross_entropy(net(x), y)
    opt.zero_grad(); loss.backward(); opt.step()
    if (step + 1) % 500 == 0:
        print(f"step {step + 1}: training mode {accuracy(True):.3f}, "
              f"evaluation mode {accuracy(False):.3f}")
```

## What would have caught it early

The training loss and the gradient norm say nothing about this failure, because by the time it happens the loss is flat and the gradient is small. Three measurements do:

- accuracy on one fixed validation batch in training mode and in evaluation mode, side by side;
- the angle each convolution's output channels turn per hundred steps;
- for one batch, how far each layer's stored running mean sits from the mean the current weights produce, in units of that channel's standard deviation.

When the first shows a gap and the third is large, the statistics are stale, and re-estimating them is the first fix to try.

The same running statistics cause a second, quieter problem when a backbone is frozen: they keep updating while the weights stand still, which I measured in [freezing a layer does not freeze its batch norm](https://ajishpradeep.com/blog/freezing-a-layer-does-not-freeze-its-batch-norm/). And the scale invariance at the heart of this note is one case of a general rule about what a loss term cannot see, worked through in [a shape term cannot constrain what it is invariant to](https://ajishpradeep.com/blog/a-shape-term-cannot-constrain-what-it-is-invariant-to/).

## What I still don't know

This is one small network on a synthetic task, mostly at a constant learning rate, with three seeds per row. It shows the mechanism and which way each fix pushes. It does not give thresholds that carry over to a real model. The exact percentages are not stable either: GPU training is not bit-for-bit reproducible, and these runs amplify the difference, so re-running the same code later gave Adam at 3e-3 an average of 66% instead of 60.0%. The ordering of the rows and the size of the gap to SGD held; a single percentage from this table is one draw. Because most real schedules decay the learning rate, I would expect the effect to show up mostly in checkpoints taken mid-run, but I have not checked that on anything larger.

I measured the link between rotation and evaluation error only through medians. My guess is that a handful of fast-turning channels do most of the damage, and the next thing I want to try is ranking channels by their lag and swapping their statistics one at a time to see how few it takes. I would also like to know how this interacts with an exponential moving average of the weights, which many training recipes evaluate instead of the raw model. That average moves more slowly than the weights, and I don't yet know whether it makes the statistics problem better or worse.

## References

- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — Sergey Ioffe and Christian Szegedy, ICML 2015 — https://arxiv.org/abs/1502.03167
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models — Sergey Ioffe, NeurIPS 2017 — https://arxiv.org/abs/1702.03275
- Adam: A Method for Stochastic Optimization — Diederik P. Kingma and Jimmy Ba, ICLR 2015 — https://arxiv.org/abs/1412.6980
- L2 Regularization versus Batch and Weight Normalization — Twan van Laarhoven, arXiv 2017 — https://arxiv.org/abs/1706.05350
- Norm matters: efficient and accurate normalization schemes in deep networks — Elad Hoffer, Ron Banner, Itay Golan and Daniel Soudry, NeurIPS 2018 — https://arxiv.org/abs/1803.01814
- AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights — Byeongho Heo, Sanghyuk Chun, Seong Joon Oh, Dongyoon Han, Sangdoo Yun, Gyuwan Kim, Youngjung Uh and Jung-Woo Ha, ICLR 2021 — https://arxiv.org/abs/2006.08217
- Decoupled Weight Decay Regularization — Ilya Loshchilov and Frank Hutter, ICLR 2019 — https://arxiv.org/abs/1711.05101
- Revisiting Batch Normalization For Practical Domain Adaptation — Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu and Xiaodi Hou, arXiv 2016 — https://arxiv.org/abs/1603.04779
- Improving robustness against common corruptions by covariate shift adaptation — Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel and Matthias Bethge, NeurIPS 2020 — https://arxiv.org/abs/2006.16971
- Layer Normalization — Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E. Hinton, arXiv 2016 — https://arxiv.org/abs/1607.06450
- Group Normalization — Yuxin Wu and Kaiming He, ECCV 2018 — https://arxiv.org/abs/1803.08494