# A zero-initialised layer learns only through a nonzero path

> A zero-initialised output layer learns because its weight gradient is the outer product of the error with its input, not because its own weights are nonzero. Put a soft bound after it whose scale is computed as zero at zero, and that gradient is zero forever; guard the limit carelessly, and it is NaN.

- Author: Ajish Pradeep (https://ajishpradeep.com/about/)
- Published: 2026-10-07
- Area: numerics
- Canonical: https://ajishpradeep.com/blog/a-zero-initialised-layer-learns-only-through-a-nonzero-path/
- Cite as: Pradeep, A. (2026). A zero-initialised layer learns only through a nonzero path. https://ajishpradeep.com/blog/a-zero-initialised-layer-learns-only-through-a-nonzero-path/

Initialising the last layer of a new branch to zero is a common trick. A new residual block, adapter or correction head then starts as the identity: it adds nothing, so the network it is attached to behaves exactly as before, and the branch grows in only where training asks for it. ReZero (Bachlechner et al., 2020) and Fixup (Zhang et al., 2019) build whole initialisation schemes on this. I wanted to work out exactly why it learns at all, since a layer whose weights are zero has zero output. Doing that properly also showed me an easy way to break it.

## Why a zero layer still learns

Write the zero-initialised layer as $y = W h + b$, with $W = 0$ and $b = 0$, where $h$ is its input from the layers before. The gradient of the loss with respect to its weights is

$$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial y}\, h^\top$$

This doesn't involve $W$ at all. As long as the error signal $\partial L / \partial y$ and the input $h$ are both nonzero, the first step moves $W$ off zero. The layers before it learn more slowly: their gradient passes through $W^\top$, which is zero, so for the first step they receive nothing. After that the branch learns normally. The chain rule needs a nonzero factor at every link, and the zero layer supplies its gradient from the links on either side of it.

## Where it breaks: a bound after the layer

It breaks as soon as something after the zero layer has a zero derivative at zero output. A natural example is a soft bound on the size of a correction, so that it can never exceed some length $B$:

$$\mathrm{bound}(v) = s(\|v\|)\, v, \qquad s(n) = \frac{B \tanh(n/B)}{n}$$

As $n \to 0$, $\tanh(n/B) \approx n/B$, so $s(n) \to 1$; the bound is the identity for small corrections. But computed literally, $s(0) = 0/0$. The tempting guard is to return 0 when the norm is zero, which is wrong: the limit is 1. With the zero-initialised layer in front, every output is exactly the zero vector at the start, every scale is computed as 0, and the error signal reaching the layer is multiplied by that 0. Its weight gradient is then zero, its weights stay zero, its outputs stay zero, and the scale stays 0. The layer is stuck at initialisation for good.

I checked three implementations on a small regression network with a zero-initialised output layer and the bound with $B = 2$, trained for 400 steps with Adam:

*(An interactive figure is on the web page; every number it shows is stated in the text.)*

| Implementation of the scale at zero | Output-layer weight gradient, step 0 | Loss, step 0 | Loss, step 399 |
|---|---:|---:|---:|
| Guarded to 0 | 0 | 0.271 | 0.284 |
| Guarded to 1, with the division still evaluated | NaN | 0.271 | NaN |
| Guarded to 1, with the division never evaluated at zero | 0.183 | 0.271 | 0.00006 |

The first version never learnt: its weight gradient was exactly zero at every step I logged, and the loss only wandered from batch to batch. The second gives the right forward values, scale 1 at zero, but computes $B \tanh(n/B) / n$ for every element before choosing between the branches. At $n = 0$ that is a NaN, and in the backward pass the unused branch's gradient is multiplied by zero, which leaves it NaN. Its gradients were NaN from the first step and its loss after that. The third version replaces the norm by 1 wherever it is zero before dividing, so no $0/0$ is ever computed, and then selects the limit. It learnt the task: the loss fell from 0.27 to 0.00006.

Numerically the limit is easy to confirm: the scale is 0.9992 at a norm of 0.1, and 0.99999988 at $10^{-3}$.

## What to check when a zero-initialised branch won't move

- The gradient of the zero layer's weights at the first step. If it is exactly zero, something after the layer has a zero derivative at zero output.
- Any function applied after the layer that divides by the output's norm, or by a quantity that is zero at the start, and its value at exactly zero.
- Branch selections around a division, where the division has to be made safe before the selection, not only after it.

A zero-initialised block inserted into a trained network shows the same chain rule at work, learning more slowly than a randomly initialised one, in [a dense predictor cannot place what its cell cannot see](https://ajishpradeep.com/blog/a-dense-predictor-cannot-place-what-its-cell-cannot-see/).

## What I still don't know

Here the bound sat right after the zero layer, where the failure is total and immediate. A function with a small but nonzero derivative at zero would not freeze the layer; it would only slow it down, and I haven't measured how a slow start of that kind interacts with the rest of the network learning around it.

## References

- Fixup Initialization: Residual Learning Without Normalization — Hongyi Zhang, Yann N. Dauphin and Tengyu Ma, ICLR 2019 — https://arxiv.org/abs/1901.09321
- ReZero is All You Need: Fast Convergence at Large Depth — Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell and Julian McAuley, UAI 2021 — https://arxiv.org/abs/2003.04887