A dense predictor cannot place what its cell cannot see

Many vision models make their predictions from a grid. An object detector, a heatmap head or an offset regressor assigns each output to one cell of a feature map and reads a small vector from that cell: a class score, a box, an offset to a point. It is an elegant arrangement, and it hides a constraint that I had only ever met as a sentence in lecture notes: a cell can only respond to the part of the image its receptive field covers.

I wanted to see what that constraint looks like when it bites. If a cell has to point to something 20 pixels away and it can only see 10, does the prediction degrade gracefully, or does it fall off a cliff? And when people add context to fix it, with a deeper stack, dilated convolutions or attention, what does each one actually buy? This note follows the experiments I ran to find out, including a first baseline that could see much further than it should have.

The cell's view, on paper

The theoretical receptive field of a cell is the patch of input pixels that can influence it at all. For a stack of convolutions it grows layer by layer. A layer with kernel size $k$ and dilation $d$, sitting on top of layers whose strides multiply to $J$, adds

pixels to the field. My baseline network has four 3×3 convolutions, two of them with stride 2, then one more 3×3 convolution, which gives a field of 21 by 21 pixels at an output stride of 4. A cell can therefore see about 10 pixels in each direction from its centre, and no further.

Luo et al. (2016)¹ showed that this number overstates what a trained network uses. The influence of input pixels on a cell, measured through gradients, falls off roughly like a Gaussian from the centre, and this effective receptive field occupies only a fraction of the theoretical one. I measure it below as the radius that holds 90% of the gradient mass of a cell's output with respect to the input.

A task that needs reach

The task is small and synthetic. Each 64×64 image holds a bright 3×3 square centred on one cell of the 16×16 output grid, and a small target blob at a random angle and a random distance of up to 30 pixels from it. The network reads only the feature vector at the square's cell and regresses the offset to the target. Two extra input channels hold each pixel's x and y coordinates, as in CoordConv (Liu et al., 2018⁴), so features can carry absolute position. Every variant trains for 4,000 steps with Adam and cosine decay, three seeds each. I score the median error on 5,000 held-out images, binned by target distance.

My prediction was simple. Inside about 10 pixels the error would be small. Outside, the cell has nothing to go on, so the error should grow, and I expected it to grow fast.

My first baseline saw too far

The first version of every network had GroupNorm after each convolution, out of habit. The local network, with its 21-pixel field, placed targets 30 pixels away with errors between 0.6 and 5 pixels, depending on the seed, when it should have had no idea where they were. That should be impossible, and the gradient measurement agreed: its effective receptive field covered the whole image, a radius of about 31 pixels.

GroupNorm (Wu and He, 2018⁵) explains it. It normalises each group of channels with a mean and variance computed over every spatial position of the image:

Every cell's output now depends on $\mu_g$ and $\sigma_g$, which summarise the whole map. With coordinate channels in the input, those statistics shift with where the target blob sits, and a cell can read a coarse version of the target's position out of the normalisation. The convolution weights never reach that far; the normaliser does. Instance normalisation and layer normalisation over the spatial axes leak the same way. Batch normalisation leaks across the batch during training, but at inference it uses fixed running averages, so a trained model's cells are local again.

For the rest of the experiments I removed the normalisation, so each cell sees exactly its receptive field and nothing more. I kept the GroupNorm network as an extra row.

The cliff

Errors are median pixel distances, averaged over three seeds. The first error column averages the 0–4 and 4–8 pixel bins, which were nearly equal.

The local network is excellent up to 12 pixels and then fails completely. From the 12–16 pixel bin onward its error is almost exactly the distance to the target, which is what you get from a prediction of no offset at all. There was no gradual decline. A target inside the field gets placed to a tenth of a pixel, and a target just outside it doesn't exist as far as the cell is concerned. The effective radius of 13 pixels is consistent with that: the square field reaches 10 pixels along the axes and 14 along the diagonals.

The GroupNorm row sits between the two regimes. The leak lets the cell reach far targets, but only roughly, through statistics that blur the target's position into a channel mean. The error grows steadily with distance and varied a lot between seeds.

Theoretical reach is not effective reach

The dilated network adds three convolutions with dilations 2, 4 and 8, which pushes the theoretical field to 125 pixels, larger than the image. Its effective radius came out at about 30 pixels, and its error grows with distance: from about 0.3 pixels near the cell to 1.0 at the far edge, and up to 2 in the worst seed. This is the pattern Luo et al. describe. The gradient-weighted field is concentrated near the centre, so the far edge of the theoretical field is weakly used, and predictions that depend on it are less precise. The 64-pixel image also caps how far any field can be measured here, so the 30 pixels is a lower bound on how far this network could reach in a larger image.

Dilated convolutions, introduced for dense prediction by Yu and Koltun (2016)², are a cheap way to grow the field without losing resolution. The table shows their cost: reach that falls off with distance.

Attention gives every cell the whole map

The third network keeps the local convolutions and adds one self-attention block over the 16×16 grid, in the style of the non-local block of Wang et al. (2018)³: every cell forms a query, compares it with keys from every other cell, and adds a weighted sum of their values to its own features. The error was flat, about 0.4 pixels at every distance, including 30 pixels. Reach is no longer a property of the position; any cell can attend to any other.

The price is quadratic in the number of cells. This grid has 256 cells, so the attention matrix has 65,536 entries per image, which is nothing here and a real cost at higher resolutions. That is why architectures tend to place such blocks on the coarsest feature map.

Adding reach to a network that already works

In practice context is often added to a network that has already been trained, and the usual advice is to initialise the new block so that it starts as the identity. ControlNet (Zhang et al., 2023⁶) connects its new branch through "zero convolutions" for this reason, and ReZero (Bachlechner et al.⁷, 2020) gates every residual branch with a parameter that starts at zero. I wanted to know what that buys, so I trained the local network first, inserted the attention block with its output projection either set to zero or initialised randomly, and continued training everything at a lower learning rate.

"Near" is the median error for targets under 10 pixels, "far" for targets beyond 16; three seeds each. The insertion columns come from two separate runs, and the near error with a random block differed between them, hence the range.

At insertion the zero-initialised block changed nothing, to the second decimal: the network computes exactly what it computed before. The random block disturbed it, raising the error on near targets three to five times. That is the guarantee zero initialisation offers.

It also learned more slowly. After 1,500 steps the zero-initialised network still had a median far error above 10 pixels, and in one seed it had barely moved, while the random one was at 4. By 6,000 steps both had learned the task. The chain rule explains the delay. The block adds $y = W_o h$ to the features, where $h$ is the attention output computed from $q$, $k$ and $v$. Every parameter upstream of $h$ receives its gradient through

which is exactly zero while $W_o = 0$.

I checked it on the first step: the query, key and value weights received a gradient of exactly zero, and the output projection received all of it. Only once $W_o$ has moved off zero does any signal reach the parts of the block that decide where to look. Zero initialisation buys safety at the moment of insertion, and it pays for it in a slower start.

What I would check in a dense head

Three measurements came out of this:

• the theoretical receptive field of the cell that makes each prediction, compared with the distribution of offsets the head is asked to predict;
• the error binned by offset distance, where a step or a steep rise with distance marks a reach problem, while a capacity problem raises the error at every distance;
• the effective receptive field from input gradients, measured with normalisation layers understood, since a spatial normaliser can make a cell look far-sighted.

The slow start of the zero-initialised block comes from the chain rule, which a zero-initialised layer learns only through a nonzero path works through, along with a way to stop it learning altogether. Reading early, high-resolution features is also how a new output can be added without moving the old ones, in adding an output to a trained network.

What I still don't know

This is a synthetic task in which the target is the only thing in the image worth seeing. In real images other structure can stand in for missing reach: a cell that cannot see a target may still see something correlated with where it is. I don't know how much that softens the cliff.

I also measured the slow start of the zero-initialised block with a single learning rate for the whole network. Giving the new block its own, higher learning rate might close most of the gap while keeping the identity at insertion, and that is the next thing I want to try. Finally, the 90% gradient-mass radius is only one convention for the effective field. Luo et al. analyse its shape more carefully, and a threshold of 50% or 95% would move the numbers in the second column of the first table.

References

1. Understanding the Effective Receptive Field in Deep Convolutional Neural Networks — Wenjie Luo, Yujia Li, Raquel Urtasun and Richard Zemel, NeurIPS 2016
2. Multi-Scale Context Aggregation by Dilated Convolutions — Fisher Yu and Vladlen Koltun, ICLR 2016
3. Non-local Neural Networks — Xiaolong Wang, Ross Girshick, Abhinav Gupta and Kaiming He, CVPR 2018
4. An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution — Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev and Jason Yosinski, NeurIPS 2018
5. Group Normalization — Yuxin Wu and Kaiming He, ECCV 2018
6. Adding Conditional Control to Text-to-Image Diffusion Models — Lvmin Zhang, Anyi Rao and Maneesh Agrawala, ICCV 2023
7. ReZero is All You Need: Fast Convergence at Large Depth — Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell and Julian McAuley, UAI 2021