Notes
- Taking a gradient spike apart, by loss term and by module
When a training run with several losses starts throwing gradient spikes, the usual first moves are to lower the learning rate, add clipping, or skip the batch that seemed to cause it. All three treat the spike as one number. I wanted to…
- The direction between two close noisy points is mostly noise
Many quantities in data analysis are directions computed from two estimated points: the orientation of an edge between two detected corners, the heading between two position estimates, the angle of a line fitted through two…
- The same weights, two execution paths, two answers
A trained model is usually thought of as its weights: run them, and you get its function. In practice the function also depends on which code runs them. Frameworks ship several implementations of the same operation, fused kernels for…
- Unlabelled is not negative, and the usual metric hides it
Dense prediction losses usually make an assumption without saying so: every location without a label is background. A detector, a segmentation head or a per-cell classifier computes its loss over the whole grid, and every cell that no…
- Restarting Adam moves every weight by the full learning rate
Training runs get interrupted and resumed all the time: to change a learning rate, to add data, to continue a run that stopped early, or to fine-tune a model that has already converged. A common shortcut is to load the weights and build…
- Sampling with replacement sees about 63% of your data
Weighted samplers are a common way to stop a large data source from drowning out a small one. Instead of shuffling the dataset, the loader draws each batch from a distribution over items, and the weights decide how often each source…
- Score perfect inputs before scoring a model
When a model is evaluated through a pipeline, the error that comes out at the end is usually attributed to the model. The model produces values, the pipeline turns them into the quantity of interest, and the difference from the truth…
- Softmax over one token has zero gradient, and Adam moves the weights anyway
Attention blocks sometimes see sequences of length one. A model may be trained on single items before it is trained on sequences, a cache may hold one entry, or a block built for many tokens may be run in a mode that feeds it one. In…
- Derivative terms in a loss amplify noise by one over h squared
Matching derivatives as well as values is an appealing idea. If a model should produce a smooth trajectory, add terms that match its velocity and acceleration to those of the target, and the fit should become smoother and more faithful.…
- Every .item() call makes the GPU wait for Python
Training loops collect numbers: the loss, gradient norms, per-layer statistics for a dashboard. It feels free, because each number is tiny. I wanted to measure it, because of how a GPU is driven. When Python calls a CUDA operation,…
- Freezing a layer does not freeze its batch norm
"Freeze the backbone and train a new head" is one of the most common moves in transfer learning. In PyTorch it usually means one line that turns off gradients for the backbone's parameters, and the mental model that goes with it is…
- Gradient clipping does different jobs under SGD and Adam
Gradient clipping is usually introduced as a guard against rare explosions. Pascanu et al. (2013) proposed rescaling the gradient whenever its norm passes a threshold $c$, so that a sudden cliff in the loss surface cannot throw the…
- How far a pretrained backbone moves decides what it keeps
Fine-tuning a pretrained network has two moving parts: the backbone, which arrives with useful features, and a new head, which arrives with nothing. With one learning rate for both, the head starts from random weights and sends large,…
- Adding an output to a trained network without moving the old ones
A network that already does its job well often needs one more output: a new class, a new attribute, a new kind of measurement. The tempting move is to add a head and fine-tune everything on data for both. The old outputs then move too,…
- Average precision scores thresholds you will never use
Average precision is the default summary for detection and retrieval, and for good reasons: it needs no threshold, it is a single number, and benchmarks such as PASCAL VOC (Everingham et al., 2010) and COCO (Lin et al., 2014) made it…
- Batch norm's running statistics describe a network that has already moved
Batch normalisation has two modes, and most of us learn only one of them properly. Training mode gets the paper, the diagrams and the intuition about smoother loss surfaces. Evaluation mode gets a line in the documentation: at inference…
- Cleaning noisy labels with a local median
Labels that come from another model, from sensors or from a pipeline are often a series: a value per time step, per position, per frame of some sequence. Such series pick up a particular kind of error: a single sample that collapses to…
- Correlation is not agreement
When a model's output, or a new way of measuring something, is checked against a reference, the first number reported is often a correlation. It is easy to compute and easy to read: 0.98 sounds excellent. I wanted to know exactly what a…
- A model that learns its own uncertainty chooses what to fit
Asking a regression network for its own error bars sounds like pure gain. Instead of one output per prediction it produces two, a mean $\mu$ and a standard deviation $\sigma$, and it is trained to maximise the likelihood of the data…
- A pseudo-labelled test set measures agreement with the labeller
When ground truth is expensive, a strong existing model is often used to label data: for training, and sometimes for evaluation as well. Training on pseudo-labels is a known trade-off. Evaluating on them is subtler, because the number…
- A shape term cannot constrain what it is invariant to
Shape terms are a common way to put prior knowledge into a model that predicts several points: keep these two points a fixed distance apart, keep these three in a line, keep this group the shape it should be. Each term is easy to write.…
- A student gains from a teacher only when the teacher knows more than the labels
Knowledge distillation is often presented as a free improvement: train a small student to match a large teacher's softened predictions as well as the labels, and the student comes out better than it would on its own. Hinton et al.…
- A zero-initialised layer learns only through a nonzero path
Initialising the last layer of a new branch to zero is a common trick. A new residual block, adapter or correction head then starts as the identity: it adds nothing, so the network it is attached to behaves exactly as before, and the…
- A confidence head learns the target you give it
Many models report a confidence next to each prediction: a detector's objectness, a per-point score in a localisation head, a probability attached to a regression. Downstream code often treats that number as "how likely is this…
- A dense predictor cannot place what its cell cannot see
Many vision models make their predictions from a grid. An object detector, a heatmap head or an offset regressor assigns each output to one cell of a feature map and reads a small vector from that cell: a class score, a box, an offset…
- A hundred thousand correlated samples can count as five hundred
Datasets are often counted in samples, while their diversity lives somewhere else. A corpus of a million sentences may come from a few thousand documents; a sensor dataset may hold millions of readings from a few dozen devices; an…
- A loss cannot fix what the training data never shows
When a model fails in a recognisable way, the natural reflex is to add a loss term that penalises that way of failing. If predicted shapes come out distorted, penalise distortion; if outputs that should be consistent aren't, penalise…
- A checkpoint is more than its state dict
"Save the state dict, load the state dict" is how most of us think about checkpoints, and when the load succeeds without an error we assume we have the same model back. I stopped assuming that while building an exact replay of a…