Derivative terms in a loss amplify noise by one over h squared

By Ajish Pradeep · 2026-10-07 · 5 min read

Claim. Supervising a model's derivatives helps when the target derivatives are exact and hurts when they are finite differences of noisy samples. In my fits a second-derivative term made the error 17 times worse on finely sampled noisy data and twice as small with exact targets.

Matching derivatives as well as values is an appealing idea. If a model should produce a smooth trajectory, add terms that match its velocity and acceleration to those of the target, and the fit should become smoother and more faithful. Czarnecki et al. (2017)¹ showed that this kind of Sobolev training helps when the target's derivatives are known exactly. In practice they usually aren't: they are finite differences of the same noisy samples the values come from. I wanted to see what happens to the idea then, and the arithmetic suggested it might turn against itself.

What a finite difference does to noise

Let the targets be samples $y_i = f(t_i) + \epsilon_i$ on a grid with spacing $h$ , with independent noise of standard deviation $\sigma$ . The usual estimates of the first and second derivatives are

\hat v_i = \frac{y_{i+1} - y_i}{h}, \qquad \hat a_i = \frac{y_{i+1} - 2y_i + y_{i-1}}{h^2}

Their noise variances follow from adding independent variances with the coefficients' squares:

\mathrm{Var}(\hat v) = \frac{2\sigma^2}{h^2}, \qquad \mathrm{Var}(\hat a) = \frac{(1 + 4 + 1)\,\sigma^2}{h^4} = \frac{6\sigma^2}{h^4}

Halve the spacing and the noise in the acceleration estimate goes up sixteen-fold. Sampling more finely, which improves everything else, makes the second-difference targets almost pure noise. A loss term that matches them asks the model to reproduce that noise, and the noise has a floor the model can't fit below without memorising it.

I checked the variances on a million samples with $\sigma = 0.1$ : 50.2 against a predicted 50 for the first difference at $h = 0.02$ , and $96.1$ million against $96$ million for the second difference at $h = 0.005$ .

The experiment

A small multilayer perceptron fits $f(t) = \sin 2\pi t + 0.5 \sin 6\pi t$ on $[0, 1]$ from samples with noise of standard deviation 0.1, at two spacings, $h = 0.02$ (51 samples) and $h = 0.005$ (201 samples). The loss is the mean squared error on the values, plus optional terms matching the model's own first and second differences to the targets'. I ran each setting with three seeds and two kinds of derivative target: differences of the noisy samples, and the exact derivatives of $f$ . The score is the root-mean-square error against the true function on a fine grid.

The same loss term with two kinds of target, at spacing 0.005 — One run (seed 0) of the note's experiment; its three-seed errors reproduce the table exactly. (interactive figure)

Spacing	Loss	Error, derivative targets from noisy samples	Error, exact derivative targets
0.02	Values only	0.072	0.072
0.02	Plus first differences	0.096	0.051
0.02	Plus second differences, weight $10^{-5}$	0.107	0.037
0.005	Values only	0.032	0.032
0.005	Plus first differences	0.042	0.024
0.005	Plus second differences, weight $10^{-6}$	0.216	0.033
0.005	Plus second differences, weight $10^{-5}$	0.567	0.015

With exact derivatives every extra term helped, and the second-derivative term halved the error at the fine spacing. With derivatives estimated from the noisy samples every extra term hurt, and at the fine spacing the second-derivative term made the fit 17 times worse than using the values alone. The same term, at the same weight, either halved the error or wrecked the fit, depending only on where its target came from.

How the noisy term takes over

The final values of the loss terms show the mechanism. At $h = 0.005$ with noisy targets, the acceleration term ended training at about $6 \times 10^7$ whatever its weight. That is the same order as the noise variance of the targets, $9.6 \times 10^7$ : the model can't match noise that changes sign from one sample to the next with a smooth function, so the term stays at its floor. Multiplied by a weight of $10^{-5}$ it still contributes about 600 to the loss, against less than 1 for the value term. The optimiser spends its effort pulling the curve towards the noisy curvature targets and gives up the values to do it.

At the coarser spacing the failure looked different. With 51 samples, the network matched the noisy second differences closely, and its value error on the training samples fell almost to zero. It had fitted the noise itself, and its error against the true function rose accordingly. A derivative term with noisy targets either dominates the loss or drives memorisation, and neither is what it was added for.

What I would do instead

The derivative terms are worth keeping when their targets are good, which means:

exact derivatives where they exist, which is the setting Sobolev training was designed for;
otherwise, derivatives estimated with a window wide enough to average the noise down, for example a local polynomial fit over several samples instead of a two-point or three-point difference;
weights set from the noise floor: a term whose irreducible value is $6\sigma^2/h^4$ needs a weight small enough that this floor doesn't swamp the value term, and the floor can be computed before training;
the value of each term, logged separately at the end of training, compared with its noise floor.

The same arithmetic, with differences in space instead of time, is why the direction between two close points is mostly noise, in the direction between two close noisy points.

What I still don't know

My signal is one-dimensional, smooth and sampled on a regular grid, which makes the variance arithmetic exact. Irregular sampling, noise that is correlated between neighbouring samples, and signals with genuine sharp changes all alter it. Correlated noise in particular can make differences less noisy than the formula says, and I haven't measured by how much. I also didn't try the windowed derivative estimates I'm recommending, so the size of their benefit in this setting is untested.

References

Sobolev Training for Neural Networks — Wojciech Marian Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Świrszcz and Razvan Pascanu, NeurIPS 2017