A shape term cannot constrain what it is invariant to

Shape terms are a common way to put prior knowledge into a model that predicts several points: keep these two points a fixed distance apart, keep these three in a line, keep this group the shape it should be. Each term is easy to write. What each one actually constrains is less obvious, and I wanted a precise way to say it before trusting one. The precise way turned out to be the term's invariances, together with an old count from structural engineering.

Invariance means a blind direction

Suppose a loss term $R$ doesn't change when the points $p$ are transformed by some family $T_\alpha$: shifted, rotated, reflected or scaled. Differentiating $R(T_\alpha p) = R(p)$ with respect to $\alpha$ at the identity gives

The gradient is orthogonal to every direction the term is invariant to. Whatever error lies along those directions, the term cannot push against it; only other terms, such as the data term, can. So listing the invariances of a shape term lists exactly what it leaves to the rest of the loss:

• a term made of distances between points is unchanged by translations, rotations and reflections;
• a term made of distances divided by the shape's own size is also unchanged by scaling;
• a term that measures only some of the distances leaves the rest of the configuration free as well.

How many distances does a shape need? Three points in the plane have 6 coordinates; translations and rotations account for 3 of them, which leaves 3 to pin down, so a triangle needs all three of its edge lengths. In general, $n$ points need $2n - 3$ independent distances, the count behind Laman's (1970)¹ characterisation of rigid frameworks in the plane. Even then, distances can't tell a shape from its mirror image, because reflection preserves them.

A missing point under three kinds of shape term

I fitted three points of a fixed triangle by gradient descent from 300 random starts, run together as one batch. Points 0 and 1 were observed with a little noise; point 2 was never observed, so only the shape term could place it.

With two edges from point 0, point 2 only has to sit at the right distance from point 0, which is a circle. The fits landed all around it, and only 7% ended near the truth. Adding the third edge fixed the shape, and the fits split almost evenly between the true position and its reflection across the line through the other two points, 53% and 47%. Adding a penalty on the triangle's orientation, the sign of its signed area, removed the mirror solution entirely. It didn't make the fit unique, though: 24% of starts stopped somewhere else, where the edge terms and the orientation penalty pulled against each other. A one-sided penalty added to a non-convex term creates new places to get stuck.

A shape term that cannot see size

The second case is the one I worried about most, because it is easy to write by accident. Dividing every distance by the shape's total size makes a term about proportions only, which sounds harmless. I gave the fit observations of all three points that were shrunk by 30% about their centre, a systematic bias, and compared two terms at the same weight: absolute edge lengths, and edge lengths divided by the shape's size.

First I checked the invariance directly. At a random configuration, the cosine between the size-free term's gradient and the direction of scaling was $7 \times 10^{-8}$, zero to numerical precision.

The absolute term pulled the shrunk triangle almost back to its true size, limited only by how strongly it was weighted against the data. The size-free term left the size wherever the data put it: fits matched in the right orientation ended at 0.700, exactly the bias. The 48% of fits that settled on the mirror image ended even smaller, 0.279, because the best way for the data term to fit a reflected shape to the observations is to shrink it towards their centre. The size-free term was satisfied in every case; its shape error was under 0.003 everywhere. It enforced exactly what it measures, and nothing it doesn't.

This also rules out a suspicion I had. With Adam, a direction the loss can't see is a direction the optimizer can drift along. I repeated the fits with plain gradient descent, and the sizes came out the same, a mean of 0.51 against 0.52. The result is the geometry of the term, not an optimizer artefact.

What I would check in a shape term

These are the questions I would now ask of any geometric term before relying on it:

• What is it invariant to? Translation, rotation, reflection, scale. Each one is a direction it cannot correct.
• Does it constrain enough distances to be rigid, $2n - 3$ for $n$ points in the plane, or does it leave points free?
• If the reflection matters, is there a term that breaks it, and does that term create new local minima?
• If the size matters, is it measured against an external scale instead of the shape's own?

The most consequential invariance in deep learning may be the scale invariance that batch normalisation gives every convolution in front of it, and what Adam does along that blind direction is in batch norm’s running statistics describe a network that has already moved. A shape term also cannot fix a failure the training data never shows, in a loss cannot fix what the training data never shows.

What I still don't know

I fitted points directly, without a network in between. A network trained with these terms sees them through its own parameters, and a term's blind directions in output space may or may not stay blind once many outputs share weights. I also used a single triangle. Larger structures can be rigid with fewer than all pairwise distances, and which subsets work is exactly the question Laman's count answers. Checking a few of them in a trained model, where a direct fit is not available, is the next thing I want to try.

References

1. On graphs and rigidity of plane skeletal structures — G. Laman, Journal of Engineering Mathematics 1970