WorkFoundationsCase file 05
Filling in missing parts of images, with little training data
My MSc thesis: a GAN that combines two kinds of attention, aimed at the ways image generators fail when training data is scarce.
- Where
- National Taipei University of Technology, 2022 — 2023
- My part
- Author
- Built with
- PyTorch, GANs, Attention mechanisms, Receptive field design
Image inpainting (filling in a masked part of an image) breaks down in typical ways when data is limited: the generator produces only a few kinds of output, copies its training images, or never fits. These have different causes and need different fixes, and treating them as one problem is why low-data work stalls.
Two kinds of attention
Contextual attention copies texture from related regions; spatial attention keeps structure aligned across the missing area.
| Attention | Looks at | Gives |
|---|---|---|
| Contextual | similar-looking regions elsewhere in the image | believable texture |
| Spatial | shapes across the edge of the filled area | consistent structure over large masks |
Widening how much of the image the model can see at once (its receptive field) made large filled areas more consistent. Models that only look locally produce believable texture in the wrong shape.
Why it still matters in my work
The attention studied here also appears in transformers. That work made later fine-tuning decisions more specific: I could identify whether a failure came from local context, global structure or the training regime.
The same habit runs through the pose work: find the specific way something fails, then pick the fix aimed at that failure, instead of trying a general remedy and watching whether the average moves.
What came out of it
- Published as an MSc thesis at National Taipei University of Technology.
- A working understanding of attention that carries straight over to transformer-based systems.