The Seam Is the Label — vuvko.net

The Seam Is the Label

Published: 2026-08-09

Last time I wrote about the “reverse” trick: if you grow a dataset by asking a generator to add glasses to a face, the generator’s fingerprint ends up in exactly the images that have glasses, and the model learns “generated means glasses” instead of learning what glasses look like.<br>The fix was to immediately ask the generator to take the glasses off again, so that both classes carry the same fingerprint and it stops predicting anything.

That trick has a requirement I only mentioned briefly: the edit has to be invertible.<br>You need to be able to undo the thing you just did and keep an image you would still put in the dataset.

Object detection breaks that requirement.<br>You can ask a generator to un-add an object, but that pass leaves artifacts of its own.<br>And you rarely regenerate the whole frame anyway — you patch the region where the object goes, because that is the cheap way to do it and because copy-paste of objects into scenes is a genuinely good augmentation on its own11.Ghiasi G. et al. Simple Copy-Paste Is a Strong Data Augmentation Method for Instance Segmentation // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. — 2021. — pp. 2918–2928. .<br>So the fingerprint is not spread over the image.<br>It sits in a region.<br>The same region you are asking the model to draw a box around.

Patching one region

Here is the setup, with the cat22.The cat comes from my photos of the Solovki islands. There are more in the series about that trip. standing in for the object.<br>I take an ellipse around it, pass just that region through a generator, and paste it back.<br>I do not actually need a sophisticated generator to illustrate the point: a round trip through half resolution followed by a sharpening pass is what an upsampling layer does inside most generators, and it leaves the same kind of trace — the fine grain in the patch is not the grain the camera put there.

I will show pieces of Python along the way, so that you can reproduce any of this.<br>Rather than cluttering every snippet with imports, here is everything they use:

import io

import numpy as np<br>from PIL import Image, ImageFilter<br>from scipy import ndimage<br>That is the whole toolbox.<br>The only external libraries are for matrix and simple image manipulation: numpy, scipy, and pillow (imported as PIL for historical reasons).

# You can think of this function as any image manipulation/generator<br>def generator(img: Image.Image) -> Image.Image:<br>w, h = img.size<br>small = img.resize((w // 2, h // 2), Image.BICUBIC)<br>back = small.resize((w, h), Image.BICUBIC)<br>return back.filter(ImageFilter.UnsharpMask(radius=2, percent=110))

photo: Image.Image # the original photo<br>mask: np.ndarray # the region we are manipulating

edited = np.asarray(generator(photo), float)<br>original = np.asarray(photo, float)<br>patched = original * (1 - mask[..., None]) + edited * mask[..., None]<br>The sharpening matters. Without it the patch is simply blurry and anyone can see it; with it, the photo looks fine and only the very finest grain is missing.<br>And what is missing can always be written off as something the lens did.

The first thing to try is error level analysis: save the image again as a JPEG and look at what moved. Pixels that have already been through the compressor sit close to where it wants them and barely shift; freshly drawn ones have further to fall.<br>The technique is a quick way to see which parts of an image might have been manipulated.

def ela(img: Image.Image, quality: int = 90) -> np.ndarray:<br>buf = io.BytesIO()<br>img.save(buf, "JPEG", quality=quality)<br>again = np.asarray(Image.open(buf), float)<br>return np.abs(np.asarray(img, float) - again).max(axis=2)<br>Top row shows original, bottom row shows patched. The photos are the same to the eye. Error level analysis is where the oval shows up. Note that with a more sophisticated generator the boundary will not be this clean, but most of the time it is still visible.<br>One statistic finds it better

ELA is doing well here, but it is answering a question about compression while being dominated by content, which is why it needs the control row to stay readable in a more real-world scenario.<br>So let’s ask something closer to how capturing an image physically differs from manipulating one digitally.<br>Cut the image into 48-pixel windows and, in each one, measure how much of the finest detail is left — the grain right at the scale of individual pixels — compared with detail one step coarser.<br>A region that came back from a generator is short of the finest grain no matter what it depicts, because the generator drew it rather than the sensor recording it.

Getting the two scales cleanly apart is a small Fourier transform per window (I will write about the Fourier transform for images some other time).<br>You can treat the next snippet as magic.<br>The point is to illustrate what is possible with some clever math.

gray: np.ndarray # our photo in grayscale<br>WIN,...