Notes on making AI-generated pixel art usable — Tokimon

I’ve been building a small desktop companion — a pixel-art creature that sits on<br>your Mac and reacts to what you’re doing. Four creatures, actually, one per<br>element. All of the art is generated: PixelLab’s create_image_pro for the<br>static poses, animate_image for the frames in between. I’m putting that in the<br>first paragraph because the generation is the boring part of this story.

Generating everything took an afternoon or two. Getting it to a state where I<br>could put it in a product took considerably longer, and most of that time went<br>into finding out how the output was wrong. Some of the failures were obvious the<br>moment I looked. Several were not, and one of them I only caught after building a<br>metric specifically to catch it, then discovering the metric couldn’t.

These are my notes. Numbers where I have them.

4 creatures<br>52 static sprites<br>128×128 every frame, RGBA<br>91% of the time, doing nothing

The contract

The thing I got right early, mostly by accident, was deciding what “correct”<br>meant before generating anything at scale.

Every sprite is 128×128 RGBA. The feet line sits at y=123. The horizontal center<br>is x=64. No pixel touches the top edge of the frame. Those four properties are<br>checked by a script, and if any of them fails the build fails.

y=123 x=64 no pixel here

earth/idle.png<br>The contract, drawn on a real sprite. The lines sit at the coordinates the script actually checks — not at positions chosen to look tidy.<br>packages/assets/tools/qa.py<br>for p, im in ims.items():<br>a = im.getchannel("A"); l, t, r, b = a.getbbox()<br>errs = []<br>if im.size != (128, 128): errs.append("size")<br>if b != 123: errs.append(f"feet y={b}")<br>if abs((l + r) // 2 - 64) > 1: errs.append(f"center x={(l + r) // 2}")<br>if any(a.getpixel((x, 0)) > 0 for x in range(128)): errs.append("top edge")<br>h = holes(im)<br>if h: errs.append(f"hole {h}px")<br>if errs: struct.append(f"{e}/{p}: {', '.join(errs)}") holes() is a flood fill from the edges: any transparent pixel it can't reach is a hole inside the creature, which the generator produces more often than you'd think.

That sounds like bureaucracy for a hobby project. It isn’t. It means pose and<br>creature are interchangeable at runtime — I can swap the fire creature’s idle<br>for the water creature’s wave and nothing shifts by a pixel. Without it you<br>spend the rest of the project hand-nudging files, and every new pose is a new<br>alignment problem. With 52 static sprites and 412 animation frames, that’s not a<br>thing you fix later.

fire/idle<br>water/wave<br>earth/idle<br>air/wave<br>Four creatures, two poses, one register. Nothing here has been nudged into place — the feet land on the same row because the contract says they must.<br>Generative models do not respect a contract like this. They produce something<br>roughly centered, roughly the right size, roughly consistent. All of the work<br>described below exists to close the gap between “roughly” and “exactly.”

Quantizing the input costs you the whole sequence

This one cost me the most and I’d never seen it written down.

animate_image takes a first frame and a last frame and interpolates between<br>them. I was sending both as base64. There’s a size limit on the argument, and to<br>stay under it I quantized the two images down to 6 colors. My assumption —<br>reasonable, wrong — was that this degraded the two frames I was sending, and that<br>the frames in between would be generated fresh.

They aren’t generated fresh. The model interpolates from what you hand it, and<br>the poverty of the input propagates through everything:

input sentcolors per generated framebase64, quantized to 6 colors30–32intact 128×128 PNG63–64<br>For context, the static sprite those frames have to blend into has 50 colors. So<br>the animation was measurably flatter than the still image sitting next to it,<br>which is exactly the kind of defect you feel before you can name it. Looking<br>closely, what’s gone is the speckling on the stone creature’s cheeks and<br>forehead, and the half-tones in its leaves, which posterize into flat bands.

the cost There’s no post-processing fix. The information was never generated. You<br>regenerate, and you accept that the motion comes back slightly different.

The water creature had the same problem in mirror image: 8 of its 13 sequences<br>came back at 31–40 colors from 12-color inputs, against 57–64 for the five I’d<br>sent intact. I hadn’t noticed until I started counting.

Color counting is the cheapest defect detector I found

That table is the whole lesson. Counting unique colors per frame is two lines of<br>Pillow, and it surfaced a defect that I had been looking directly at without<br>seeing.

two lines of Pillow not in the repo<br>from PIL import Image<br>len(set(Image.open(path).convert("RGBA").getdata()))

I now count colors on every batch. Current floor per creature, across all<br>intermediate frames: fire 49 (a yawn frame), water 54, earth 64. When something<br>drops well below its neighbors, something went wrong upstream.

The related metric that...