Frame selection is the whole game: engineering notes from making LLMs watch video
← leoaido.com<br>GitHub
engineering notes · open source
Frame selection is the whole game: notes from making LLMs watch video
A vision LLM can realistically afford about 150 images per video. Which 150 you pick decides whether the model watched the video or just a slideshow about it. These are my notes from getting this wrong over and over while building claude-real-video, an MIT-licensed local pipeline.
A video is not an article about the video
Why feed a model a video at all, when a writeup of the same thing costs way fewer tokens?
Because an article is someone's compression of an event. A human watched it, decided what mattered, threw the rest away. The framing, the timing, the stuff on screen they didn't think was relevant. When an LLM reads the article, it learns inside that author's choices. It can't recover what got cut, and it can't disagree with a selection it never saw.
Give the model the video and the compression step moves to the model. It sees what the demo actually looked like, not what the reviewer said it looked like. It notices the error message that flashed by in a tutorial nobody bothered to transcribe. It catches that the "quick setup" took eleven cuts. Same subject, very different position: reading a witness statement vs being the witness. That's why the token cost is worth paying, and why the rest of this post is about spending it well.
The budget problem
Most "video understanding" today is a transcript plus frames sampled on a timer. One frame per second, or per ten seconds. Uniform sampling fails in both directions at once: it buries the model in near-identical frames of a talking head, and it skips right past the one second where something actually happened. Same root cause both times: the sampler has no idea what changed.
The constraint that shapes everything is token budget. Images are the most expensive thing you can put in a context window. Once you accept "about 100 to 150 frames per video, period", extraction stops being the problem. Selection is the problem. Every kept frame has to earn its slot.
Scene detection, and why a fixed threshold isn't enough
The first pass is standard: ffmpeg's scene score. Take a frame at every scene change, plus a low density floor so a long uncut shot still produces something. All in one chronological pass, because dedup later wants to compare true neighbours.
Fixed thresholds die on a specific kind of content: animation and slow camera work. A cartoon character squashing and stretching, or a slow pan, changes constantly but never sharply. The score never crosses the line and the sampler sleeps through the whole thing. The fix is boring but it works: compute per-frame scene scores in a metadata-only pass, then keep a frame whenever its score beats a multiple of the rolling average. High-motion footage raises its own bar, quiet footage lowers it. Nobody has to tune anything.
Deduplication: one channel became three
Dedup started as one comparator and grew a channel every time real footage embarrassed it.
1. The global channel
Downscale to a 16×16 RGB signature, count cells that moved more than 25/255 in any channel, drop the frame if fewer than about 8% changed. RGB, not grayscale, because a red-to-green cut with equal luminance looks identical to a grayscale comparator. And you compare against a sliding window of the last four kept frames, not just the previous one. Otherwise A-B-A cutting (interview shot, reaction shot, back to interview) re-admits a shot the model already saw, just because a different frame sat in between.
2. The action channel
A percentage threshold is structurally blind to small subjects. A person taking up 0.5% of a wide shot can never change 8% of the pixels no matter what they do, so the one second that mattered gets deduplicated away. I didn't find this in a benchmark. A user found it after running the tool on 2,181 real videos. The patch: on a 32×32 grid, if even a handful of cells change hard (more than 45/255), the frame counts as new regardless of percentage. Small subject, sharp change, kept.
3. The settled channel
The global channel also can't see small local state changes: a caption swap, a line of ink appearing on a whiteboard, a UI element updating. Those measure 0.0% at 16×16. So the third channel works on a 192×192 signature and looks for pixels that differ strongly from every frame in the kept window. It allows a plus-minus 1 pixel shift when matching, because film weave, sensor jitter and grain otherwise register as change everywhere.
The guards matter more than the detector here. It only runs when the scene is otherwise static, since motion is the other channels' job. Candidate pixels also have to survive a second, stricter tolerance pass. That kills soft-contrast drift like smoke dissipating, while ink and text keep a hard core and survive. And every triggered keep raises the...