Machines do not press play.<br>Request Access

Share<br>X LogoHacker NewsYRedditLinkedIn Logo

Loading…

Query<br>event-aligned clip

For decades, video has been optimized for one consumer: a human pressing play. That consumer wants a smooth stream. Seek near a keyframe, decode forward, display frames at wall-clock speed. The only real job is to stay ahead of the viewer. Video playback is a tightly scoped engineering target, and the codecs and containers nail it.<br>But machines do not press play. A multimodal warehouse — storage built to serve video alongside every other modality a model trains on — touches the same files for a workload that looks nothing like playback.<br>Inside any modern video file is a queryable structure, the rules for turning stored bytes back into pixels. This deep dive is about treating that structure as data. We take H.264 inside an MP4 container apart at every layer a warehouse has to reason about, follow a frame-level query of your choice from request to finished pixels, and show two ways the warehouse can do better: run a smarter query against the file as it stands, or reshape the file so the query is cheap by construction.<br>Let's unpack this together. Pick a video to explore.

Pick a video<br>The deep dive follows whichever file you select.

Drivingloading…<br>Pexels stock clip.

Robot arm<br>Robot arm picking up a wine glass.

Street<br>Short street scene.

Click to choose your own video file (browser-side only, nothing uploads).

Metadata loading…

What machines ask of video<br>Video systems serve a small set of recognizable workloads. Most production data pipelines around video are some combination of:<br>Full-Clip Playback A simple query that touches every frame in the file, making it the most decode-heavy of any access pattern and an exact analog to human-oriented linear playback.

loading file…

Thumbnail Extraction One representative frame at time t, often the first or middle frame of a clip, for indexing and previews. The low overhead means thumbnails are often pre-materialized, however there are still cases where very sparse single frame reads are required without knowing the frame index up-front.

loading file…

Deterministic Evaluation Samples K frames evenly spaced across the whole clip for reproducible scoring during evaluation — linspace(0, duration, K). Spreading the samples across the whole file stresses the decoder at many disjoint positions rather than within a single window.

loading file…

Down-Rate Playback Every Nth source frame at a target cadence (e.g. 10 fps from a 30 fps source) — common preprocessing for video-LLM ingest where the source frame rate is more than the model needs.

loading file…

Event-Aligned Clip Frames before and after an anchor t at a target sampling rate. The event at time t typically comes from a label, lidar pulse, action timestamp, or other annotation. Drag the bounds to change before/after independently; the center handle is t.

loading file…

Scene-Boundary Detection The first frame of every scene. Another example of a pre-processing stage that is typically materialized ahead of time, but there are still cases where the parameters for scene detection may need to be tuned after-the-fact, for example a fade-out. This access pattern takes the first frame of each scene by measuring the amount of change between adjacent frames and often aligns with where a video encoder might itself insert keyframes.

loading file…

Scattered Timestamps K independent timestamps randomly distributed across the clip. The contrasting case to dense window queries where sometimes every output frame corresponds to a different keyframe.

loading file…

Despite their differences, all of these access patterns reduce to a query over a set of (file, timestamp, output_shape) tuples, possibly aligned across modalities and possibly batched across many files. The job of a multimodal warehouse is to efficiently serve the required access patterns from compressed bytes all the way to on-device RGB tensors, all while keeping within reasonable limits of storage, compute, and networking cost.<br>Playback isn't the antagonist here. It's the workload that existing video formats and tools were built for, and the warehouse inherits all of that machinery. The interesting question is how to use the same machinery for a differently shaped demand.

Why H.264 in MP4?<br>This deep dive uses H.264 inside MP4 because it is the mainstream baseline many datasets already contain: common in production video collections, broadly supported by tools, and widely hardware-accelerated through NVDEC, VideoToolbox, AMD VCN, and Intel Quick Sync hardware decode paths. It is also old enough that its container/codec split is well understood. The point is not that H.264 is the only interesting format. It is that the warehouse problem shows up in the ordinary format people already have.<br>H.264 also keeps the mechanics legible. The MP4 container exposes the timing and byte-address side; the H.264 bitstream exposes the prediction graph;...