Runway News | AVTensor: A High-Performance Rust Media Decoder for Training Pipelines

Enterprise SalesLoginTry Runway

AVTensor: A High-Performance Rust Media Decoder for Training Pipelines<br>July 10, 2026<br>by Rik Heijdens, Research Engineering

Media containers keep time in ways that are easy to get subtly wrong—for example, negative timestamps or audio and video streams that don't start together. After an off-the-shelf decoder silently misaligned audio and video in our training data, we wrote our own. Today we're open-sourcing it: AVTensor, a Rust media decoder that demuxes both streams in a single pass and improved our training MFU by 1.8 percentage points.

Training audio-visual models requires audio and video in sync. A pipeline that decodes a clip implicitly assumes frame i of the video corresponds to sample window i of the audio. A few tens of milliseconds of drift is enough to notice, lip sync especially—yet containers don't encode that alignment in any straightforward way.

To see why, look at what a video file actually is. An MP4 or MOV is a container holding several independent streams : usually one video stream, one or more audio streams and sometimes subtitles. Each stream is stored as a sequence of compressed packets and keeps its own clock. When you play the file, a demuxer pulls the packets apart and hands them to codecs, which decode them into frames of video and samples of audio.

container (MP4)video packets · time base 1/24000audio packets · time base 1/48000subtitles (optional)demuxervideo codecframes [T, H, W, 3]audio codecsamples [C, T]Inside a media container: independent streams, each on its own clock.

Three properties of an ordinary MP4 cause most of the trouble:

Presentation order isn't decode order. Codecs don't store every frame as a complete image. An I-frame is a full picture; a P-frame stores only what changed since the previous frame; a B-frame stores changes relative to both the previous frame and the next one. The decoder must decode that future frame first, so the container stores packets out of playback order and each frame carries two timestamps: a decode timestamp (DTS) and a presentation timestamp (PTS). Only PTS reflects what a viewer sees. That's also why you can't "seek to frame 500": you seek to the nearest I-frame before the point you want and decode forward.

Timestamps aren't frame counts. Containers express time as integer ticks over a stream-specific time base (1/24000, 1/90000, …), a convention that dates to 1953, when American color TV shifted the broadcast framerate from 30 to 30/1.001 (~29.97) to fit the color signal and cinema's 24fps became 24/1.001 for broadcast. "Frame number × frame duration" has been a floating-point footgun ever since. Variable-framerate streams abandon uniform spacing entirely: an encoder can drop a near-identical frame and extend the previous one; a screen-recording stream might have two frames a minute at inconsistent times.

Streams don't have to start at zero—or together. Video frames and audio samples often use different clocks and offsets.

We hit assets whose first video packet had a PTS of −3.337 seconds while the first audio packet sat at −0.023 seconds. Most players present this plausibly (applying container timing metadata like edit lists), which is why it goes unnoticed—in datasets and in some of the tools we used to review them.

At the time, our pipeline decoded video with torchcodec and audio with a different library entirely—torchaudio's StreamingMediaDecoder—then stitched the results together. On assets with negative PTS, torchcodec's first frame appeared at 1.6s (it dropped everything before zero), while the audio reader started from a different origin. The first frame stayed frozen for over a second while audio played and the rest of the clip ran desynced.

−3.337s01.6svideoaudiofirst frame the decoder returnsaudio with no matching frames:trains as a frozen first frameA real asset from our data: three different starting points for the same clip.

It's a silent failure: the training pipeline runs fine. But the model learns that lips and speech are only loosely coupled and you start hunting for the problem in the architecture—because the first instinct is that it's always a RoPE issue, right?

That's why we keep repeating the mantra: always look at the data. We traced quality problems in audio-visual training back to decoded video and audio returning different effective lengths and origins. Our stopgap was defensive cropping (cut both streams to the shorter one and assert they agree within 10 milliseconds), which treated the symptom without fixing the decoder.

torchcodec, PyTorch's official media decoder, is good at what it's built for: random-access video decode with per-frame timestamps. But when we evaluated it (v0.11 at the time), its audio decoder was still early—upstream reworked the seeking logic in later releases—so our audio path ran through torchaudio's StreamingMediaDecoder instead. Neither library was...