ShotPlan: Cinematic Video Generation with Learnable Planning Token

ShotPlan generates multi-shot videos with hard cuts at user-specified timestamps, smooth soft transitions (cross-fades), and temporally localized camera movement — all from a single text prompt with per-shot captions.

Frame-Accurate Hard Cuts

Shot boundaries land exactly where you ask — transition deviation of just 0.64 frames, an order of magnitude below prior methods.

Soft Transitions

The same planning-token mechanism handles cross-fades: separate learnable tokens mark fade-in begin and fade-in end.

Timed Camera Movement

Camera motions like circle-left can be triggered at an exact frame, enabling shot-level cinematography inside one continuous take.

Abstract

One token per transition, planned in latent space

Cinematic multi-shot video generation requires precise control over when shots change, not just what they contain. Existing approaches struggle to place transitions at user-specified timestamps, drifting by several frames or ignoring the request altogether. We present ShotPlan , a framework that equips a pre-trained text-to-video diffusion transformer with learnable planning tokens : a single learnable embedding is replicated once per requested transition event and concatenated with the visual tokens. To tell each planning token where in time it acts, we introduce Fractional Temporal RoPE (FRoPE) , which assigns each token a fractional latent-time coordinate derived from its target frame — video tokens keep their original positions, so the backbone's spatio-temporal prior is untouched. The tokens participate in standard self-attention with no masking, steer the denoising process toward the requested shot structure, and are simply discarded before decoding. Trained on 13K curated multi-shot clips, ShotPlan achieves frame-accurate hard cuts, smooth soft transitions, and temporally localized camera movement, outperforming existing multi-shot methods in transition accuracy, cross-shot consistency and narrative coherence.

Method

How ShotPlan works

ShotPlan is built on Wan2.1-T2V-14B and changes nothing about the backbone's treatment of video tokens — planning happens entirely through a handful of extra tokens.

Latents are patchified as usual; for a request like hard cuts at frames 21, 46, 64 , three copies of the learnable planning embedding are appended with fractional temporal positions 6.00, 12.25, 16.75 . They attend jointly with visual tokens through every DiT block and are removed before unpatchify.

01 — PLANNING TOKENS

One embedding, replicated per event

Each transition type (hard cut, fade-in begin, fade-in end, camera move) owns a single learnable embedding F. It is replicated once per requested event and concatenated with the visual token sequence — no attention masks, no architectural surgery.

02 — FRACTIONAL ROPE

Frame-level timing in latent time

The 3D VAE compresses time 4×, so integer latent positions can't express arbitrary frames. FRoPE gives each planning token a fractional temporal coordinate, letting it point between latent frames while video tokens keep standard RoPE.

03 — REMOVE & DECODE

Zero inference overhead in the output

After the final DiT block the planning tokens are sliced off; only visual tokens are unpatchified and decoded. The tokens exist purely to steer denoising — the output video has the exact same shape as the base model's.

t̃u = 1 + (tu − 1) / st<br>A cut requested at video frame t​u maps to fractional latent time with temporal stride s​t = 4 — e.g. frame 21 → position 6.00, frame 46 → 12.25, frame 64 → 16.75.

Results

Video gallery

All videos are generated at 832×480, 81 frames, from a single prompt with per-shot captions. Gold markers on each timeline show where each cut happens. Hover to play with sound off; videos loop.

Timeline legend — ▮ gold ticks : cut frames · ● teal dot: playback position · all clips are 81 frames.

Soft Transition

Cross-fades, planned the same way

Two planning tokens — fade-in begin and fade-in end — bracket the transition window, producing a smooth dissolve between shots instead of a hard cut.

Camera Movement

Temporally localized camera control

A camera-movement planning token triggers a specific motion — circle, zoom, dolly, truck — at an exact frame inside one continuous take. The gold tick marks where the movement is scheduled to begin. For easier viewing, a blue box is drawn around the outer border of the video at the frame where the camera movement starts.

Benchmark

Quantitative comparison

On the multi-shot generation benchmark, ShotPlan cuts transition deviation to well under one frame while also leading on consistency and narrative coherence.

Multi-shot generation benchmark

Transition Deviation in frames (lower is better); other metrics higher is better.

MethodTrans. Dev ↓Text Align ↑Char. Cons ↑Scene Cons ↑Narrative...