Visual Prompt Engineering for Video Models
TL;DR: Just as text-based prompt engineering systematically improves language model performance, visual prompt engineering (VIPE) serves as a simple, compute-efficient approach to elicit better visual reasoning performance from video models by transforming the input task image (e.g., making abstract sketches photorealistic).
Original Solution Attempt
After Visual Prompt Engineering
Physics VPCT
Conjunctive Search
Connect the Dots
Sort 3 Numbers
Maze
RushHour
Physics VPCT
Conjunctive Search
Connect the Dots
Sort 3 Numbers
Maze
RushHour
Abstract
In the age of foundation models, a model is only as good as its prompt. For this reason, prompt engineering has become an essential technique for improving language model performance. Since video models are currently becoming foundation models for visual tasks (e.g., visual reasoning), we here ask whether they similarly benefit from visual prompt engineering: automatically modifying the task image to improve model performance. For example, for a visual physics reasoning task ("Where does the ball land, after passing a set of obstacles?"), an abstract sketch-like scene can be turned into a photorealistic version with a simple call to an image editing model. We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks. In fact, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering or test-time scaling. Ultimately, just as text-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute-efficient approach to elicit better visual reasoning performance from video models.
Podcast
Listen to the NotebookLM-generated podcast overview of our paper:
Your browser does not support the audio element.
Task Examples
Below, we show examples for each task and split. We show a solution attempt on the original image (which often fails), a solution attempt on the most successful VIPE variant for the task and split, and other VIPE variants obtained by freeform VIPE or atomic concept edit (ACE) VIPE.
Physics VPCT<br>Predict which bucket the ball will land in after rolling down the ramps.<br>Text Prompt: "The ball moves down in a physically plausible way, sliding down the obstacles, and ends up in one of the three containers at the bottom. [...]"
Original Solution Attempt Failed
Best VIPE Variant Industrial factory chutes<br>Success
Other Variants
Ancient stone ruins
Baking kitchen prep
Billiard trick shot
Candy sweets landscape
Claymation stop motion
Construction site steel
Garage rube goldberg
Macro forest nature
Marble run toy
Mini golf course
Neon cyberpunk arcade
Office desk supplies
Retro pachinko machine
Science chemistry lab
Steampunk brass mechanism
Water park slides
Winter ice slopes
Wooden physics puzzle
Zen garden bamboo
Conjunctive Search<br>Locate the target object defined by a conjunction of features (e.g., color and shape).<br>Text Prompt: "The two blue balls begin to glow."
Original Solution Attempt Failed
Best VIPE Variant Globe lightbulbs case<br>Success
Other Variants (Freeform)
Arcade joystick panel
Billiard table felt
Candy assortment countertop
Deep sea bioluminescence
Dj mixer interface
Fantasy crystal orbs
Gemstone jewelry velvet
Gym equipment wood floor
Holiday ornaments snow
Lab plasma balls
Macro marbles and dice
Neon mini golf
Night pool led floats
Photorealistic 3d shapes
Pinball arcade bumpers
Sci fi energy cores
Synthwave neon grid
Twilight led bocce
Vintage toys hardwood
Other Variants (ACE)
REPLACE flat color of the two blue circles with a 3D glossy material
REPLACE flat rendering of blue circles with 3D glossy volumetric material
SET border outline of the two blue balls as a thick white stroke
REPLACE flat color of blue balls with 3D glossy material
REPLACE flat material of blue circles with glossy 3D rendering
SET material of the two blue circles as glossy 3D glass
SET rendering material of the blue circles as 3D glossy
SET material of the blue balls as glossy 3D spheres
SET materiality of the two blue circles as 3D glossy spheres
REPLACE size of the blue circles with a significantly larger scale
SET size of the blue circles as significantly enlarged
SET material of the blue circles as glossy 3D
SET color saturation of non-target shapes as completely desaturated grayscale
REMOVE color saturation from all non-target objects
REPLACE colors of all red circles and blue squares with grayscale
SET material of the blue circles as glossy 3D spheres
SET the material of the blue circles as glossy 3D spheres
REPLACE flat shading of the two blue circles with a glossy 3D material
SET material of the blue circles as glossy 3D
Connect the Dots<br>Connect same-colored dots.<br>Text Prompt: "Connect each pair of same-colored circles with a line."
Original Solution Attempt Failed
Best...