IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer

Abstract

Real-world spatial intelligence requires agents to understand scenes from continuous video streams, where objects move, persist, disappear, and reappear over time. While recent spatial foundation models have enabled generalizable feed-forward 3D reconstruction, most streaming methods remain geometry-centric and lack temporally consistent object-level understanding. Meanwhile, existing semantic reconstruction and 3D-aware vision-language methods largely rely on externally extracted 2D semantic cues or loosely coupled geometry inputs, limiting unified geometry-instance learning in long dynamic scenes. In this paper, we propose IGGT4D , a streaming instance-grounded geometry Transformer for online 4D scene understanding. IGGT4D processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation of camera motion, geometry, and object identity. This enables long-sequence feed-forward reconstruction with geometry-instance consistency in dynamic environments. To address the lack of high-quality 4D supervision, we further construct InsScene4D-147K , a large-scale dataset spanning real/synthetic and static/dynamic scenes, with RGB images, depth, poses, and temporally consistent instance masks generated by an automated geometry-guided annotation pipeline. Experiments on 3D reconstruction, pose estimation, instance spatial tracking, and open-vocabulary segmentation demonstrate that IGGT4D outperforms existing streaming baselines while maintaining scalable online inference for long dynamic sequences.

Key Contributions

Unified Geometry-Instance 4D Reconstruction

IGGT4D unifies camera motion, scene geometry, and consistent instance features within a single feed-forward 4D reconstruction framework.

Streaming Reconstruction & Instance Clustering

Frame-by-frame causal inference and lightweight online clustering enable efficient reconstruction and persistent object identities over long 4D sequences.

InsScene4D-147K Dataset

We curate 147K real/synthetic and static/dynamic sequences, with 4D-consistent instance annotations from an automated geometry-guided pipeline.

InsScene4D-147K Dataset Construction

InsScene4D-147K dataset statistics and automated geometry-guided annotation pipeline.

Beyond existing public datasets, we further curate three complementary data sources for InsScene4D-147K:

RoboTwin 2.0

Using the official RoboTwin 2.0 pipeline, we generate a new collection of RGB-D sequences with temporally consistent instance masks.

RealEstate10K

Using DA3 depth and ground-truth poses, our pipeline reconstructs a TSDF mesh for ID inheritance, then fuses SAM2 auto and box-prompted masks to refine annotations.

HOI4D

Our pipeline removes dynamic regions before static-mesh reconstruction, then updates projected pseudo-labels using provided dynamic-object annotations.

IGGT4D Architecture

IGGT4D processes streaming video with optional camera poses to jointly predict camera motion, geometry, and instance features, then derives spatial-temporal consistent instance masks through online clustering.

Causal Geometry-Instance Transformer

Causal attention and KV caching reuse historical context to build a unified geometry-instance representation frame by frame over long video streams.

Tri-DPT Geometry-Instance Head

The Tri-DPT head jointly decodes depth, rays, and instance features, using geometry-aware attention to inject geometric priors into instance representations.

Streaming Instance Clustering

Per-view cosine-similarity clustering groups instance features into instance masks, matches them to persistent IDs in a global codebook, and updates centers for streaming association.

Streaming Geometry-Instance Predictions

Frame-by-frame visualization of RGB inputs, depth predictions, PCA-projected instance features, and instance masks in streaming video.