ARDY: Interactive Human Motion Generation
Spatial Intelligence Lab
ARDY:<br>Autoregressive Diffusion with Hybrid Representation<br>for Interactive Human Motion Generation
Kaifeng Zhao1,2
Mathis Petrovich1
Haotian Zhang1
Tingwu Wang1
Siyu Tang2
Davis Rempe1
1 NVIDIA
2 ETH Zürich
ACM Transactions on Graphics · SIGGRAPH 2026
Paper (PDF)
Code
Models
ARDY is an autoregressive diffusion model designed for interactive motion generation, supporting online text prompting and flexible long-horizon<br>kinematic constraints (root paths/waypoints, full-body keyframes, and sparse joint positions/rotations) with real-time responsiveness.
Your browser does not support the video tag.
Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics.<br>While recent offline motion generation approaches like Kimodo offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings.<br>Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows.<br>In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints.<br>ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning.<br>We propose a two-stage autoregressive transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints.<br>By training on a large-scale motion capture dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses,<br>ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations<br>demonstrate strong motion quality and constraint adherence, and we present an interactive demo with dynamic text control, keyframe constraints,<br>path following, and real-time locomotion control.
Key Capabilities of ARDY
Online Text-to-Motion Generation
ARDY supports interactive text-conditioned motion generation across a wide range of behaviors.
Limp
Pick & Put
Stealthy Walk
Victory Dance
Lean and Peek
Zombie Sit
Prompt Streaming
Kinematically Constrained Motion Generation
ARDY supports flexible kinematic constraints, including root trajectories or waypoints, full-body keyframes, end-effector joint positions and rotations, as well as arbitrary combinations of these constraints.<br>Constraints can also be specified far into the future (beyond the current generation window) to enable long-horizon goal reaching.
Root Trajectory 1
Root Trajectory 2
Root Waypoint 1
Root Waypoint 2
Fullbody Keyframe 1
Fullbody Keyframe 2
End Effector Position
End Effector Position+Orientaion
Long-Horizon Goal 1
Long-Horizon Goal 2
Compositionality
Kinematic Constraints Chain
Application: Interactive Humanoid Control
-->
ARDY enables online motion synthesis for interactive applications, which can be valuable for game character control, downstream robotics, and simulation workflows.<br>It supports real-time locomotion control via mouse waypoint editing and keyboard velocity commands.
Humanoid Robot Control
By combining ARDY’s real-time humanoid motion generation with the SONIC physical tracking policy, we enable interactive robot motion control with streaming constraints and user inputs. We demonstrate applications on the Unitree G1 robot.
Method
-->
Our synthetic dataset enhances performance on various downstream autonomous driving tasks by providing diverse and challenging scenarios.
-->
ARDY is an autoregressive diffusion model for interactive motion generation. It is built around two key ideas:<br>(1) a hybrid motion representation that combines explicit global root motion with a compact latent embedding of body motion, and<br>(2) an autoregressive two-stage transformer denoiser that generates motion in a streaming fashion while conditioning on online text prompts<br>and flexible, spatiotemporally sparse kinematic constraints over long horizons.
Motion Tokenizer. The encoder first embeds the patchified body motion into a latent representation. This latent body motion is concatenated<br>with the patchified global root motion to form our hybrid representation, which is decoded back to reconstruct the body motion.
This hybrid representation balances precise, interpretable root control (useful for global scene-space constraints) with efficient generative<br>learning in a lower-dimensional latent space for body motion.
Autoregressive Two-Stage Transformer Denoiser. (Left) Conditioned on a variable-length history context and optional spatial goal constraints, the<br>autoregressive denoiser predicts a sequence of C...