Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design

Transformer Transformer

A Unified Model for Motion-Conditioned Robot Co-design

Huy Ha,<br>Karen Liu,<br>Shuran Song

Paper

arXiv

Video

Code

Not all robots are created equal

Not all robots are created equal — but what if you could design one for a specific task?<br>We propose Transformer Transformer , a unified model that does exactly this: hand it<br>a manipulation demonstration, and it generates a complete robot — every link, joint, motor, and<br>inertial property — optimized for that motion. We fabricated one such design for cloth<br>flinging on an ALOHA2 bimanual platform; it reduced<br>tracking error by 73% and max joint speed by 30% versus the original.

Behind that result is a diffusion transformer trained on RoboTokens , a unified<br>tokenization of robot embodiments, states, and actions. The same architecture spans embodiment spaces<br>(wheeled bimanual, quadrupeds, humanoids) and use cases (embodiment generation, cross-embodiment<br>control). Rather than overfitting to one reward function, it is a dynamics model whose reward-agnostic<br>predictions are converted into reward-specific value predictions at inference time, then used to steer<br>embodiment diffusion through a procedure we call Dynamics Self-Guidance . Experiments<br>across three design spaces show zero-shot optimization of unseen rewards and trajectories, improving<br>performance and runtime over an evolutionary baseline.

Technical Summary Video

A 17-minute walkthrough — or scroll for the figure-and-caption version.

What is the best robot<br>for a given manipulation task?

A robot's embodiment decides what tasks it can do well. We pour effort into making robots smarter<br>— bigger datasets, better algorithms, more capable policy architectures — but a bad embodiment can hamper<br>even a great policy. You can collect all the tossing data in the world, but if your robot's shape is far<br>from optimal, it might toss itself instead of the ball.

So let's frame the problem concretely. Given as input a target end-effector motion and a reward function,<br>we want to generate a complete robot embodiment for the task — by complete we mean every link,<br>joint, motor, and inertial property, plus a controller that drives the whole thing. We call this<br>motion-conditioned robot co-design .

A randomly procedurally-generated robot attempting the target motion: it falls before it can<br>finish 🙃

Demonstrate, Generate, Validate

Our framework reframes robot co-design as a three-step process: you demonstrate the<br>desired end-effector motion (e.g., from a UMI3 human<br>demonstration), our model generates an optimized embodiment, and the same model<br>validates the design by directly controlling it. The same network plays three roles —<br>generator, critic, and cross-embodiment controller — by training jointly on one unified<br>tokenization of robots.

Three steps, one model. Given target end-effector motions (e.g., from a human demo) and a<br>user-defined reward, Transformer Transformer generates a complete robot — geometry, kinematics,<br>inertia, motors — then controls that robot to track the motion. No separate optimizer, critic, or<br>controller pipeline.

The paper has three parts: a unified robot representation (RoboTokens), a unified<br>architecture (Transformer Transformer), and a unified training objective<br>(Dynamics Self-Guidance). You have until the architecture section to figure out the pun 🥸

PART 1

RoboTokens<br>A Unified Robot Representation

Every robot becomes a typed token sequence. Blue tokens encode embodiment (links,<br>joints,<br>motors); orange tokens encode dynamics (states and actions). The same tokenizer ingests a quadruped, a<br>humanoid, or a dexterous hand — no per-robot adapter — so a single model can learn across all of them.

If we want one model to co-design any robot, the network needs a shared vocabulary that spans<br>every embodiment. RoboTokens are that vocabulary: typed tokens for every link, joint, motor, state, and<br>action — organized so a single sequence describes both a robot's embodiment (time-invariant) and<br>its dynamics (time-varying).

RoboTokens has the following key design goals:

Complete. A RoboToken sequence captures everything about a rigid articulated robot: 5<br>embodiment token types (links, fixed joints, rotating/sliding joints, ball joints, motors) plus state<br>and action tokens.

Flexible. Tokens reference each other by ID — a joint token knows the two link tokens<br>it connects — so the same format handles a 6-DoF arm or a 35-DoF biped.

Consistent. The tokenizer canonicalizes redundant spatial offsets and stores inertial /<br>transform data in a single frame, reducing the variance the model has to learn through.

Extensible. New token types slot in without changing the architecture — we add<br>target end-effector pose tokens for trajectory tracking by pointing them at the relevant end-effector<br>and timestep IDs, no retokenization of the rest of the robot required.

Optimizable. Unlike autoregressive text formats like...