The data layer tax for robot learning — Rerun

Search documentation / esc<br>Start typing to search the docs

Navigate Open

The data layer tax for robot learning<br>Written by Nikolaus West 1 month ago

Scaling laws are starting to work for robotics, producing capabilities that were unthinkable just a few years ago.<br>End-to-end models predict robot actions directly from sensor inputs, which simplifies the on-robot software but makes everything from data collection to training dramatically harder.<br>LLM teams scaled on mature data infrastructure to improve performance through fast iteration on data. Robotics teams are trying to scale without it.

Most teams build data tooling from scratch because existing infrastructure wasn't designed for the multi-rate and multimodal data that powers robotics learning.<br>Across the data journey from collection to training, common operations are harder and slower than they should be. The cumulative cost, in iteration speed, engineering focus, and GPU utilization, is what we'll call the data layer tax.<br>Reducing this tax is a major lever to move faster and scale in the race towards what looks like the biggest market the world has ever seen.<br>Architecturally, the data layer owns storing, modeling, and accessing data.<br>The data layer for Physical AI is still immature, and the cost is visible at every stage of the pipeline.<br>If you're building or investing in robot learning, this post is a map of where that tax comes from.<br>We'll walk backwards from evaluation to collection, showing how requirements cascade upstream and why the tax compounds with data scale, source variety, and curation sophistication.<br>Policy evaluation<br>An extensive set of “evals” is core to any LLM team’s ability to make rapid progress.<br>Evaluation of robot behavior is much more difficult, which has a cascading effect on the entire pipeline.<br>For robotics teams, even a small real-world evaluation on a trained policy takes hours or days of robot trials and careful design and operations.<br>This means making rapid progress against extensive, repeatable, and fast evals aren’t feasible in robotics.<br>Teams instead rely on proxy metrics that score data quality directly, things like reward models that assess task progress, 3D reconstruction quality as a signal for calibration correctness, or just estimating the jerkiness of trajectories.<br>These tell you whether individual episodes or samples look good or bad, not whether they produce a better policy.<br>Since real evaluation runs are harder to do, it’s important to study each one deeply.<br>Many important decisions come from researchers who are deeply steeped in the data, watching eval rollouts and using their intuition about the full system to decide how to proceed.<br>From a data infrastructure perspective, evaluation looks a lot like collection.<br>You record model inputs, outputs, and targets along with metadata like model version, subtask, and environment configuration.<br>Researchers then review large numbers of rollouts, aggregate them by metrics, and drill into specific recordings.<br>Tracing a rollout back to the training data that caused it often requires manual detective work across disconnected tools and formats.<br>Every point of friction adds up to slower iteration times and insights that don’t feed back to training better policies.<br>Model training<br>Robot behavior learning shares many foundations with other machine learning tasks.<br>What changes is that these models output actions over time.<br>This added time dimension drastically increases the complexity of the data layer that supports training for two key reasons: sample construction and video compression.<br>Sample construction during training<br>When training large models, we must feed the expensive GPUs data fast enough to maximize utilization.<br>Researchers steer the behavior of the model by selecting what data to include and how to sample it.<br>Researchers often want to compose multiple datasets together when training models. It’s common to use a weighted combination of multiple datasets and sometimes even per-time step weights for sampling probability or loss weighting.Consider training a vision-language-action model (VLA) with action chunking like ACT or pi0.5. A humanoid robot model could consume three video streams from head and wrist cameras, positions and velocities from 30+ joints, gripper states, and a language instruction.<br>Architecture overview of the GROOT N1 model by Nvidia.Each training sample in a batch starts from a single time step from one of the episodes in the dataset.<br>For a basic VLA, the sample itself consists of camera frames from each view, the robot's current state, and a chunk of future actions, often the next 50-100 time steps. Somewhere between recording on robot and training these inputs all need to be time-aligned, which is a common source of subtle bugs.<br>Basic sample construction for VLA model training. Note that target actions are taken by looking forward in time from the sample starting point.In this case, a naive...