YouTube System Design for Robotics Data Infrastructure | Hebbian Robotics

8 min read<br>When we first started building a data platform for robotics, we weren’t thinking about YouTube.

Robotics teams wanted to be able to upload their LeRobot datasets, search across demonstrations, inspect synchronized camera feeds and robot states, and export curated episodes for training. That sounded like a fairly specialized problem in robotics. So we built Pareto.

Once we got into the infrastructure work, the requirements started to feel familiar. We needed to reliably upload large media files, preserve the originals, process them in the background, generate thumbnails and preview videos, index their metadata, and stream them to a browser without downloading the entire dataset.

“Design YouTube” is a classic system-design exercise. It can feel a little pointless: YouTube dominates, and almost none of us will ever need to build a global consumer video platform at that scale. If we reframe thousands of users uploading single videos to your platform as a single robotics dataset uploading thousands of videos, the similarities become much clearer.

ByteByteGo scopes the exercise to upload and playback: original files go to blob storage, background workers transcode them, a completion flow updates metadata, and the CDN delivers the finished versions. That is the simplified shape above.

An episode of robot data

The classic video platform starts with one dominant object: a video, perhaps with audio and maybe subtitles. A robot-learning dataset has a more complicated unit of data.

An episode of robot data is a multimodal record: it can contain several camera feeds (wrists, top, chest, etc.) alongside joint positions, actions, force readings, timestamps, task descriptions, and other sensor streams. The cameras may be the largest part of the dataset, but they are not useful in isolation. The data only makes sense when it remains connected to what the robot was sensing and doing at that moment.

Pareto searches across episodes, so someone searching for an “orange block” can scrub the surrounding trajectory, compare cameras, and decide whether a demonstration belongs in a training set.

LeRobot v3 makes this concrete. Episodes can share Parquet data files and MP4 shards, so reading an episode means resolving both its structured row range and its timestamp range inside each camera’s video. Once we wrote the data path down, it looked familiar: upload -> process -> store -> index -> play.

Pareto is open source, so you can inspect the system and run it yourself.

1. Reliable uploading

When it comes to multimedia, we get into the art of troublesome large uploads. Connections fail, processes restart, and retrying an entire dataset because the final file was interrupted is expensive. Video platforms address this with resumable uploads and an explicit transition from uploading to ready-for-processing.

In Pareto, we keep receiving source files separate from converting and indexing them. Source files are stored under versioned storage paths, completed batches are checkpointed, and the final manifest is written last. Downstream jobs use that manifest as the completion marker rather than guessing whether a directory is complete. It also lets us trace any generated output to the exact source files and batch that produced it.

Keeping the original dataset also means future processing can change without asking the user to upload the data again.

2. Processing becomes a pipeline

For any site that streams video, uploads have to be processed into multiple versions: different resolutions and codecs, thumbnails, previews, captions, and other assets. The original is one input to a larger background pipeline.

For robot data, we generate sampled frames, thumbnails, preview videos, downsampled state and action series, vector embeddings, search indexes, and Rerun recordings. Each serves a different use: search, browsing, synchronized playback, analysis, or export.

The browser can show a search result without opening the full dataset, just as YouTube can start playback without serving the original upload.

3. Running long jobs

Pareto runs different jobs in the pipeline as durable workflows using Temporal. They have a hosted service but it is also open source and self-hostable.

Workers are separated by capability and by the shape of the work. We shard independent episode ranges across GPU embedding workers, then fan in at a finalization step that merges the shard outputs and runs the global work once. Other jobs that don’t require a preloaded warm model are split into independent chunks that can run across elastic GPU or CPU pools. That keeps bursts of one job class from consuming the capacity another kind of work needs.

Not every generated output needs to be ready before a dataset is useful. We moved many lightweight, recoverable computations to lazy, on-demand paths, keeping the critical path focused on making data searchable and viewable. That...