Why Robot Brains Will Live in the Cloud — Dreamscale Labs

The promise of generalist robotics is automating physical work such that no human has to perform dirty, dangerous, or demeaning labor.

To achieve this, robots must be able to perform a wide range of tasks, adapt to a broad variety of environments, and handle an indefinite number of unexpected scenarios. This requires much more intelligent physical AI models than we have now, especially ones that are trained on extremely large training corpora and are equipped with deep reasoning capabilities. We think this necessarily results in a substantially larger model that just won’t be able to fit on GPUs on-board the robot, and that these ‘brains’ will live in the cloud where they can be served on powerful servers.

Conventional wisdom states that running robotic AI models in the cloud is a terrible idea. For robots, a fast reaction speed is critical for both performance and safety, and running the model over the internet is slow and unreliable. Robot builders want to run the most intelligent models which tend to be large, but due to this seemingly inevitable challenge of remote inference, they are forced to work with the constraints of running smaller models on board the robot itself. Furthermore, how can contending with network latency possibly allow robots to react quickly and safely to spontaneous changes in the world?

It appears that off-board inference introduces an insurmountable number of engineering problems relating to speed, reliability, control and safety, such that it seems much more straightforward to focus on optimising on-device infrastructure. After all, running models locally guarantees the tight control loops that enable smooth and responsive actuator reactions to dynamic environments — which is the reason why a lot of recent research efforts have gone into making inference efficient on edge compute.

However, we believe that we can increase the tractability of this ‘fast intelligence’ problem by flipping the direction of thinking on its head — taking it from the realm of pursuing difficult research that compresses intelligence into a small number of parameters, to the realm of solving the engineering challenges to enable large, capable models to run quickly and reliably over a remote connection.

In this post, we’ll walk through the limitations of on-board inference, how cloud inference addresses these problems, then ultimately show how, with the right tradeoffs, a future for off-board inference can and should exist.

Scaling Laws are Here to Stay

We, like others, also believe that much of the next waves of progress for generalist robotics R&D will adhere to scaling laws.

We first saw sparks of general world knowledge in embodied AI models with DeepMind’s RT-2. Its demo of moving a coke can to an image of Taylor Swift was only possible by leveraging a VLM’s web-scale pre-training to understand information about the world. Robot specific data has scaled with π0, and continues strongly with models up to π0.7 (and beyond), yielding transfer to unseen environments and objects.

Generalist AI uses half a million hours of robot data to achieve exceptionally robust dextrous manipulation. They’ve provided generous evidence for the existence of scaling laws, analogous to LLMs, for embodied AI.

Fig. 01<br>Increasing model size and pretraining data both improve performance. The curves point toward the same conclusion: capability will continue to demand more compute.

Beyond just data, they've also reported that robot performance increases with model size. Models smaller than 7 billion parameters ossify and are unable to learn beyond a certain point, while models in the tens of billions have shown more promise of generalizing. This need for larger model size is accelerated by the recent paradigm shift towards world models, leveraging future state prediction and spatiotemporal priors from video foundation models.

Physical Intelligence has also hinted at moving up one order of magnitude for parameter count too, scaling even to high double-digit billions. While they haven’t quite gotten there yet, we see early evidence with their latest π0.7 VLA’s use of a 14B world model for subgoal image prediction.

For companies like NVIDIA and 1X who use World Action Models (WAMs), these models are even more compute-intensive. NVIDIA’s DreamZero 14B model takes 5.7 seconds to generate a single action chunk even on a H100 GPU. It requires two of the powerful, server-grade GB200 instances with a series of complex system-level and implementation-level optimisations to run a Flash variant at the advertised 7 hz closed-loop control. And while 1X doesn’t disclose information on their models extensively, we know it takes 11 seconds for their 14B world model to generate a 5 second future video prediction.

It’s clear that scaling laws are here to stay.

The State of On-board Compute

Right now, there are only a handful of hardware options for onboard inference. Many...