Any Engine, Any Topology, Any Infrastructure: How We Designed Modelplane · Modelplane Blog← All posts<br>Cluster-level inference tooling is good now. What's been missing is a fleet-level API. This post covers the one we designed, how its shape accommodates any engine, in any topology, on any infrastructure, and, now that v0.1 has been out for a couple of weeks, what's actually under the hood.

The problem#

Inference outgrew the single cluster the same way cloud-native workloads outgrew it a decade ago. Every important horizontal ecosystem eventually converges on a neutral control layer that lets independent parts work as a coherent whole. Kubernetes did it for compute, and Crossplane for cloud infrastructure. Inference is reaching that point now.

A team picks a serving engine, stands up a cluster, installs a GPU operator, wires in a gateway, and serves a model. Inside that one cluster, the tooling is good. vLLM, SGLang, and TensorRT-LLM are fast and well-understood, schedulers handle placement, and gateways handle routing. The cluster-level problem is, for the most part, solved.

The fleet-level problem is not. Organizations running inference at scale don't run a cluster. They run tens or hundreds of them, across clouds, neoclouds, and on-premise, with different accelerators in each. At that scale the questions stop being cluster-shaped:

Placement is fragmented. One cluster is GPU-starved while another sits idle, and there's no fleet-wide view that can see both and decide.

Capacity is trapped in islands. Each cluster provisions its own nodes independently, with no mechanism to treat the fleet as one pool.

Every team rebuilds the same thing. Weight distribution, cross-region routing, fleet autoscaling, sovereignty policy: every serious operator is building this in house, from scratch.

So the hard question isn't "how do I serve this model on this cluster." It's "what API lets me describe a deployment once and have it run on any engine, in any topology, on whatever hardware is available, anywhere in the fleet?"

That API is what we set out to design. This post covers its shape, the decisions behind it, and, since we're a few weeks past the v0.1 launch now, how it's actually built underneath.

One API, two roles#

Modelplane runs as a control plane on its own cluster, above the inference clusters that serve models. Its API is two small sets of resources, one per role, with everything in between composed for you.

Platform teams declare the fleet:

InferenceCluster : a GPU cluster in the fleet, provisioned by Modelplane or brought as is.

InferenceClass : a hardware recipe describing the devices a node pool offers and how to provision it.

InferenceGateway : the front door for the fleet's endpoints.

Developers declare a workload:

ModelDeployment : a model's engines, replica count, and serving topology.

ModelService : one OpenAI-compatible endpoint across the replicas it selects.

From those, Modelplane composes what sits in between: a ModelReplica per cluster and a ModelEndpoint per replica, then reconciles the fleet to match. A developer never names a cluster, and a platform team never touches a model rollout. The whole design rests on that separation, plus one more decision: the developer-facing API describes the shape of a deployment and nothing about the engine's internals.

Abstract around engines, not over them. Every system that tries to abstract over engines eventually falls behind them. Modelplane describes the shape a deployment takes and leaves the engine to do its work. The next three sections are that one idea applied three times: to engines, to topologies, and to infrastructure.

Any engine#

We decided not to model the engine. There is no field for tensor parallelism, no quantization enum, no schema for KV-cache transfer. A ModelDeployment says how many pods to run, on how many nodes, with which devices, and points at a container image and its args. The engine flags inside that image, things like parallelism, quantization, and KV-cache transfer, are yours. Modelplane never reads or injects them.

apiVersion: modelplane.ai/v1alpha1<br>kind: ModelDeployment<br>metadata:<br>name: qwen-demo<br>namespace: ml-team<br>spec:<br>replicas: 1<br>engines:<br>- name: qwen<br>members:<br>- role: Standalone<br>nodeSelector:<br>devices:<br>- name: gpu<br>count: 1<br>selectors:<br>- cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0<br>template:<br>spec:<br>containers:<br>- name: engine<br>image: vllm/vllm-openai:v0.23.0<br>args: ["--model=Qwen/Qwen2.5-0.5B-Instruct"]<br>Engine flags move faster than any API we could keep up with. When vLLM added --kv-transfer-config for its NIXL connector, you could run prefill and decode disaggregation on Modelplane the same day, because to Modelplane it's just another arg. Had we modeled the engine, we'd have shipped a release to add a field for it, then done it again every time an engine grew a new knob. The catch is that we can't check your flags for you: get one wrong and the pod crashes instead of the API...