ZML/LLMD alpha | ZML - Model to Metal
Back to posts
Today we’re releasing ZML/LLMD. It’s a self-contained inference server that runs LLaMa, Gemma, Qwen and Mistral LLMs<br>transparently on 5 architectures : NVIDIA CUDA , AMD ROCm , Google TPU , Intel oneAPI and Apple<br>Metal .
Modern serving features
ZML/LLMD supports modern serving features: continuous batching , paged attention , tensor parallel sharding ,<br>prefix caching , tool calling and does so on all platforms .
Metrics are also exposed in the Prometheus format via the /metrics endpoint.
Supported models
ZML/LLMD alpha ships with support for the following models:
Qwen 2, 3, 3.5 and 3.6 series (dense and MoE)
Gemma 3 and 4 series (dense and MoE)
LFM2.5 series
Mistral 3 and Ministral series
LLaMa 2 and 3 series
DeepSeek, Kimi, GLM, MiniMax and StepFun models are coming soon.
Native HuggingFace, S3 and GCS support
Built on ZML, ZML/LLMD has native support for ZML’s VFS subsystem, which allows for zero-copy loading of models from<br>HuggingFace (hf://), S3 (s3://), and GCS (gs://). This means that you can load models directly from these sources<br>without having to download them first, saving time and storage space. Authentication is handled with the standard<br>environment or paths for each backend.
For instance, loading from HuggingFace is as simple as using the hf:// prefix on any flag that expects a path:
$ docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \<br>--model=hf://Qwen/Qwen3-8B
Notice we’re also passing the HF_TOKEN environment variable to authenticate.
DFlash speculative decoding
ZML/LLMD alpha ships with native support for DFlash on Gemma 4 series and soon Qwen series models. DFlash is a new<br>speculative decoding algorithm that can speed up tok/s/user by up to 10x on supported models.
To use it, just pass the --dflash-model flag when launching LLMD:
$ docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \<br>--model=hf://google/gemma-4-31B-it \<br>--dflash-model=hf://z-lab/gemma-4-31B-it-DFlash
One nice benefit of ZML’s programming model is the compounding of features across platforms. For instance, DFlash<br>transparently runs very well on Intel and Apple GPUs.
Read about DFlash on the Z-Lab blog and the<br>currently released DFlash drafters.
Optimized Docker image
On each platform, ZML/LLMD ships the platform runtime as part of the image in a sandboxed manner. Per file optimization<br>reduces image sizes by 5-10x. On top of that careful ordering of layers and custom compression improves<br>docker pull time by using download/extraction overlapping.
The result is a small image that can be pulled and run in seconds:
$ time docker pull zmlai/llmd:cuda<br>cuda: Pulling from zmlai/llmd<br>...<br>Status: Downloaded newer image for zmlai/llmd:cuda<br>docker.io/zmlai/llmd:cuda
real 0m13.766s<br>user 0m0.021s<br>sys 0m0.022s
Platform<br>Image<br>Size
CUDA<br>zmlai/llmd:cuda<br>1.7 GB
ROCm<br>zmlai/llmd:rocm<br>3.9 GB
TPU<br>zmlai/llmd:tpu<br>280 MB
OneAPI<br>zmlai/llmd:oneapi<br>350 MB
Metal<br>brew install zml/zml/llmd<br>140 MB
Each image ships with everything it needs to run. For instance on Metal the Apple Metal compiler is shipped in the<br>archive, or ROCm itself in the AMD image.
Built-in CUDA Compatibility
On the CUDA platform, ZML/LLMD ships with the CUDA Compatibility layer<br>transparently built-in that enables ZML/LLMD to run on a wide range of driver versions without intervention.
This feature doesn’t require any special configuration as the probe is done automatically at runtime.<br>It is also fully sandboxed and doesn’t require any system libraries to be installed.
Automatic sharding support
ZML/LLMD supports tensor parallel inference on all platforms. It automatically shards the model across multiple devices<br>and handles the communication between them transparently.
Currently only tensor or expert parallel sharding is supported. More control will be added in the near future, for<br>instance pipeline parallelism.
Python-free Execution Path
ZML/LLMD runs on ZML’s own ML framework, ZML, built with Zig, MLIR, OpenXLA. It compiles binaries and runtimes ahead of<br>time into a single, hermetic artifact. Since the ZML programming model is explicit compilation passes, once compiled,<br>there are no hidden compilation happening in the hot path. Latencies are flat and predictable.
Performance
google/gemma-4-26B-A4B-it (bs=16)
NVIDIA H100x2<br>Avg<br>Min<br>Max<br>P50<br>P75<br>P90<br>P95<br>P99<br>P99.9
TTFT (ms)<br>127.64<br>70.39<br>162.08<br>141.91<br>142.04<br>142.1<br>143.1<br>158.28<br>161.7
ITL (ms)<br>12.03<br>0.01<br>85.58<br>11.67<br>12.28<br>13.14<br>16.35<br>68.24<br>74.42
Total tok/s<br>1317.93
Intel B70x2<br>Avg<br>Min<br>Max<br>P50<br>P75<br>P90<br>P95<br>P99<br>P99.9
TTFT (ms)<br>426.05<br>117.36<br>1027.4<br>375.38<br>375.4<br>612.75<br>850.48<br>992.01<br>1023.86
ITL (ms)<br>81.83<br>0.03<br>499.75<br>81.62<br>82.72<br>83.7<br>84.17<br>84.6<br>279.99
Total tok/s<br>179.31
AMD MI300X<br>Avg<br>Min<br>Max<br>P50<br>P75<br>P90<br>P95<br>P99<br>P99.9
TTFT (ms)<br>101.86<br>36.3<br>145.35<br>109.7<br>109.76<br>110.39<br>116.76<br>139.63<br>144.77
ITL...