Why we write our own C and C++ engines · LocalAI

★ 48,042<br>Install<br>← All posts<br>Most LocalAI backends wrap somebody else&rsquo;s engine, and that is the right default. llama.cpp, vLLM, whisper.cpp, stable-diffusion, MLX and the rest are maintained by people who are better at those models than we are, and wrapping them costs a Dockerfile and a gRPC shim.<br>Eighteen of our backends do not wrap anything. They are C or C++ ports we wrote from scratch, and each one exists because wrapping the upstream engine would have meant shipping something we could not ship: a multi-gigabyte Python install, a non-portable CUDA-only stack, or a model that had no C++ implementation at all. This post is about what those ports buy, measured, and what they cost.<br>What you get: one file, and memory you can predict<br>Deploying a Python inference stack means resolving a dependency tree at install time, on the target machine, against whatever CUDA and glibc it has. Deploying a ggml port means copying a shared library and a GGUF file.<br>The clearest measurement of that difference is vllm.cpp, our C++20 port of vLLM&rsquo;s V1 serving architecture. Installing vLLM produces a 9.1 GiB virtualenv. Installing vllm.cpp produces a 66 MiB binary. The engine implements the same things the Python original does, including paged KV cache, continuous batching, prefix caching, the scheduler and the sampler, with no Python, no PyTorch and no ggml at inference.<br>The obvious question is what that costs in throughput. On an NVIDIA GB10 running Qwen3.6-27B in NVFP4, greedy, closed loop, against vLLM in its production graphed configuration rather than --enforce-eager:<br>Concurrency12481632vllm.cpp tok/s86.05 159.68 292.34 508.77 801.76 1095.01 vLLM tok/s82.32158.03290.31505.46789.161076.25Ratio1.045x1.011x1.007x1.007x1.016x1.017x<br>We are ahead at all six points, and five of those six are ties. Our run-to-run noise band is 0.5%, and concurrency 2 through 32 land between 0.7% and 1.7%, so the honest reading is that only the single-stream case (4.5%) is clearly outside noise. Output is token-for-token identical to vLLM at every point on that curve. Peak host memory is 24.88 GiB against 28.18 GiB.<br>A tie against a mature CUDA stack is a good result for a 66 MiB binary, and it means the footprint saving is not paid for in throughput. Against llama.cpp on CPU from the same GGUF file, prefill runs 1.18x faster (223.8 against 177.3 tok/s), decode is a tie inside llama.cpp&rsquo;s own spread, and the tokens are byte-identical to its greedy decode. Against MLX-LM on an Apple M4, prefill time to first token is 1.5% ahead and warm total throughput is 97.6% of MLX-LM, a real 2.4% gap that sits entirely in decode.<br>Sometimes the port is simply faster<br>depth-anything.cpp is a port of ByteDance&rsquo;s Depth Anything 3, which gives you metric depth in metres from one ordinary photo, plus per-pixel confidence, camera intrinsics and extrinsics, and a back-projected point cloud. On CPU it is faster than PyTorch running the same model.<br>EngineQuantModel MBLoad msInfer msPeak RAM MBvs PyTorchPyTorchf32516749416.913281.00xC++/ggml q8_014240 319.4 363 1.31x<br>Same model, 1.31x the speed, 27% of the memory, and a load that finishes in 40 ms instead of 749 ms, on a Ryzen 9 9950X3D at 504x336 with 16 threads. The quantized q4_k build is a 99 MB file and stays near-lossless. Output correlates 1.0 with the reference forward pass, component by component, across 37 parity tests.<br>The reason it is faster has nothing to do with writing better matmul kernels than PyTorch. Two positional embeddings, the DPT head&rsquo;s UV embedding and the backbone&rsquo;s bicubic position embedding, were being recomputed on every forward pass with single-threaded scalar sin, cos and bicubic loops, even though they depend only on the input geometry and are identical every call. Caching them removed about 95 ms of host-side overhead per forward, which is most of the gap. PyTorch builds the same embeddings with vectorized operations and never paid that cost.<br>That is the general shape of these wins. The heavy GEMMs are close to a wash, because everyone is calling into the same class of BLAS kernel. The difference sits in host-side work that a Python reference implementation never bothered to optimize, and in not loading an interpreter and a framework to do inference. On GPU the picture flips back to parity: with the ggml CUDA backend and flash attention on a GB10, depth-anything.cpp ties PyTorch&rsquo;s tuned cuDNN at 47.3 ms per forward, and wins only the cold start, loading 1.75x to 2.9x faster.<br>Parity is the gate, speed is the follow-up<br>face-detect.cpp and voice-detect.cpp replaced LocalAI&rsquo;s Python insightface and speaker-recognition backends. Both are the case where we do not claim a CPU speed win, and both shipped anyway.<br>face-detect.cpp runs the whole insightface buffalo chain, so SCRFD detection, five-landmark similarity-transform alignment to 112x112, and the ArcFace embedding, out of one self-contained GGUF...