Kimi K3 Is Here: Efficient Day-0 Support on vLLM | vLLM Blog

Menu<br>Theme

Table of Contents

We're thrilled to announce efficient day-0 vLLM support for Kimi K3, one of the most powerful open-weight models ever released.

Last week, we previewed the production-scale integration work for Kimi K3; today, Moonshot AI's weights are public and the support is live.

Kimi K3 day-0 support on vLLM<br>Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model (16 of 896 experts active per token) built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context window and native vision. For us, the most exciting challenge Kimi K3 brings is making KDA, MXFP4 MoE, KV cache management, prefill/decode disaggregation, speculative decoding, and long-context deployment recipes work together in a runnable serving engine.

The preview post explained the kernel and cache architecture, in particular the challenge of prefix caching that works on recurrent state. This release post is the practical guide: how vLLM adapts to Kimi K3's architecture, the kernel work behind the numbers, and what is ready on day 0.

Quick start

code]:block [&>code]:w-fit [&>code]:min-w-full font-mono" tabindex="0" data-language="bash" data-theme="github-dark-default github-light-default"># See the linked recipes for the exact Docker command.<br>vllm serve moonshotai/Kimi-K3 \<br>--tensor-parallel-size 8 \<br>--trust-remote-code \<br>--load-format fastsafetensors \<br>--enable-prefix-caching \<br>--enable-auto-tool-choice \<br>--tool-call-parser kimi_k3 \<br>--reasoning-parser kimi_k3

The easiest way to run Kimi K3 is to use 8 NVIDIA B300 GPUs or 8 AMD MI355X GPUs with the above command.

Inferact has also trained and open-sourced a DSpark speculator for Kimi K3. Enable it by adding the following option to the serve command:

code]:block [&>code]:w-fit [&>code]:min-w-full font-mono" tabindex="0" data-language="bash" data-theme="github-dark-default github-light-default">--speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'

For more details, including Docker images for various platforms and deployment strategies, refer to the detailed recipes. Because of complicated dependencies, only Docker images are usable now. The Docker images depend on several pre-release dependencies, including FlashInfer.

TL;DR

A 2.8-trillion-parameter multimodal MoE: Kimi K3 activates 16 of 896 experts per token, supports a context window of up to 1M tokens, and is built on Kimi Delta Attention, Attention Residuals, LatentMoE, and native MXFP4 (4-bit) weights.

Up to 370 tok/s per user: vLLM serves Kimi K3 at 118 tok/s without speculative decoding and 370 tok/s (a 3.14× improvement) with DSpark on 16 NVIDIA GB300 GPUs, powered by extensive optimizations for Kimi K3's architecture.

Broad production feature support: vLLM supports speculative decoding, prefill/decode disaggregation, agentic KV caching with Mooncake, tool calling, reasoning output, and structured output, with NVIDIA (Hopper and Blackwell) and AMD (MI355X) support at launch.

Open-source DSpark support: vLLM supports the state-of-the-art block-diffusion speculative decoding algorithm for Kimi K3, trained with vLLM and TorchSpec and open-sourced by Inferact.

Hybrid prefix caching: serving Kimi K3's recurrent and full-attention design required a redesign of hybrid prefix caching over recurrent KDA state. This change now benefits every hybrid linear model.

Kimi K3's architecture, and how vLLM serves it

Kimi K3 architecture innovations<br>Kimi K3 architecture innovations, from the original release blog post.

Kimi K3's architecture departs from a standard Transformer in a few ways, and each one changes what a serving engine has to do. The preview post covers the internals in depth; here we recap what's new and focus on how vLLM adapts to serve it.

Kimi Delta Attention: a hybrid recurrent + full-attention stack

What's new: Most of Kimi K3's layers are KDA, a linear-attention mechanism that keeps a fixed-size recurrent state instead of a growing KV cache, interleaved with periodic full-attention layers that preserve exact global recall. That is what makes a 1M-token context affordable.

How vLLM serves it: A single hybrid KV-cache manager holds two kinds of memory side by side under one scheduler: paged KV blocks for the full-attention layers, and compact recurrent-state blocks for the KDA layers. A dedicated KDA attention backend runs FlashKDA for prefill and a fused CUDA kernel (or the Flash-Linear-Attention/Triton path when running speculative decoding) for decode.

The hardest part is prefix caching across Kimi K3's hybrid cache: full-attention layers store per-token KV, while KDA layers update recurrent and convolution state at every token but cannot afford to retain a snapshot at every possible prefix boundary. vLLM decouples the large physical KDA...