-->

Deploying Kimi-K3 on Day 0: Can a 2.8T Model Run on a Single Node of NVIDIA B300 x8? - Fixstars Corporation Tech Blog

This website uses cookies to improve your user experience on this site. By continuing to<br>view this site, you agree to our company's privacy policy.

I accept

Deploying Kimi-K3 on Day 0: Can a 2.8T Model Run on a Single Node of NVIDIA B300 x8?

Junko<br>|July 28, 2026<br>AI Engineering<br>Edge Robotics

Introduction

On July 27, 2026, Moonshot AI released the model weights for Kimi-K3. Ever since the model was announced on July 16, it had generated significant buzz as "the largest open-weight model in history."

At Fixstars, we deployed the model to a single-node NVIDIA B300 x8 environment on the very day the weights were released and ran an inference performance benchmark. This article is our first-look report. From the start of the download through the completion of the benchmark, we managed to run through the whole process on release day.

What Is Kimi-K3?

Kimi-K3 is a frontier-class MoE (Mixture of Experts) model developed by Moonshot AI. The main specifications published in its model card are as follows.

ItemSpecificationTotal parameters2.8T (2.8 trillion)Active parameters104BExpert configuration16 of 896 experts activated per token (+ 2 shared experts)Number of layers93 (69 KDA layers + 24 Gated MLA layers)Precision of released weightsMXFP4 weights / MXFP8 activations (quantization-aware training)Context length1M tokens (1,048,576)MultimodalNative visual understanding supported (MoonViT-V2, 401M)Reasoning modeThinking mode always on (intensity configurable: low / high / max)LicenseKimi K3 License (custom license)

On the architecture side, Kimi-K3 adopts two proprietary techniques: Kimi Delta Attention (KDA), a hybrid linear attention, and Attention Residuals, a replacement for residual connections. The majority of the attention layers (69 of 93) are KDA, with only 24 conventional (Gated MLA) attention layers. As discussed later, this hybrid configuration directly affects memory management on the inference server.

Another important point is that the model is distributed as MXFP4 weights that underwent quantization-aware training (QAT) starting from the SFT stage. Rather than being quantized after training, it was designed from the outset to be used in MXFP4, so it can be deployed as-is without any additional quantization work.

In the various evaluations published at announcement, Kimi-K3 is rated as delivering the highest performance ever for an open-weight model, coming close to top-tier proprietary models such as Claude Fable 5 and GPT 5.6 Sol. Its strength on coding and agent tasks has drawn particular attention.

Test Environment

ItemDetailsGPUNVIDIA B300 SXM6 x8 (single node)GPU memory288GB HBM3e x8 = 2,304GB totalCPUIntel Xeon 6767P × 2 sockets (64 cores/socket, 128 cores total, 1 thread/core)System memory3.0 TiB DDR5InterconnectNVLink 5.0, 18 links/GPU, 53.125 GB/s/link = 956 GB/s/GPU (unidirectional)OS / DriverUbuntu 24.04.4 LTS (kernel 6.8.0-71-generic), CUDA 13.0, Driver 580.105.08Inference engineSGLang 0.5.16 (Docker container sglang-kimi-k3)

Why SGLang?

At release, both vLLM and SGLang supported Kimi-K3, but we chose SGLang. The reason is its support for DCP (Decode Context Parallelism).

For a hybrid model like Kimi-K3, the memory efficiency of the KV cache determines the effective context length. At release, vLLM did not support DCP, so the KV cache is replicated across all GPUs. SGLang, on the other hand, supports DCP and can distribute the MLA KV cache across 8 GPUs, allowing it to achieve 8x the effective context capacity with the same amount of memory.

Considering the Memory Footprint

A model with 2.8T total parameters would require roughly 2.8TB just for the weights even in FP8, which would not fit within this environment’s total GPU memory (about 2.2TB). However, Kimi-K3’s released weights are provided in MXFP4, and in our measurements the weights came to about 196GB per GPU, or about 1.57TB across 8 GPUs. This works out to fit as-is on a single B300 x8 node — a design that, despite the model’s enormous scale, was clearly built with running "on a single node" in mind. The measured memory breakdown is described later.

Deployment

Downloading the Weights

The MXFP4 weights total about 1.6TB. The download completed in about 1 hour. Partway through, some files got stuck downloading; killing the process and rerunning it completed without issue. Access is heavily concentrated right after release, so if you run into the same phenomenon, we recommend retrying.

Launch Command

docker run --gpus all \<br>--shm-size 32g \<br>--network host \<br>-v /data/hf_cache:/root/.cache/huggingface \<br>--ipc=host \<br>lmsysorg/sglang:kimi-k3 \<br>sglang serve \<br>--trust-remote-code \<br>--model-path moonshotai/Kimi-K3 \<br>--tp-size 8 \<br>--dcp-size 8 \<br>--disable-custom-all-reduce \<br>--mem-fraction-static 0.85 \<br>--reasoning-parser kimi_k3 \<br>--tool-call-parser kimi_k3 \<br>--mamba-full-memory-ratio 5...