Patronus AI | Getting GLM-5.2 NVFP4 Post-Training off the ground

Announcing our $50 Million Series B 🎉<br>Read Blog Post Here

Use Cases

Research

News

Resources

About Us

Docs<br>Contact usLogin in App<br>Close

Contact usLogin

Getting GLM-5.2 NVFP4 Post-Training off the ground<br>The goal was deceptively simple to state: take GLM-5.2 , a 744B-parameter mixture-of-experts model quantized to 4-bit NVFP4 , attach a bf16 LoRA adapter, and train it with reinforcement learning until it could play a level of Super Mario Bros., emitting button presses, reading the terrain ahead, and running for the flag.<br>Realizing that specification required resolving a set of defects, which fall into three broad classes. The first is arithmetic: the 4-bit base did not fit within the memory available to it. The second is distributed-systems correctness: the warm-start adapter silently loaded the same expert block on every expert-parallel rank. The third, and the most resistant to diagnosis, concerns training stability: the reward collapse described below persisted under every standard remedy we applied, and was resolved only by removing a regularization term rather than introducing one.<br>TL;DR: the entire setup GLM-5.2 is a 744B-parameter MoE (~40B active). NVFP4 base, frozen (the published nvidia/GLM-5.2-NVFP4 checkpoint), which lands as ~110 GB of weights per training GPU on 8×B200 once Transformer Engine loads it (Part I halves that) + bf16 LoRA (rank 64, MLP-only). Trainer: Megatron with tensor/expert parallelism (TP4·EP4·ETP2) on 1 node of 8×B200; for RL, rollouts are served by SGLang on a second node of 8×B200 (disaggregated, so serving never competes with the trainer for memory); orchestrated by miles/slime . RL algorithm: GRPO, 16 samples per prompt. Reward: how far Mario travels through level 1-1, minus a time penalty, plus a flag bonus.One caveat up front: "NVFP4" is selective, not applied everywhere. Only the routed MoE expert weights are actually in 4-bit. The attention layers, the dense early layers, the shared experts, embeddings, lm_head, the norms, and (of course) the LoRA adapter all stay in bf16. In other words, this is a mixed-precision model, and "the NVFP4 model" is shorthand. This is deliberate on NVIDIA's part: the same "keep the sensitive, low-volume layers in higher precision" logic that the related work below leans on.<br>The typical post-training arc is two stages, SFT → policy-RL , and that is the sequence this article follows too.<br>The toolchain<br>Five pieces of open infrastructure do the work here, in two camps: a trainer that holds the weights and takes gradient steps, and a rollout engine that generates episodes, with an RL framework wiring them together.<br>Megatron-LMTrainer NVIDIA's framework for training very large models.github.com/NVIDIA/Megatron-LM ↗Transformer EngineLow-precision kernels NVIDIA's library of FP8/FP4 building blocks (quantized GEMMs, attention, LayerNorm) that actually execute the NVFP4 math on Blackwell. The "secretly-8-bit" memory bug lives here, in how it keeps a transposed copy of each weight.github.com/NVIDIA/TransformerEngine ↗SGLangRollout engine A fast inference/serving engine. During RL it hosts the current policy and generates the rollouts (the Mario episodes). Getting it to serve an NVFP4 MoE with a LoRA overlay on the experts took a specific combination of its runner and quantization backends.github.com/sgl-project/sglang ↗slimeRL framework An open-source RL post-training framework that connects a training backend (Megatron) to a rollout engine (SGLang) over Ray, running the generate → score → learn → sync-weights loop. Notably, slime does not currently support LoRA: it assumes full-parameter training.github.com/THUDM/slime ↗   z.ai/blog/glm-5.2 ↗milesRL framework + LoRA The slime-derived RL framework this project builds on. Its decisive difference from slime: miles supports LoRA : the reason this 744B model can be adapted with a small bf16 adapter instead of full-parameter RL. But miles does not officially support NVFP4 ; combining 4-bit quantization with LoRA (QLoRA) is what our fork adds. The stack is a chain of gaps filled, each layer supplying what the one below it lacks: slime → LoRA (miles) → NVFP4 QLoRA (our fork). That chain is the whole reason a 744B model is trainable here at all. Most fixes in this log (dropping the second quantized copy, EP/ETP-aware loading, DAPO wiring) live in that fork's patches to miles, Megatron, and SGLang.github.com/radixark/miles ↗<br>Getting 744 billion parameters to train at all<br>Before any learning question could be asked, the model had to fit in memory, load successfully, and survive a forward and backward pass. A chain of infrastructure bugs stood in the way: in the environment, in the memory arithmetic, in the serving kernels, and in the distributed loader. Together they are the price of admission for QLoRA on a model this size. The standouts:<br>The 4-bit base that was actually using 8-bit memory<br>NVFP4 stores each weight in 4 bits, so after...