The 4-bitter Lesson | humans&

Skip to article

The 4-bitter Lesson

Balancing Stability and Performance in NVFP4 RL

By Ziang Li & friends at humans& · July 10, 2026

Table of contents

Introduction

Baseline: A starting recipe with stable training dynamics

Improving Gradient Stability

Four-Over-Six for RL Weights and Activations

Selective Layer Precisions: Spending Where It Matters

Putting it all together: Final Recipe

Final thoughts

References

Citation

Introduction

Interactive figure (requires JavaScript): a timeline of samplers and a trainer trading off-policy staleness and rollout precision against throughput.

RL Training Simulator. The simulator models an asynchronous RL system where samplers continuously generate rollouts while the trainer updates the policy. Policy mismatch arises from both off-policyness (stale rollouts) and quantization error , accumulating into policy drift that eventually degrades reward if it exceeds the optimizer's correction capacity.

RL & Efficiency Knobs. Off-policy, weight sync, batch size, and horizon control the degree of asynchrony and policy staleness. MXFP8 and NVFP4 improve training and rollout efficiency, while dequantized backward, BF16 last 15%, and shared experts improve numerical stability.

How to use it. Explore the throughput–stability tradeoff by varying algorithmic and systems knobs. Increase asynchrony or lower precision to improve utilization, then observe how stabilization techniques recover the stability margin. The goal is to identify configurations that maximize throughput while keeping policy mismatch below the critical threshold where drift accelerates and reward collapses.

The essence of RL is to teach a model through action and consequence; at humans& , we use RL to train models that understand the long-term impacts of their interactions with people. In an RL loop1Specifically a policy gradient algorithm., a model acts, gets a reward, then updates the actions' likelihoods. However, in real-world RL training for LMs, we want to update the policy from observations as soon as they're available, even as other rollouts are still being sampled. For the long-horizon multiplayer rollouts central to our mission, we may even take dozens of training steps as a model completes one rollout.

This introduces the tug-of-war between throughput and stability in RL. On one hand, we want to train on as many examples as quickly as possible2Make each training step faster, each rollout faster, or overlap trainers and samplers more.. On the other hand, most techniques to increase throughput cause the sampled policy and the trained policy to diverge, potentially slowing learning and destabilizing training. Quantization exemplifies this tradeoff: while low precision formats enable faster communication and computation on hardware, quantization hurts stability. Fast and accurate low precision formats like NVFP4 paired with hardware support3Allowing up to 9x more operations per second than 16-bit training on NVIDIA Rubin GPUs. Dense Tensor Core PFLOPS per GPU from NVIDIA's HGX platform specs (B200/B300 from 8-GPU HGX systems, Rubin from the HGX Rubin NVL8 table):GPUBF16FP8FP4B2002.254.59B3002.254.513.5Rubin (NVL8)417.535 have driven large throughput increases in both model training and inference separately.

However, there is no stable, hardware-native 4-bit4While INT4 QAT recipes exist to simulate 4-bit quantization, these have used 16-bit activations (Kimi K2 Thinking), and MXFP4 recipes like in DSv4 have used MXFP8 activations (DeepSeek-V4 on Day 0). Our approach uses NVFP4 quantization for both weights and activations. RL recipe in the open-source community, largely because the sampling and training instabilities compound in RL. In a long-running collaboration with the open source community, we have developed and shared a low-precision RL recipe preserving higher-precision training dynamics. In this recipe, we needed to address instability from the forward pass due to policy quantization errors, from the backward pass due to gradient mismatches, and at the intersection of both due to a small set of particularly sensitive weights. Below, we explain how we addressed each and validated the final recipe.

Note this effort could not have happened without our amazing collaborators at RadixArk and NVIDIA, and their work across the training, inference, and RL stacks.

Baseline: A starting recipe with stable training dynamics

For consistent comparison, unless otherwise noted, all experiments in this report use the Qwen3-30B-A3B model trained with 8k sequence length on DAPO-math-17k datasets.

We start with a baseline recipe that uses the NVFP4 format. NVFP4 is a 4-bit floating-point format that improves throughput and memory efficiency on newer NVIDIA GPU architectures. It uses hierarchical block scaling: each block of 16 values has an FP8 E4M3 scale while the full tensor has a global FP32 scale. Together, these scales can recover a higher-precision...