Compute-Optimal Is Not Cluster-Optimal — Sheng ZhaA scaling law converts a compute budget into a model: fix the FLOPs, minimize the predicted-loss curve, read off a parameter count and a token budget. It is the most useful instrument pretraining has, and it is denominated in a currency no cluster accepts. Clusters do not bill FLOPs. They bill GPU-hours.

I have been pulling at this thread on this blog for a while. The post on model size and training horizon ended by converting compute into machine time,

Tmachine=CtrainρMFU⋅ρgoodput⋅S,T_{\text{machine}} = \frac{C_{\text{train}}}{\rho_{\text{MFU}} \cdot \rho_{\text{goodput}} \cdot S},Tmachine​=ρMFU​⋅ρgoodput​⋅SCtrain​​,<br>and the pretraining research map filed MFU-aware comparison under where scaling laws get hard: two designs can tie on loss-per-FLOP and differ on loss-per-GPU-hour, and the second number is the one on the invoice. Our new paper with Soumajyoti Sarkar and Yuxin Tang takes that observation to its conclusion. The framework, MOSAIC, folds the systems stage into the scaling-law stage: one optimization picks the architecture, the token budget, and the distributed execution layout, for a specific cluster and a specific training window. The title gives away the finding: compute-optimal is not cluster-optimal.

Optimizing the numerator

Look at the denominator of that fraction. Goodput — how often the job runs, after checkpointing, node failures, and input stalls — is a property of the fleet, fixed before any architecture is chosen. Peak throughput SSS is a property of the hardware. MFU — how well the job runs while it runs — is a property of the architecture under its best feasible parallel layout, and it is not a small correction: two designs with identical predicted loss can differ several-fold in realized throughput.

The convention treats it as a constant anyway. A scaling-law stage picks the architecture and the token budget in FLOPs; a systems stage inherits the architecture and tunes the implementation around it. (The Kimi K2 report, for example, treats sparse-MoE scaling laws and systems tuning as separate stages.) That workflow optimizes the numerator while the cluster charges for the whole fraction.

Sparse MoEs are where it breaks

A mixture-of-experts model replaces the dense FFN with a set of experts and a router, which decouples active parameters — the per-token FLOPs — from total parameters — the memory footprint. That introduces axes a Chinchilla-style law cannot see: sparsity, and the expert split factor GGG, which measures how finely the FFN width is partitioned into experts. The paper fits a joint law over total parameters, sparsity, tokens, and GGG on roughly 150 from-scratch MoE pretraining runs, spanning 104M to 2.7B active parameters and total sizes up to 79B.

The fitted law delivers an uncomfortable result. Under a fixed model-FLOPs budget, predicted loss falls monotonically with sparsity across the entire calibrated range. There is no interior optimum. The FLOPs-only answer is “as sparse as your search space allows,” which is not a design principle so much as the absence of one.

Price the same grid on hardware and the ordering inverts. At a world size of 512 GPUs, wall-clock per delivered model FLOP rises monotonically with sparsity, up to 1.70× the densest configuration. The design that ranks first on loss per FLOP is the one the cluster runs slowest.

The same design grid, scored two ways. On loss per model FLOP (left), sparser and finer-grained wins. On wall-clock per delivered FLOP (right), the same picks are the slowest — 1.70× on the sparsity axis, 2.66× on the expert-split axis. The winners sit at opposite ends. From Figure 8 of the paper.

Putting the cluster inside the problem

MOSAIC replaces the model-FLOPs budget with deliverable FLOPs: the cluster’s raw ceiling times the MFU that candidate can attain under its best feasible layout, times goodput. A candidate is feasible only if the FLOPs its recipe requires fit inside the FLOPs the cluster can deliver while executing it.

The MOSAIC workflow. Cluster inputs enter on the left; a geometry ladder proposes realizable architectures; the performance model prices each in MFU and the scaling law prices each in loss. What comes out is a model configuration together with the layout to execute it. From Figure 2 of the paper.

Selection becomes a bilevel problem: the outer loop minimizes predicted loss over a discrete grid of realizable geometries and token budgets; the inner loop maximizes deliverable FLOPs over parallel layouts — tensor, pipeline, data, context, and expert degrees, micro-batch size, activation checkpointing — subject to device memory. The MFU and memory surfaces are measured lookups rather than symbolic functions, so the paper solves it by structure-exploiting enumeration instead of a generic solver.

A worked example shows what the constraint buys. Prescribe 3.3×10213.3 \times 10^{21}3.3×1021 model FLOPs and give it four p6-B200 nodes for five days. The...