NVIDIA Exemplar Cloud: Lessons for Unlocking Full Performance on AI Infrastructure | NVIDIA Technical Blog

Technical Blog

Subscribe

Related Resources

Data Center / Cloud

English한국어中文

NVIDIA Exemplar Cloud: Lessons for Unlocking Full Performance on AI Infrastructure

Jul 30, 2026

By Emily Potyraj, Pavan Sridhar, Sriharsha Niverty, Suryakant Patidar and Charlie Huang

Like

Discuss (0)

AI-Generated Summary

Like

Dislike

Material differences in training throughput across clusters built from identical NVIDIA H100, GB200 NVL72, or GB300 NVL72 systems result primarily from compounded configuration gaps at the kernel, hypervisor, BIOS, and NVIDIA Collective Communications Library (NCCL) levels, frequently causing deployments to miss the 95% threshold for NVIDIA Exemplar Cloud validation.<br>Four real-world case studies highlight recurring sources of performance loss: missing SMMU capabilities and improper virtualization configuration on NVIDIA Grace CPUs; CPU C-state and NUMA misconfiguration leading to sub-optimal turbo frequencies and memory locality; insufficient NCCL queue-pair concurrency on high-bandwidth fabrics such as ConnectX-8 SuperNICs; and failure to propagate NCCL topology files into containers, resulting in silent and severe AllGather/ReduceScatter slowdowns.<br>Infrastructure engineers can close performance gaps by systematically verifying SMMU and VM kernel capabilities, ensuring CPU power management and NUMA/process bindings are optimized, tuning NCCL queue-pair concurrency to match fabric scale and workload, and guaranteeing all topology/environment variables are accessible inside the intended containerized training environment.

AI-generated content may summarize information incompletely. Verify important information. Learn more

Two AI computing clusters built from identical NVIDIA H100, GB200 NVL72, or GB300 NVL72 systems can deliver materially different training throughput. We routinely see 8% to 12% gaps between partner deployments and the corresponding NVIDIA reference architecture (RA) on the same workload, same model, same global batch size.

The cause is often a stack of configuration choices in the kernel, hypervisor, BIOS, and NVIDIA Collective Communications Library (NCCL) settings, each costing a few percent, that compound into a gap large enough to miss the 95% threshold required for NVIDIA Exemplar Cloud validation.

This post walks through four debugging investigations from real partner clusters. Each diagnostic isolates a distinct layer of the stack: system memory management unit (SMMU) and page-table behavior on NVIDIA Grace CPU; power management and non-uniform memory access (NUMA) placement on x86-based CPU; NVIDIA NCCL queue-pair concurrency on 1.6 Tbps fabrics; and silent hardware-installation defects. The post also shows the specific signal in perf, NVIDIA Nsight Systems, or NVIDIA NCCL tests that pointed to the root cause, alongside the tuning change that closed the gap.

Infrastructure engineers and performance architects who already run these benchmarks can benefit from these diagnostic patterns we use internally to run against their own clusters before formal RA validation.

Prerequisites

To reproduce the diagnostics in this post, you will need:

An NVIDIA HGX H100, HGX H200, HGX B200, GB200 NVL72, or GB300 NVL72 systems cluster with NVIDIA Quantum InfiniBand or RoCE interconnect.

A distributed training workload with stable iteration timing—NVIDIA NeMo on Llama 3 model, NVIDIA Nemotron, or DeepSeek configuration is a reasonable reference.

Root access on at least one node for perf, BIOS/UEFI changes, and kernel parameter changes.

nccl-tests built against the same NCCL version your training stack uses, NVIDIA Nsight Systems, and Linux perf with kernel symbols available.

Common patterns behind training performance gaps

Recent Exemplar training engagements show that performance gaps rarely come from a single obvious failure. More often, they come from configuration details that become visible only under workload pressure. Some recurring patterns include:

Grace and virtualization readiness: Missing platform capabilities, SMMU overhead, IOMMU behavior, or page-size settings that don’t match the expected configuration.

CPU power and process placement: Cores running below expected turbo frequency, ranks or helper threads placed on the wrong cores, or NUMA/PCT bindings that don’t match the platform topology.

Runtime topology: Host topology files or NCCL settings that are correct on the node but missing inside the workload container or launcher environment.

Fabric and collective behavior: NCCL settings that don’t match the target fabric, message size, or scale of the training workload.

Application-to-platform binding: Training processes binding by core ID or rank order instead of topology-aware affinity.

These aren’t the only causes of training performance gaps, and checking them doesn’t replace validation with real applications.

Four case studies below show...