GEMM Performance Measurement Methodology Guidelines — NVIDIA CUTLASS Documentation

Skip to main content

Back to top

Ctrl+K

NVIDIA CUTLASS Documentation

Choose version

GEMM Performance Measurement Methodology Guidelines#

Introduction#

As linear algebraic computation - particularly general matrix multiplication (GEMM) - forms the computational backbone of modern artificial intelligence and machine learning applications, the need for precise and robust performance measurement methodologies has never been greater. Rapid advances in NVIDIA GPU architectures continually push the boundaries of hardware and algorithmic efficiency, making it crucial to benchmark GEMM operations with rigorous, reproducible standards. Reliable measurement frameworks not only enable fair comparisons between devices and software stacks but also illuminate subtle bottlenecks, guide system optimizations, and ensure that performance gains reflect true progress rather than artifacts of measurement inconsistency. This document outlines the methodology for measuring GEMM performance to ensure results are accurate, reproducible, and consistent across different NVIDIA devices, firmware, and software.

Benchmark Requirements#

When writing or using a benchmark to profile GEMM performance on a GPU, several techniques must be encoded into the benchmark to ensure stable and accurate measurements.

Separate warmup and profiling iteration loops

A user should be able to separately specify the number of warmup and profiling iterations. The warmup and profiling loops must be separated by calls to cudaProfilerStart/Stop, for use with external profilers, and/or calls to cudaEventRecord, for self-profiling benchmarks. See Warm-ups and Iterations below for recommendations on configuring the number of warmup and profiling iterations.

Minimal gaps between kernel launches

There should be minimal overhead between kernel launches in both the warmup and profiling loops. All data buffers must be allocated and initialized before entering either loop. There must be no additional GPU work between kernel launches, this includes data allocations, deallocations, and copies as well as extraneous kernel launches. CPU work must be minimized so the CPU does not limit the kernel launch rate.

Tensor buffer rotation

Special care must be taken to ensure that all data caches are “cold” at the start of each kernel iteration. To achieve this without reallocating data between iterations and violating the “Minimal gaps between kernel launches” requirement, benchmarks must pre-allocate multiple buffers for every memory surface used by the kernel and change buffers each iteration. See Buffer Rotation below for more details.

Testing Methodology Overview#

Test Type Specification:

There are two main types of tests that are commonly run when profiling GPU GEMM performance: fixed frequency tests and fixed power tests.

Fixed frequency tests run on the GPU at a predefined frequency and are useful to measure architectural and software efficiency characteristics. Precautions must be taken to ensure the chosen frequency can be sustained. When workloads run on the GPU, the computation and data movement to process the work encapsulated result in energy consumption and hence power. It is essential to adhere to the electrical and thermal constraints prescribed for the product, so power and thermal controllers may induce clock variability even when the user specifies a locked clock frequency. To mitigate this risk, a user should use zero-fill data to reduce power consumption and monitor the clock frequencies during testing to watch for variation. If clock frequency variation is observed, the user should select a lower frequency that can be sustained for their testing. Note this is not a set up for measuring the maximum performance of the product.

Fixed power tests run on the GPU with a given power limit and mimic real-world scenarios.

GPU products are regulated by power limits with dynamic voltage and frequency scaling (DVFS) enabled in real-world usage. On reaching the power limit, a throttling mechanism enables the clocks to adjust to operating points within the allowable power budget. This is the standard configuration for GPU products when they are operated as received from the factory. Fixed power tests are the most realistic, but they are also subject to a higher degree of run-to-run variation, device-to-device variation, and sensitivity to firmware versions. As such, fixed power tests must be meticulously controlled to ensure a stable testing environment (refer to Matrix Fill Methodology, Warmups and Iterations, and Input Buffer Rotation below).

Data Collection:

For all tests, it is recommended to collect kernel durations with nsight-systems or CUDA events to calculate the achieved FLOPS/s. The GPC frequency should be profiled with nvidia-smi or NVML to monitor for clock variation and to identify the settled frequency.

For fixed frequency tests, monitor the GPC...