Attention Decode on AMD MI450 GPUs: A Gluon Kernel Optimization Guide — ROCm Blogs

Skip to main content

Back to top

Ctrl+K

ROCm blogs

Attention Decode on AMD MI450 GPUs: A Gluon Kernel Optimization Guide

Contents

Attention Decode on AMD MI450 GPUs: A Gluon Kernel Optimization Guide#

July 27, 2026 by Pengzhan Zhao, Lixun Zhang, Lei Zhang antiagainst.

14 min read. | 3480 total words.

Software tools & optimizations

AI/ML, Linear Algebra, Performance, Profiling, Optimization, Hardware, Compiler

HPC

Pengzhan Zhao, Lixun Zhang, Lei Zhang antiagainst

English

-->

Agentic AI applications are pushing LLM inference into a new regime. A request can now reach one million tokens from aggregated long prompts, tool calls, retrieval results, and multi-turn reasoning. During the text generation phase, each new token must attend to all previous tokens from the KV cache, so the kernel needs to repeatedly read past states from HBM. As a result, the performance bottleneck shifts from compute units to the memory system.

The AMD Instinct MI450 series GPU introduces a new set of hardware features for memory-sensitive AI workloads. This blog uses attention decode as a case study and discusses how to design a high-performance kernel on MI450. We will walk through core hardware features, explain how kernels can use these features, and showcase the performance of an optimized Gluon kernel, which achieves 85% of the peak HBM bandwidth on MI450 as an early result.

MI450 Hardware Overview#

MI450 is a large step forward from the MI350 series for AI workloads. Compared with previous generations, MI450 provides more on-chip resources and larger HBM capacity with higher bandwidth. The following table summarizes the key spec changes between MI350 and MI450. A Workgroup Processor (WGP) is the hardware unit that runs a workgroup (named CU in earlier MI-series). In this model, one WGP contains four SIMD32 units, each with 32 lanes. Each SIMD32 has its own Vector General-Purpose Registers (VGPRs), which are used by the Vector ALU unit (VALU) and Wave Matrix Multiply-Accumulate unit (WMMA). All SIMD32 units in a WGP share the same Local Data Share (LDS) memory.

Specification

MI350 series

MI450 series

VGPRs per SIMD

512

1024

Max LDS per WGP

160 KB

320 KB

HBM capacity

288 GB

432 GB

HBM bandwidth

8.1 TB/s

19.6 TB/s

In addition to more on-chip resources, MI450 introduces a specialized hardware unit - TDM, to help move structured tensor data between global memory and LDS. With TDM, the kernel describes the target tensor to access using its memory address, shape, strides, and layout, then issues a bulk data transfer asynchronously. TDM helps accelerate memory access and also makes it easier to write pipelined kernels where compute and memory can be overlapped. This marks a significant change from MI350, where the kernel must issue many small vector loads to move data from global memory to LDS.

Features

MI350 series

MI450 series

Memory instruction

Global/Buffer load to LDS

TDM load to LDS

Load granularity

32/96/128-bit vector loads

Descriptor-based tensor tiles

Memory units per WGP

Moreover, MI450 further extends the workgroup model with workgroup clusters. A normal workgroup runs on one WGP, independently of other workgroups. A cluster lets several workgroups, each running on its own WGP, coordinate through hardware-supported cluster barriers. Workgroups that access the same data can use multicast loads to share data across WGPs. This gives kernels a way to express cooperation at a larger scope, without falling back to a separate kernel launch or using global memory for synchronization.

The figure below shows a kernel programmer’s view of the whole MI450 GPU hierarchy.

Kernel programmer view of the MI450 GPU hierarchy.#

To write the MI450 kernel more efficiently, we will use Gluon in this blog. Gluon is a Triton-based DSL that keeps the tile-based SPMD programming model. Unlike Triton, tensors in Gluon must carry an explicit layout, which describes how each element is distributed across registers, lanes, waves, and workgroups. The layout also affects how the compiler generates instructions to move data among different memory hierarchies. This makes Gluon a lower-level programming language than Triton, and especially useful for kernel experts who want explicit control over generated instructions.

We assume the reader is familiar with the basics of Gluon semantics and its programming model. For Gluon kernel examples, please refer to the MI450 Gluon Examples.

Attention Decode Basics#

With this hardware context in place, we can now turn to the workload: attention decode. We will start with a baseline attention decode kernel, then use it as the reference point for the optimizations in later sections. The attention operation can be expressed as follows: first, we perform a matrix multiplication of Q and K, then apply softmax to get P, and multiply P with V to get the final output O.

Attention formula.#

In...