VRAM Management Part 2: Beyond the Limits of Physical VRAM | pixelcluster's GPU blog

Earlier this year, I blogged about work I did to improve VRAM management<br>for games. Now, after many months of floating around in mailing lists, the kernel patches<br>are finally merged upstream and queued for Linux 7.3! Hooray!

To celebrate, let’s look a bit deeper at one sentence I wrote in my previous post:

[Games] should perform much more stable - as long as the game itself doesn’t<br>use more VRAM than you actually have.

So, one may ask: What if they do, in fact, use more VRAM than you actually have?

Typical expectations for this seem to be that once this happens you’re pretty much screwed.<br>Games will start crashing left and right, performance plummets to unplayable levels,<br>a good gaming experience becomes impossible.

But is that really just an unavoidable fact of life? What really makes running out of<br>VRAM suck so hard? And, most importantly: How can we make it suck as little as possible?

Setting expectations

In theory, running out of VRAM should exclusively be a performance issue, not a stability one.<br>Support for overcommitting VRAM has existed for as long as GPU drivers have: If the driver<br>overcommits VRAM, you are generally allowed to request as much VRAM as you’d like,<br>and you’ll get as much as the kernel driver decides it can fit into the physical memory that exists on GPU.

On the performance side, the big-picture reason for bad performance when you run out of VRAM is fairly simple.<br>As soon as the game requests more VRAM than is physically present, some of the game’s memory will<br>have to be moved/evicted to CPU RAM instead. For the GPU, accessing CPU RAM is much slower than VRAM:<br>Not only is CPU RAM slower than a dedicated GPU’s VRAM in general, all memory accesses also<br>have to go over the PCI bus. The PCI bus adds latency and is typically also the limiting factor in bandwidth when<br>fetching from CPU memory.

Due to PCI speed limitations, there are some truly unavoidable performance constraints when overcommitting VRAM.<br>Assuming the GPU is hooked up via a PCIe 4.0x16 connection, you get a little less than 32GiB/s of bandwidth. Each<br>millisecond, that PCIe bus can transfer ~32.2MiB of data. For a minimum framerate of 30 frames per second (33.3ms<br>per frame), the absolute maximum amount of data the GPU is able to access is ~1,075.5MiB, a tiny bit over 1GiB of<br>data. In other words, if so much memory gets evicted that the GPU needs to fetch more than 1GiB from evicted memory<br>in one single frame, it is simply impossible to still hit 30 FPS.

Not all memory is equal

At the same time, just reading a little bit of CPU memory on the GPU is not immediately a death sentence for performance.<br>In fact, GPU drivers sometimes decide to let things like command buffer data and related allocations live in CPU<br>RAM even when there’s plenty of VRAM available! Whenever the GPU executes these commands, it has to access CPU memory, and yet<br>in these cases everything runs completely fine. So what makes these accesses different - why are they fine and yet<br>running out of VRAM seems catastrophic?1

One thing that influences the calculus significantly is caching. Since the access latency in case of a cache hit is the same<br>regardless of whether the cached memory lives on CPU or GPU, the high initial cost of fetching over the PCI bus can be<br>amortized by cache hits (to some extent). We can estimate latency differences between fetching CPU RAM and VRAM by writing microbenchmarks<br>that measure access latency for different buffer sizes<br>(using an adversarial access pattern to minimize cache hitrates as far as possible). The result you get may<br>look something like this (captured on RDNA3):

As expected, if the buffer fits into L2 (or any higher-level cache), access latencies are exactly the same for<br>memory backed by CPU RAM and memory backed by VRAM, because the data gets fetched directly from cache in either case.<br>At a size of 6MB (the L2 cache size on RDNA3), CPU memory latencies go up to about 2400 cycles per access,<br>while device memory latencies stay within the same rough ballpark.<br>Note that VRAM accesses also go through the Infinity Cache, but CPU memory accesses do not (they hit PCIe directly on an L2 miss).<br>I suspect this is because the Infinity Cache sits directly on top of VRAM, so any access that doesn’t hit VRAM also doesn’t reach<br>the Infinity Cache.

Obviously, memory doesn’t start off with being cached anywhere, so the first access will still have considerably higher<br>latency. Also, losing the Infinity Cache definitely hurts as well: PCIe fetches seem to have somewhere around 7.3x as much<br>latency than an Infinity Cache hit, and around 4.6x as much latency as a fetch from VRAM.<br>This increased latency needs really high cache hitrates to fully amortize the cost of going over PCIe.<br>That means there is only a small set of use cases where using CPU memory has such minuscule slowdowns<br>that you’d actively decide to use it in favor of VRAM when you...