Fixing AMDGPU's VRAM management for low-end GPUs | pixelcluster's GPU blog

It may sound unbelievable to some, but not everyone has a datacenter beast with 128GB of VRAM shoved in their desktop PCs.<br>Around the world people tell the tale of a particularly fierce group of Linux gamers: Those who dare attempt to play games with only 8 gigabytes of VRAM, or even less.<br>Truly, it takes exceedingly strong resilience and determination to face the stutters and slowdowns bound to occur when the system starts running low on free VRAM.<br>Carnage erupts inside the kernel driver as every application fights for as much GPU memory as it can hold on to. Any game caught up in this battle for resources will<br>surely not leave unscathed.

That is, until now. Because I fixed it.

Q: I don’t care about long-winded rants about Linux graphics drivers! Where do I get moar perf?

A: You need some kernel patches as well as additional utilities to make use of the kernel capabilities properly.

The simplest option is to use CachyOS (with KDE as your desktop). Their kernel includes the patches you need from version 7.0rc7-2 and up, and the userspace utilities are available in the<br>package repositories. All you need to do is use CachyOS’s 7.0rc7-2 kernel, install the packages called dmemcg-booster and plasma-foreground-booster, and you should be good to go.

UPDATE : CachyOS’s 6.19.12 kernel also includes this, now. No need to use the -rc kernel anymore.

Q: I use another Arch-based distro! What now?

The dmemcg-booster and plasma-foreground-booster utilities are available in the AUR as well (plasma-foreground-booster carries the package name plasma-foreground-booster-dmemcg), so you can install them from there.

For the kernel side, you can either use the CachyOS kernel package on a non-CachyOS system by retrieving the package from their repository,<br>or you can compile your own kernel. Installing linux-dmemcg from the AUR will compile the development branch I used to develop this.<br>Being a development branch, this carries the risk of some stuff being broken, so install at your own risk!

If you want to apply the kernel patches yourself, you need these six .patch files:

Patch 1

Patch 2

Patch 3

Patch 4

Patch 5

Patch 6

I’m not sure how easily they apply on specific kernel versions, but feel free to leave a comment if you run into issues and I’ll try to help out.

Q: I don’t use an Arch-based distro (or the instructions don’t apply to me for some other reason)! What now?

Maybe wait a bit. Eventually I’d expect this to trickle down into more distros. If I notice this work being packaged by other distros or being installable by other means, I will update this blogpost.

Q: I don’t use KDE! What now?

A: For games where you care about VRAM usage, you can use newer versions of gamescope. Newer versions of gamescope will also try to make use of these kernel capabilities,<br>so running your games through that should be sufficient. You will still need the dmemcg-booster utility in any case.

Q: I don’t use systemd! What now?

A: All the user-space utilities hard-depend on systemd. Without systemd, you’d need to write your own utilities that make use of my kernel patches.<br>Something needs to manage cgroups in your system, and that something needs to enable the right cgroup controllers and set the right limits (see also the long-winded explanation about how this works).

Q: I do care about long-winded rants about Linux graphics drivers! How does this work?

Let’s first look at what problems we actually run into when we have games running on GPUs with little VRAM.

On a standard desktop system, the game won’t be the only application that runs on the GPU at a time at all. If it’s anything like my system, there’s always at least one browser window with way too many tabs open,<br>plus an assortment of apps (many of which are actually web apps running in their own browsers under the hood). All of this eats up quite a bit of VRAM, as well.

To properly stress-test kernel memory management when working on this issue, I would go ahead and open up nearly every app with an integrated browser engine that I had installed. Viewed in amdgpu_top, the result<br>of that looks something like this:

Ouch, there goes 1/4 of VRAM. Now, let’s try and launch Cyberpunk 2077 on top of that:

As expected, the game uses a lot of VRAM (I cranked the settings really high). However, a lot of memory allocations also end up in a memory region referred to as “GTT”. This is memory that is accessible by the GPU, but physically located in system RAM .<br>From the GPU’s point of view, system RAM memory has to be accessed over the PCI bus. Accessing memory over the PCI bus is typically really, really slow. On my system, instead of the 256GB/s bandwidth VRAM could provide, we’re suddenly<br>stuck with a meager 16GB/s at absolute maximum, paired with significantly worse latency.

Some amount of memory landing in GTT is normal - many games will intentionally allocate memory in GTT because it is...