Oodle 2.9.14 and Intel 13th/14th gen CPUs | The ryg blog

Skip to content

Follow:<br>RSS<br>Twitter

The ryg blog

When I grow up I'll be an inventor.

Home

About

Coding

Compression

Computer Architecture

Demoscene

Graphics Pipeline

Maths

Multimedia

Networking

Papers

Stories

Thoughts

Uncategorized

Oodle 2.9.14 and Intel 13th/14th gen CPUs

May 21, 2025

There’s a hardware problem affecting Intel 13th/14th gen CPUs, mostly desktop ones. This made the rounds through the press last year and has been on forums etc. for much longer than that. For months, we thought this was a rare bug in the decoder, but from stats in Epic’s crash reports for Fortnite (as well as stats for other Unreal Engine and Oodle licensees) it was fairly striking that this issue really seemed to affect only some CPUs.

Much has been written about this problem already. At RAD we have an official page about it, which includes a link to an older version (when we still weren’t sure what was causing it).

Intel’s statement on the issue confirms that it’s a hardware problem where there’s physical degeneration of the clock tree circuitry over time resulting in clock skew which ultimately makes affected CPUs behave in increasingly glitchy ways.

This is a hardware issue, and Intel has now released multiple versions of microcode updates to try and prevent or at least limit that degradation. But the symptoms are just frequent crashes or errors in certain components. For Unreal-using games, the most common symptoms are

Shader decompression failures (as mentioned on our official page)

Shader compilation errors in the graphics driver

Spurious "out of memory" errors reported by the graphics driver (even when there’s plenty of GPU and system memory available)

These shader (and sometimes other) decompression errors are where Oodle (the data compression library I work on) comes in. Specifically, we notice during decompression that the data we are currently decoding is corrupted because internal consistency checks trip and return an error code. Most commonly, these errors occur as a result of either corruption of on-disk data, or bad RAM. But in this particular instance, we were (and still are) seeing these errors on machines with RAM that passes memory tests just fine and where the on-disk data passes hash validation.

We just released Oodle 2.9.14 which includes an experimental workaround that seems to at least reduce the frequency of failed Oodle decompression calls on affected machines. We’re not yet sure whether it helps with overall crash rate on those machines or if crashes happen with more or less the same frequency, just elsewhere. We’ll have more data in a few months! But meanwhile, the background for that workaround is interesting, so I’m writing it down.

Background: spring 2023 to early 2024

I’ll keep this history section relatively brief, because this is not the main point of this post.

We first started hearing about Oodle crashes on the then-new Intel 13th-generation CPUs around March 2023, concurrently from three separate sources: Fortnite was seeing a big spike in crash reports especially due to shader decompression failures (as mentioned above), with a smaller spike of errors in non-shader decompression; other UE licensees as well as Oodle-but-not-UE licensees were reporting something similar in their crash reports; and we were even getting a few mails from end users asking us about frequent Oodle crashes. (This is rare since RAD/Epic Games Tools is B2B, we don’t normally interact with end users directly).

We did notice that all of the reports seemed to involve recent Intel CPUs. From there follows months of red herrings and dead ends as well as trying (and failing) to reproduce these issues consistently. Condensed version:

Our first lead was that we had seen a lot of instability with a bunch of new demo machines at a recent event. Those occurred on a specific motherboard/CPU combination, and ultimately turned out to be due to bad sound drivers (which corrupted the contents of vector registers in user-mode code, yikes). The first batch of reports we had were all from one single motherboard vendor, so we thought that might be it.

Then we had a report from an UE licensee, still on the same motherboard, but removing the faulty sound drivers didn’t seem to help. They later reported that their crashes went away after they disabled AVX support in the BIOS. Between this and the vector register content corruption for the bad sound drivers, we started going over all SIMD code in Oodle Data with a fine-toothed comb.

By June, we still hadn’t found anything useful; we did have some experiments that disabled vector/SIMD usage in the decoder entirely but that didn’t seem to help, so back to square one.

Around July, so forum posts were making the rounds that disabling the E-cores seemed to help. We also had some reports from other motherboard vendors at the time, ruling out our initial theory that it might just be one.

We still...