How we cut Flink OOMKills by 91.2%: Zombie block cache, phantom CPUs (and some bonus AI Learnings)Enhance your career, get your certificate as a Data Streaming Engineer | Get your Certificate
Two months after joining Confluent on the Apache Flink team, I was asked if I’d like to investigate and fix Out-of-Memory kill events (OOMKills).<br>OOMKills are a particularly disruptive failure mode for streaming workloads: the Flink Task Manager is terminated by the kernel, in-flight work is lost and the Flink job must recover from its last checkpoint, taking precious seconds.<br>Diagnosing OOMKills is not straightforward because they are a symptom of many possible causes, e.g. leaks, fragmentation, misconfiguration.<br>An impactful and difficult project this early into a new job?<br>I bit immediately, perhaps a bit too naively, not knowing how deep and lengthy it would get.<br>By fixing native-memory leaks and jemalloc fragmentation in Flink's use of RocksDB, we cut OOMKilled Task Managers by 91.2%, and reduced peak memory usage by 13%.
Closing the diagnostic loop
Early in the project we identified a gap in how we diagnose OOMKills.<br>Flink container OOMKills are usually caused by excessive native memory usage but common JVM tools such as Java Flight Recorder or async-profiler primarily profile Java heap and not native memory.<br>Closing that gap meant enabling profiling on Flink's native memory allocator jemalloc which carries runtime overhead.<br>So by design, jemalloc profiling is disabled and related tooling is not bundled in our production Flink Docker images.<br>The tradeoff was sound for steady-state operation, but it made ad-hoc investigation expensive.<br>Capturing a profile meant patching, waiting on a CI pipeline, and migrating the target job, taking roughly a day of turnaround before any data could be collected.<br>In practice, that put jemalloc profiling out of reach for on-call engineers already triaging live incidents.
We closed the gap on two fronts.<br>First, we extended the CI pipeline to build and publish profiling-capable Flink images alongside every base Flink image, so the tooling is always available on demand with no rebuild.<br>Second, we streamlined the operational tooling so that migrating a job to the profiling image and capturing a jemalloc profile takes a couple of minutes and a few clicks.<br>The same tooling automatically captures and uploads information from any OOMKilled job, so the evidence is available when an engineer begins to investigate.
If you’re curious, here’s how we enabled profiling.<br>See the jemalloc profiling wiki for more details:
# Enabling heap profiling: prof:true<br># Report memory still live at shutdown: prof_leak:true<br># Dump a profile every 2^26 bytes (64 MiB) of allocation: lg_prof_interval:26<br># Sample once per 2^16 bytes (64 KiB) of allocation: lg_prof_sample:16<br># Path for dump files: prof_prefix:prof_prefix:/tmp/example/jeprof/prof<br>MALLOC_CONF=prof:true,prof_leak:true,lg_prof_interval:26,lg_prof_sample:16,prof_prefix:/tmp/example/jeprof/prof
The Zombie Block Cache
With the improved tooling, we were able to capture jemalloc heap profiles in frequently OOMKilled Flink jobs.<br>An interesting profile emerged where we saw 7.28GB allocated to RocksDB block cache via BlockFetcher::ReadBlockContents.
bash-5.3$ jeprof --text --cum ... /tmp/jeprof/...heap 2>/dev/null<br>Total: 9738.4 MB<br>0.0 0.0% 96.9% 7461.9 76.6% rocksdb::BlockFetcher::ReadBlockContents<br>...<br>0.0 0.0% 96.9% 3252.7 33.4% rocksdb::MergingIterator::Seek<br>...<br>0.0 0.0% 96.9% 2482.2 25.5% Java_org_rocksdb_RocksDB_createColumnFamilyWithImport<br>...
This was more than twice the configured capacity of 3.25GB!<br>Flink’s RocksDB block-cache-usage metric was also reporting exactly 3.25GB of usage.<br>Knowing that jemalloc profiling uses sampling to construct a statistical representation of the heap profile made me suspect that the heap profile might be skewed and inaccurate.<br>Process memory usage (anon RSS), however, was showing a few GBs of additional unexpected memory usage, indirectly supporting excessive block cache usage.<br>It is only a circumstantial metric at best, as anon RSS may also include unrelated usage, e.g. memory fragmentation.
To confirm the validity of the profile, jemalloc was configured to sample often and minimize memory fragmentation.<br>The heap profiles were still showing excessive block cache usage and malloc_stats ruled out memory fragmentation as there was effectively none.<br>This gave me the confidence that the jemalloc profile captured a memory leak invisible through Flink.
I have the luxury of working with some of the folks who have deep Flink knowledge, including many of the committers to the project.<br>After consulting them, experiments were run and a couple of other possible root causes such as excessive memory usage from iterators and the use-ingest-db-restore-mode feature were ruled out.
The leak, active when the manual-compaction feature is enabled, was eventually traced to an unclosed ColumnFamilyOptions wrapper in the Compactor...