OOMKilled tells you what happened, not why | NuphosLog inDownload

Use CaseJuly 29, 2026<br>OOMKilled tells you what happened, not why<br>Three failure modes share the OOMKilled status. A field guide to telling a leak from a limit from a changed baseline, drawn from three of our own incidents.

Ling Wu<br>kubernetesinfrastructuredogfoodingincident-responsememory

At 19:59 on a Wednesday in April, the Node process serving our landing page died:

textCopy<br>19:59:59<br>19:59:59 FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory<br>20:00:07 BackOff: Back-off restarting failed container<br>20:00:18 BackOff: Back-off restarting failed container

The supervisor restarted it. About three and a half minutes later it died again, and the pod settled into the crashloop rhythm every on-call engineer recognizes.

The status line most of these deaths wear is one word: OOMKilled. It tells you what happened. It does not tell you why, and the why matters, because at least three different diseases produce the same symptom, and each one has a different fix. Raise the limit on a real leak and you have only delayed the next crash. Hunt for a leak when the limit was simply set too low and you will burn a day proving that nothing is wrong with your code.

This is a field guide to telling those three apart, drawn from three incidents on our own infrastructure. We run Zeabur, a cloud platform where a team of fewer than ten engineers operates clusters across more than eight clouds, so we collect OOM stories at a decent rate. These are ours, failures included. A pattern to watch for as you read: most of the hour an OOM investigation costs is not thinking, it is reassembling evidence from four different places.

Who actually did the killing

OOM deaths come from two different killers, and they leave different exit codes. That difference is a free diagnosis, and it is the first fork in the investigation.

The kernel. When a container's cgroup exceeds its memory limit, the kernel's OOM killer sends SIGKILL: exit code 137 (128 + 9), which nothing can catch and nothing cleans up after. Kubernetes reports the container state as OOMKilled, and the evidence lives in node-level events and cgroup accounting, and in the pod state itself:

textCopy<br>Last State: Terminated<br>Reason: OOMKilled<br>Exit Code: 137

The runtime. A garbage-collected runtime with its own heap ceiling, like V8 or the JVM, aborts from the inside when its heap is exhausted, before the kernel ever gets involved: SIGABRT, exit code 134 (128 + 6), and the word OOMKilled never appears in the pod state at all. The pod just says Error and crash-loops. The evidence is a crash log in the container's own output. Our April incident was this kind: V8 hit its old-space limit and aborted, and the container's cgroup limit was never the binding constraint.

So read the exit code before anything else: 137 sends you to cgroup accounting and node events, 134 sends you to the container's own logs, which are often the single most useful artifact in the whole case. And note the trap in the naming: the runtime kind of memory death, the kind that hit us in April, never earns the OOMKilled label you were grepping for.

Three diseases, one symptom

Once you know who did the killing, the question becomes why the process was that big. In our experience the answer is one of three shapes.

A leak grows monotonically, mostly indifferent to load. Something retains references that should have been dropped: a module-scope cache, a listener registry, a growing map keyed by request. Restarts fix it for a while, and the interval between restarts is roughly constant.

An undersized limit produces a process that is behaving exactly as designed, inside a box that was drawn too small. The curve tracks load and peaks with traffic, and the previous container's logs are clean right up to the kill.

A changed baseline is the sneaky one. Nothing is growing without bound; the working set simply moved above the ceiling, usually because a rollout changed the process's behavior. The service was fine for months, then every replica starts dying within minutes of boot.

Memory curveLogs before the killFirst questionFixLeakClimbs regardless of loadAllocation errors, GC noiseWhat is growing?Fix the codeUndersized limitTracks load, peaks with trafficClean to the endUsage vs. limit?Raise the limitChanged baselineFine for months, saturated within minutes of bootClean, but recentWhat changed?Roll back, or resize deliberately

The reason to classify before fixing is the last column: the three fixes point in different directions.

Three incidents, one of each

Theory is cheap, so here is one real case of each from our own clusters.

The limit that was always too small

A TCP proxy on one of our shared clusters kept getting OOMKilled. The investigation was one comparison long:

textCopy<br>memory limit: 128Mi<br>average usage, normal load: ~130Mi

Nothing was leaking and nothing had regressed. The limit had been set below the...