Karpenter's consolidation behaviour is counter-intuitive
SubscribeSign in
Karpenter's consolidation behaviour is counter-intuitive
drmorr<br>Aug 04, 2026
Share
OK, this story starts off with a colleague messaging me at 9am on a Tuesday morning: “Hey, drmorr, Karpenter is behaving a little bit strangely in this cluster, could you take a quick look?”<br>“Sure,” I said, before I’d gotten my morning caffeine. It’ll probably take about 15 minutes and then I can get on with my day, I thought to myself1. Five hours later, I finally understood what was going on. Two hours after that, I understood it well-enough to be able to explain it to my colleague2. A week or so later, I understood it well enough to write this blog post.<br>Spoiler warning: there’s no bug in Karpenter, and everything in the sequence logically follows from one step to the next, but (at least for me) the outcome was still surprising enough that I thought I’d write a post about it.<br>We’re just a few months out from KubeCon! This publication is one of the ways readers can support our conference attendance, which in turn helps keeps this publication going. Would you consider a paid subscription?
Subscribe
It all starts with a graph
So just to be clear, I’ve filed off all the serial numbers on this post to protect… I dunno, probably nobody, I don’t think there’s anything particularly sensitive here, but anyways. The issue started out with a graph that looks like this:
Figure 1: Hypothetical graph of unused CPU resources over time as stacked timeseries data; all data has been fabricated to protect the innocent.<br>This graph is showing the unused CPUs in a cluster, aggregated by the nodes in the cluster. In other words, each line represents a different node, and the value at any point is the number of available (schedulable) CPUs on that node. It is a stacked graph so that we can get an idea of both the distribution of unused CPUs as well as the total unused CPUs in the cluster.<br>This graph is pretty hard to read! But, if you squint at it for long enough, two things become apparent: first, the “available CPU capacity” in the cluster is pretty constant: we more-or-less always have 5 CPUs available. And secondly, despite this, nodes in the cluster are churning constantly. This pattern in this graph repeated for well over 12 hours. What??? Why would Karpenter be doing this? Is this even Karpenter’s fault???<br>However, after checking the Karpenter logs, I was able to confirm that Karpenter was terminating these nodes because of node consolidation. But I’m still very confused: Karpenter is supposed to consolidate nodes to binpack the cluster better—in other words, after a node consolidation event, the “available capacity” in the cluster should decrease! But that’s not what we see: after a node consolidation event, the available capacity stays constant.<br>To understand what’s happening here, we need to know about two things: what’s running on this cluster, and how Karpenter’s node consolidation works. Let’s first talk about Karpenter node consolidation, because I hear a lot of lies subtly imprecise discussion about how it works3.<br>Who’s consolidating who?
The standard sales pitch4 for Karpenter goes like this: “Install this magic autoscaler and all your problems will go away!”<br>Sorry, that was snarky. Let’s try again. The standard sales pitch for Karpenter goes like this: “Install this magic autoscaler and it will make your cluster much, much cheaper!” That was still snarky5, but is closer to the truth. The question is, “How does Karpenter make things cheaper?” The answer is node consolidation.<br>At a very high level, Karpenter has two control loops: one is the “scale up” loop and the other is the “consolidation” loop (I wrote about analyzing the performance of these two control loops way back in the SimKube 1.0 days). The scale-up loop is designed to be fast: it responds to any Pending pods very quickly, and launches new nodes to get those pods scheduled ASAP. The consolidation loop is slower; it’s constantly on the hunt for the “optimal” configuration of pods and nodes, where “optimal” in this case means “cheapest”. In practice, what this means is that it is constantly trying to shrink the cluster (aka, bin-pack it) by moving pods off of “lightly used” nodes so that those nodes can be terminated6. In theory, this results in lower cost since you’re not paying AWS for as many compute resources.<br>Let’s dig into the consolidation behaviour in a bit more detail; if you read the Karpenter docs on this topic, you might be surprised to learn that there are very few knobs you can turn here! You can only set two parameters (on a per-node-pool basis): consolidationPolicy and consolidateAfter. The first parameter controls how Karpenter performs consolidation, and you can either set it to WhenEmpty or WhenEmptyOrUnderutilized7; the second parameter controls how long a node has to sit around before the consolidation mechanism kicks in.<br>In general, if you’re running Karpenter, you...