How we reduced API requests in the the HCCM
Engineering
How we reduced API requests in the Hetzner Cloud Controller Manager<br>July 28, 20268 min read
Back to Overview
The four controllers behind HCCM<br>A shared cache for the node controllers<br>Starting with the routes controller<br>Working upstream<br>Measuring the impact<br>Wrapping up
In this article
The four controllers behind HCCM
A shared cache for the node controllers
Starting with the routes controller
Working upstream
Measuring the impact
Wrapping up
Close
TL:DR<br>The "hcloud-cloud-controller-manager" (HCCM) links Kubernetes® with Hetzner Cloud and Robot so that cluster resources can be matched with servers, routes, and Load Balancers. In its previous form, HCCM performed more API lookups than necessary, especially when nodes were added or removed and when route checks ran despite no changes in the cluster. A shared cache now reuses server data across controllers, while route updates are triggered mainly by relevant events instead of a fixed timer. This reduces API requests against the Hetzner Cloud API by about 76% during scale-up and by 94% in steady state. The routing change was also contributed to Kubernetes itself, and HCCM v1.33 includes the full set of improvements.
The "hcloud-cloud-controller-manager" (HCCM) is the bridge between your Kubernetes cluster and the Hetzner Cloud and Robot APIs. It implements the interfaces defined upstream in Kubernetes’ "k8s.io/cloud-provider" package, which lets cloud providers plug their own logic into a shared set of controllers. In HCCM, four of these controllers do the work.
The four controllers behind HCCM<br>The node controller initializes each Kubernetes node, attaching metadata such as the instance type and, crucially, the provider ID. That provider ID is the ID of your Hetzner Cloud or Robot server, prefixed with "hcloud://" or "hrobot://". It is immutable and serves as the unique link between a Kubernetes node and its counterpart in the API. Beyond initialization, the node controller also runs a status-update loop every five minutes by default and is tunable via "--node-status-update-frequency". An uninitialized node makes no API call, but each already-initialized node triggers one call to "InstanceV2.InstanceMetadata".<br>The node lifecycle controller watches Kubernetes nodes that have stopped being "ready". By default it checks every five seconds whether such a node still exists in the cloud provider and whether it has been shut down. If a node is shut down, the controller applies a corresponding taint. If the associated Hetzner Cloud or Robot Server no longer exists, the node object is deleted.<br>The routes controller adds one route per node to the Private Network, mapping each Kubernetes node’s Pod CIDR to its private IP. For example, if node-1 has private IP "10.0.0.2" and Pod CIDR "10.244.0.0/24", HCCM creates the route "10.244.0.0/24 → 10.0.0.2". Traffic destined for any pod on node-1 is then routed directly to that node over the Private Network. This avoids the overhead for an overlay network like VXLAN.<br>Finally, the service controller creates and configures Hetzner Cloud Load Balancers for every Kubernetes "Service" of type "LoadBalancer".<br>Each of these controllers talks to the Hetzner APIs, and that traffic counts against your account’s rate limit. The work described below was aimed at cutting unnecessary requests without sacrificing correctness.
A shared cache for the node controllers
First we turned to the node and node lifecycle controllers. Both are built on the cloud provider’s InstanceV2 interface, which exposes a node’s metadata ("InstanceMetadata"), whether it has been shut down ("InstanceShutdown"), and whether it still exists ("InstanceExists").<br>These controllers get especially noisy during scaling events. The node controller generates a lot of traffic during scale-up and scale-down, and the node lifecycle controller adds to it. The lifecycle controller can be particularly aggressive when a group of nodes is shut down: every five seconds it checks whether each node still exists ("InstanceExists") and, if so, whether it is shut down ("InstanceShutdown"). Since those are two sequential method calls, every check costs two API requests. That repeated pattern is exactly where a cache earns its keep.<br>A scale-up also needs information about every Kubernetes node, which suggested a particular design: a cache that pulls all servers in a single request, holds them for a configurable interval, and refreshes them together rather than one at a time. The routes controller already had a cache of this kind, so we reworked that cache to fit our needs and made it shareable across the controllers.<br>The default TTL is ten seconds and can be adjusted. The caching strategy is configurable too: the cache can pull and store each server individually instead of all at once. We fetch all servers by default, because a single batched request is what cuts API usage significantly.<br>The result is that bursts of activity...