How Zalando Built an In-Process Client-Side Load Balancer for One Million Requests per Second - InfoQ
BT
InfoQ Software Architects' Newsletter
A monthly overview of things you need to know as an architect or aspiring architect.
View an example
Enter your e-mail address
Select your country
Select a country
I consent to InfoQ.com handling my data as explained in this Privacy Notice.
We protect your privacy.
Close
Helpful links
About InfoQ
InfoQ Editors
Write for InfoQ
About C4Media
Diversity
Choose your language
En
中文
日本
Fr
Aug26,2026
AI Security & Privacy Engineering Certification
Secure and govern production AI systems, from sensitive data to guardrails, evals, and audits.<br>Online. Register now.
Aug13,2026
Architect Certification
Distributed systems, decentralized decisions, platform engineering, and AI architecture.<br>Online. Register Now.
Aug21,2026
Engineering Leadership Certification
Work through leadership decisions with senior peers facing similar technical trade-offs.<br>Online. Register Now.
Nov16-20,2026
QCon San Francisco
What's working across AI, architecture, and leadership, from the teams doing it.<br>Register. Early bird ends August 11.
Dec15-16,2026
QCon AI New York
Production AI across agents, context, evals, security, and infrastructure, from the senior engineers building it.<br>Registration open.
Apr13-16,2027
QCon London
What early-adopter teams have proven in production, across 15 engineering tracks.<br>Register. Early bird ends August 11.
InfoQ Homepage
News
How Zalando Built an In-Process Client-Side Load Balancer for One Million Requests per Second
Architecture & Design
How Zalando Built an In-Process Client-Side Load Balancer for One Million Requests per Second
Jul 25, 2026
min read
by
Renato Losio
Follow us on
Youtube232K Followers
Linkedin26K Followers
InstagramNew
RSS19K Readers
X57.1k Followers
Facebook21K Likes
BlueskyNew
Listen to this article - 0:00
Audio ready to play
Your browser does not support the audio element.
0:00
0:00
Normal1.25x1.5x
Like
Reading list
The engineering team at Zalando recently described the design and implementation of an in-process, client-side load balancer for a high-throughput API handling around 1 million requests per second. The result was more predictable latency, a drop in infrastructure costs, and better visibility into where failures actually originate.
Zalando's Product Read API is designed to serve millions of requests per second with single-digit-millisecond latency across the 25 markets of one of the largest European online fashion retailers. Its batch endpoint fans a single request into up to 100 parallel calls to individual product pods, each transiting Skipper, the shared cluster edge load balancer. As a batch waited on the slowest of 100 hops through infrastructure the team didn't own, they could not separate Skipper latency spikes from their own. To address that, they moved routing for high fan-out internal traffic in-process, while keeping Skipper for edge and single-GET traffic. Conor Gallagher, senior principal engineer at Zalando, explains:
We decided that for high fan-out internal traffic, the routing decision should live inside the calling process itself. Edge traffic, where Skipper excels, should stay exactly where it is. We were not replacing Skipper; we were graduating the internal fan-out path to a client-side load balancer that runs inside the process.
Product-sets routing directly to product pods. Source: Zalando blog
The team reimplemented Skipper's exact algorithm (xxHash64, 100 virtual nodes per endpoint) so both paths route identical keys to identical pods during migration, avoiding cache splits, with unit tests pinning this behavior. Gallagher writes:
This means that adding or removing an endpoint remaps only about 1/N of keys, minimising cache churn. And because both Skipper and our library use the same hash function and the same number of virtual nodes, they produce identical rings for the same set of pods.
The article covers the engineering decisions and challenges of client-side routing, including consistent hashing, occupancy-based load balancing, cache-aware scaling, rollout strategy, observability improvements, and experiments with availability-zone routing.
The team replaced control-plane-crashing polling with a watch-based Kubernetes informer, rebuilt a pipeline that took hours into fast, reversible deploys, and ramped the load balancer from 1% to 100% behind toggles, shrinking Skipper's fleet from 50+ pods to 8 and cutting deployment daily cost from $450 to $110. To eliminate scale-up latency spikes, they added N-ring fade-in over 30 seconds on a ^2.5 curve so new pods warm only the products they'll actually serve. Werner Vogels, CTO of Amazon, writes:
I am not a big fan of doing this client-side, for a whole slew of reasons, but this is impressive engineering by the Zalando team. I like the "lessons learned" they conclude with.
Alexey Kuznetsov,...