You Don't Need Kubernetes

I've been sitting on my homelab for over a year, constantly adding to it and expanding its scope. What started as a simple Raspberry Pi cluster for learning purposes during COVID expanded into a mix of x86 NUCs and RPi 4s with 8GB of RAM. The NAS morphed into an iSCSI target, then an NFS share, then was considered for MinIO or Ceph duty before falling back on the elegant simplicity of NFS. The NUCs, the Pis, they sat dormant as goalposts moved, perceived needs changed, and perfect became the enemy of good.<br>I recently had an interview for a Principal Engineer role with a small investment firm. $200k salary, hybrid work schedule, a small team to manage, outcomes to own. It was everything I wanted in my next role, all the career growth I could ask for with a technology estate that was suitable to the company's needs, if not necessarily cutting edge. I was incredibly okay with this, and made it clear this was exactly what I was looking for in my next opportunity.<br>I'm pretty sure I did not get the role.<br>Kubernetes was the guilty party to both of these recent scenarios. My insistence upon remaining on the cutting edge of infrastructure engineering hindered my homelab deployment and likely cost me the role I'd wanted. In the case of the former, its release cadence, constant partner evolution, and lack of a firm spec meant I was always playing catch-up in my homelab trying to find the right combination of OS, control plane, taints, ingress controllers, network overlays, and storage mediums to fucking work consistently. With regards to the job opportunity, it created an appearance that what I really wanted was to work on the cutting edge, and that anything less would be "boring"; this is despite the fact Kubernetes is mentioned exactly one time on my resume, and in the context of a Proof-of-Concept project for the private cloud at that.<br>Funny enough, it was the job interview that opened my eyes to what I'd lost by pursuing Kubernetes so voraciously in my personal time and professional career. Opportunities for personal growth, for enjoying the fruits of my labor, for experimentation, all lost because I was trying to untangle Cilium, or figure out why a pod that worked yesterday went TITSUP today with no other change. Weeks spent figuring out microk8s, and Talos, and k3s, and bare-metal K8s. Months studying for CKA and CKS, of reading Xe's (amazing) blog for homelab ideas and creative workarounds, of learning from their mistakes and successes both.<br>I wasn't enjoying the work anymore, and I didn't feel as if I was learning anything valuable beyond the reality that Kubernetes isn't meant for two NUCs, a smattering of Pis lacking RTCs, and a NAS that can't do PXE boots. My takeaway from two years digging into the guts of this thing and trying to make it work at home in a way that let me explore, and experiment, and learn, was that I didn't actually need it at all to do what I wanted.<br>I learned that Kubernetes is an amazing tool. It's also one I didn't need at home.<br>Going Fishing with Nuclear Weapons<br>Kubernetes is the prime example of an "enterprise-first" technology. If you're running anything more substantial than Minikube or kind, you're looking at an investment of three physical boxes just for HA Control Planes, plus another set of boxes for worker nodes.<br>But you can just do a single Control Plane and a Worker Node! Heck, you can taint the workload to run on Control Plane nodes as well!<br>Yeah, you could. Except then you've thrown out everything that makes Kubernetes, well, Kubernetes. You've created an overly complicated Docker node with manual failover capabilities. You've tripled the amount of YAML you have to write to execute the same thing as a Docker Compose file, and you have the "benefit" of now also wrangling things - namely High Availability - you can't have with a single control plane node. Plus, now you have etcd to backup and maintain (with its own controls that aren't tied to kubectl for some - likely asinine - reason).<br>In the real world of Kubernetes, however, you need three Control Plane nodes for HA. You also need some sort of HA etcd deployment or an external database with its own backup and maintenance schedule. Then you have to define the network overlay to use, which is its own ball of wax, and then you have to define the storage layers for the nodes to use so that pods can spin up and access storage regardless of the worker node they run on. Only after that's done can you dig into the Gateway API, which replaced Ingress controllers for L4 and L7 routing, and setup the necessary infrastructure to even begin serving workloads and services.<br>It's a lot of fucking work! It has to be a lot of fucking work, because Kubernetes is built for gargantuan deployments. We're talking hundreds, thousands, millions of containers covering global datacenter infrastructure whose node counts are measured in the double or triple digits. It's the king of abstraction layers, letting users...