How we built an autonomous SRE agent for Kubernetes

Learn

DocsCompany

PricingTry LangSmith

Get a demo

Try LangSmith

Get a demo

How we build an autonomous SRE Agent for Kubernetes Deployments

Eric Johanson

August 5, 2026

min

Go back to blog

Create agents

Share

How we build an autonomous SRE Agent for Kubernetes Deployments<br>I'm a Deployed Engineer at LangChain, and a good part of my job lives on top of Kubernetes. I maintain our internal self-hosted cluster (also where new self-hosted features land first, so our team can test them as they arrive), and I help customers stand up and upgrade their own self-hosted environments. Doing this requires a mental model of a live cluster deep enough that when ours goes down I can read the architecture, find the fault, and fix it. Doing that continuously, on top of a full-time job, is exhausting, and it isn't unique to me. Anyone who works on infrastructure knows these pains.<br>So we built an autonomous SRE Agent to reduce time to triage and time to remediation. The goal was to improve infrastructure reliability and reduce cognitive load on the team. We wanted it to triage Kubernetes health, propose fixes, and pull in a human only when a cluster or infrastructure changes need to happen. This post covers why, how we built it, why LangSmith makes it trustworthy, and the benefits we've seen.<br>Part 1: Why We Built It<br>Kubernetes emits a firehose of signals (pod phases, restart counts, HPA (Horizontal Pod Autoscaling) states, node conditions, warning events, deployment readiness, across dozens of namespaces) and almost no synthesis. On-call engineers use these signals to answer three questions: Is anything broken now (a crash loop, an OOM kill, zero ready endpoints); Is anything about to break (an HPA pinned at max, a single-replica service, :latest image tags); What do we do about it? Answering these questions well takes judgment, so it falls to infrastructure engineers or subject matter experts, but 90% of this work is mechanical triage that mostly comes back clean. This is the toil that burns people out and trains them to inadvertently skim past important alerts.<br>Part 2: What we built<br>Proactive monitoring. A scheduler checks health every N minutes without waking the full agent. It collects raw cluster state through the Kubernetes Python client (zero LLM tokens), then makes one Claude Haiku call with forced tool-use to produce a structured health report that lands in Slack, sorted by severity.<br>On-demand investigation. When an issue needs diagnosis, the orchestrator fans out to specialized subagents in parallel: pod-inspector, scaling-analyzer, performance-analyzer, log-analyzer, security-auditor, reliability-auditor, and more. Each reads the cluster independently before it synthesizes one prioritized report.<br>The safety model<br>The agent can read the entire cluster but change nothing on its own. Every write (scaling a deployment, restarting a rollout, patching an HPA) lives inside a single change-executor subagent, and each write tool is gated by a human-in-the-loop (HITL) interrupt. The agent proposes a remediation, a person approves, rejects, or edits, right from a Slack message. Read is autonomous and writing is always gated through HITL. It’s enforced structurally and mirrored by in-cluster RBAC (cluster-wide read, tightly scoped write).

SRE Agent ArchitecturePart 3: How We Built It (and Why)<br>Each choice below was a fork where the obvious path and the right path diverged.<br>Deep Agents, not a raw loop. We build on Deep Agents (create_deep_agent()) over LangGraph, which gives us a planning loop (write_todos), first-class subagents, and built-in HITL interrupts out of the box, all things we'd otherwise need to hand-build. Our rule of thumb is to reach for the highest-level framework that doesn't fight you on the workflow you need to build.<br>Narrow subagents, not one omniscient prompt. Specialist agents each own a slice of the cluster, which buys parallelism, tighter context (fewer hallucinations), and the option to run well-scoped tasks on a cheaper model.<br>Sonnet where it thinks and Haiku for scale. The synthesizing orchestrator runs on Claude Sonnet; the read-only subagents and the scheduled check run on Claude Haiku. Pay for intelligence only where it's needed.<br>The scheduler bypasses the agent. It used to run the full orchestrator (~20 model calls) just to confirm "all healthy." Now it collects state in plain Python and makes one Haiku call. This approach achieves a 95 to 99% cost cut per check with no loss in catching issues. We reserve full power for on-demand investigations.<br>Give the human an approval they can actually read. The changes that are proposed need to be clear in all the components they affect. Scaling a deployment to 3 is legible in a glance; a helm upgrade is one click that rewrites dozens of resources you can't see from the approval prompt. So we kept write tools narrow and legible and deliberately withheld coarse, high-blast-radius ones, even...