Train the Agent, then Replace It! - by Allie
All These Things Substack
SubscribeSign in
Train the Agent, then Replace It!<br>Or, "Stop paying engineer rates for a secretary!"<br>Jul 03, 2026
Share
There’s a seductive lie that arrives the moment you deploy your first useful AI agent. It goes: I don’t have to worry about hygiene and chaos anymore. AI will just deal with it.<br>Messy inbox? The agent will sort it. Fragile process? The agent will absorb the chaos. Undefined rules? The agent will figure it out, every hour, forever. FOREVER.<br>And the thing is, the agent will. That’s what makes the lie so easy to believe. The agent really does deal with your growing chaos. It just does so at a price, and it keeps charging you that price on every single run, whether the work is worth the price or not.<br>This post is about the step that comes after “my agent works!!” It’s a step nobody talks about, because everyone stops at the triumphant blog about an agent doing the thing. But if you’re like me, running agents on a recurring schedule, that first successful run is where the cost problem starts, not where our heroic story ends.<br>All These Things Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Subscribe
Now, for the step nobody talks about
Most agentic-workflow content follows the same story arc. The hero identifies a task, deploys an agent, relishes the way it handles ambiguity…posts about it! (Guilty).<br>Done. That’s the end of the story. The hero retires.<br>But not so fast! There’s a second phase: the phase that separates a fun prototype from an operational system: once the agent has learned the rules and codified them, you ask the agent to help you rip those rules back out, and hardcode the predictable parts into infrastructure that runs for free.<br>The title of this post is the thesis. You train the agent. Then you replace it . Or, more precisely, you replace the predictable and codified 75% of it, and only keep the agent focused on the 25% that actually needs LLM-ish reasoning.<br>If you’ve ever used a compiler, you already understand the mechanism. When you write code in a flexible language, the computer can’t run it directly; something has to translate it into fast, rigid machine instructions. A good compiler doesn’t just translate blindly. It optimizes: if you write 2 + 3, it notices the answer is always 5 and just hardcodes 5, rather than making the machine add two numbers every time the program runs. It pulls predictable stuff out of the expensive live path and resolves it once, ahead of time.<br>That’s the metaphor here. Imagine creating an “Inbox Triage Agent” for the sake of this post. Your agent, on every run, is “interpreting” some rules that aren’t really judgment calls.<br>If the sender is no-reply@tableau.com, label it “@Reference”
That’s not reasoning. That’s a lookup table or filter logic, but you’re paying LLM prices to run it. Step 2, the optimization step, is spotting those and compiling them down into free, deterministic infrastructure.<br>So, Step 1 is train the agent to discover the rules. Then, replace the parts that were never worth an LLM in the first place.<br>Let me show you what that looked like in practice.<br>The case study: my beloved inbox
I use Glean (an AI-powered enterprise search and workplace assistant) to build and deploy custom agents that run workflows for me. On my personal AI maturity ladder, this sits squarely in the productivity tier: core efficiency tasks and automations, not role-based deep thinking. (Glean doesn’t have a project-based system yet, so I don’t reach for it when I need sustained, stateful reasoning. Gotta use the right tool for the right tier.)<br>The problem I pointed it at was the oldest one in knowledge work: my inbox. Hundreds of emails every day, a list of manual Gmail filters I’d built and left for dead over years that suffered from the 3 “F”s: fragile, fragmented, and failing. Most of the volume broke through them and landed in my inbox anyway. And I do not have the attention span to self-categorize hundreds of emails. I never will. That’s not a moral failing; it’s just a principle I live by, and any system that depends on me changing it is a system that will fail.<br>So I built an agent, as one does in 2026. Here’s the framework that emerged from my journey: a three-phase process I’m officially calling agentic refactoring . Not to be confused with agents who refactor code, by the way.<br>Phase 1 — Prototyping & Rule Discovery (le sandbox)
The instinct, if you’re obsessive (which I’m absolutely not), is to sit down and try to anticipate every kind of email you receive and write filters for all of them up front. Don’t. You’ll spend hours writing complex rules for a reality you don’t actually understand yet, and you’ll be wrong about the edge cases because you haven’t seen them.<br>Instead, I deployed a flexible, high-reasoning Glean agent and let it loose. In order to deploy the Glean agent, I’ll be...