The operational risk of AI coding agents in B2B SaaS – Jim Grey on software management
Skip to content
The operational risk of AI coding agents in B2B SaaS
In B2B SaaS right now, every operator I talk to is under the same pressure. Boards want companies to go faster with AI. Vendors are promising huge productivity gains.
Some of the gains are real. I go to a monthly meetup for people using AI coding agents in their work, and I’m seeing in their live demos how they’re shipping way more pull requests now.
Earlier this year I ran a pilot using AI agents for unit test generation. We chose unit tests in part because they were sorely needed, but more because they were out of the main production stream. It was a good case to learn on. It produced a meaningful reduction in cycle time for the work in the pilot.
I’m not an individual developer using AI to accelerate just my work. I’m an engineering leader figuring out how to do it safely at scale. Left on its own, an agent can produce a lot of plausible code that should never get near Production.
The failures that make the news tend to be dramatic: a consumer platform ships AI-generated, AI-reviewed code with no human in the loop, and the result is a simple vulnerability that lets an attacker take over user accounts. The story is all over the news for a few days. Some users quit in protest, and then the incident fades from memory.
That is not how failure works in vertical B2B SaaS.
A coding agent makes a plausible change — not obviously insane, not an “AI hallucination.” Just the kind of change a tired reviewer might skim past because the tests were green and the PR looked reasonable. But plausible and reasonable are not the same as correct. A plausible-but-wrong change can break a customer’s workflow, or introduce a cluster of small paper-cut defects, or create a security hole.
This happens without AI in the mix, of course. But AI lets you do it a lot more often, a lot faster.
I saw just this happen recently. One of my clients had a vendor in their delivery chain ship a terrible release full of regressions and a new performance bottleneck. I don’t know whether AI had anything to do with it, but that’s not the point. My client’s response was essentially: you get to do that only once. They are well connected in the market, and their opinion will reach other buyers before those buyers ever talk to the vendor’s sales team.
Consumer AI failures fade because consumer users do not know the whole rest of the user base. The B2B market is different. The buyers know each other. They sit on the same standards committees, show up at the same regional conferences, and compare notes when a vendor makes their lives harder. A bad failure in a small B2B vertical becomes a reputation that precedes you.
So don’t treat agentic coding as a productivity feature. Treat it like operational risk.
We do plenty of things to limit the risk of running Production software. Good monitoring and alerting. Automated paging of engineers when something is wrong. Incident response discipline. Plenty of unit and integration tests.
Those practices exist because Production systems fail. Good operators try to catch failure before customers do. When they can’t, they limit the damage, fix the problem, and learn from what happened afterward.
AI coding agents belong in that same operating model. What are you doing to keep your agents inside known boundaries, and how will you know when they start drifting outside them?
When I say “agent,” I do not mean a magic developer replacement. I mean a custom tool in your delivery process, built to do one constrained job in the context of your codebase, architecture, business rules, and failure modes.
The pattern I’ve used this year is simple. You bathe a new agent in human judgment until you find as many of its failure modes as you can. Then you build the tests, constraints, and review gates that keep it operating inside known boundaries. Only then do you let it do work at real scale.
The teams I’ve been working with are creating narrowly scoped agents: one to write unit tests, another to fix defects within a limited feature area. That may sound fussy, but I think it’s the right model.
The danger is not only that the agent makes a bad change. The danger is that it makes a bunch of plausible changes, the pull requests are long, everyone is under schedule pressure, and the humans start skimming through reviews.
That is a slower start than turning the thing loose and celebrating the pull request count. Each agent after that comes faster because of the scaffolding you’ve built and the things you’ve learned. The tests, review habits, constraints, and observability get reused and improved instead of invented from scratch every time.
Eventually the job is not babysitting agents. The job is running a system.
After trust is earned, nobody should have to hand-inspect every single output forever. That doesn’t scale either. But you still have to...