Agentic Chaos - ninjapenguin

I recently wrote about how Agents are taking us back to our engineering roots. Whilst the fundamentals have never gone away (maybe we just got better at hiding them in our default tooling[1][2]), Agents now force us to re-consider just how we’re addressing some of these basics.

I think Agents are here to stay, and if they’re going to be a permanent fixture in production systems, making real-world decisions that have very real-world consequences, then we need to come up with real-world solutions to providing the degree of engineering rigour we’ve become used to.

And the Agentic world is a messy one: tools fail, models degrade, integrations return strange data, retries compound and the delegation of autonomy needs serious governance. Agents bring Chaos!

So if Agents are bringing the hard engineering problems back again, then resilience engineering/testing has to become part of the platform (and the standard Agent development process) - not a one-off exercise condemned never to be repeated again.

The bluff of the successful demo

I previously wrote about how the era of the AI demo is closing. Let me dig into one component of that further…

…the demo went great, it called the right tool, produced the right answer and streamed something that looked shiny: you breathe a sigh of relief!

But production? It will ruthlessly expose any and all shortcomings in that shiny demo.

What if the tool call timed out?

What if the tool response was malformed?

What if the model vendor was unavailable?

What if you seamlessly degraded to a fallback model but it somehow changed the agent’s behaviour?

What if that agent states it’s ‘done’ after that failed tool call, maybe worse, what if it simply carries on?

Running your agent, and seeing it work, tells you little about how it will break in production.[3]

I’ll again make the point that agents are a little bit different to the way we’ve often talked about failure/degradation within software to date.

Traditional software often fails around explicit guarantees (contracts): a parameter was not passed, the return format didn’t match the expected shape. Agents are exposed to these same failures but add their interpretation of those errors on top in how they deal with/respond to them. Do they continue after a failed tool call, retry it, infer missing context, or pause execution to request clarity - or do they simply report success?[4]

Introducing Chaos

Agent Chaos is the ability to intentionally inject controlled failure modes into an agent’s run so you can observe its specific behaviour in response.[5]

It allows you, per agent run, to inject controlled, bounded faults and then fully introspect the agent’s response/behaviour.

The point is not to break your agent, it’s to build evidence.

We want to be able to demonstrably answer questions like:

if a critical tool fails, what does the agent do?

does the agent tell the user the truth?

does the agent accurately present the outcome of the run?

does the agent stop/halt before actioning potentially unsafe side effects?

is revision B more resilient than revision A

Agent Chaos is one of the primary tools we’re using for resilience testing our agentic workflows.

Embracing Chaos

Chaos runs define controlled scenarios with repeatable injection. The crucial component here is that those faults become both deterministic and observable. When automated using evals, we can then define the acceptable reaction to each fault.[6]

A single chaos-enabled run is interesting. 50 chaos-enabled runs build a body of evidence and confidence in behaviour. Couple this with our ability to build, evaluate and deploy incremental behaviour through versioned agent revisions and the value becomes not just knowing that the agent can work but instead knowing exactly how its behaviour changes across those evaluated scenarios.

When an agent runs, and works, it provides confidence that is built atop a number of assumptions. Agent Chaos allows us to directly target and challenge those to generate a body of evidence for the agent’s behaviour when those assumptions break.

Of course the detail of that evidence is key. Chaos without observability is just noise. For a chaos-enabled run we record:

what was injected where

why that fault was injected

what the agent’s response to the fault was (i.e., did it request human-in-the-loop approval)

what the ‘result’ of the agent run was and what it output to the user

This gives us the ability to make evidence-backed statements on an agent’s run like:

“The run failed because the tool call was deliberately made to fail. The agent detected that failure, avoided any side effects, explained its limitation, and ended cleanly.”

Here’s an all-too-common example of bad output given an agent run:

“Run Complete”

Given some chaos-enabled tuning, here’s a much better example:

“I could not complete this task because the calendar lookup failed. I did not make any changes. You can retry the...