A Data-Driven Explanation: Why Do AI Agents Still Fail
Jeremy Tian
SubscribeSign in
A Data-Driven Explanation: Why Do AI Agents Still Fail
Jeremy Tian<br>Aug 04, 2026
Share
It’s been about two years since AI agents went mainstream. According to Capgemini’s 2025 report, AI Agents were estimated to generate roughly $450 billion in economic value across 14 surveyed countries by 2028. That’s less than a year and a half from now. So, how are AI agents doing now?<br>Coding agents: remarkably well. Claude Code, Codex, and others have crossed the threshold from autocomplete to shipping real work end-to-end. Anecdotally, I know many people who don’t even open their IDEs anymore. But it turns out, coding agents are the exception.<br>According to Stanford’s 2026 AI Index, organizational AI adoption continues to rise, but AI agent deployment remains in the single digits across almost all business functions. For AI agents specifically, researchers from Stanford, Berkeley, IBM, etc. found that 68% of deployed agents execute no more than 10 steps before human intervention.<br>It’s tempting to mentally extrapolate improvements across AI: “if AI can handle something as hard as disproving the Jacobian conjecture (an 87-year-old math conjecture), surely it can handle my back-office workflow.” This line of reasoning is in fact so common that there is a term for it: the “Jagged Frontier” of AI, which describes the imbalance of AI capabilities.
An interesting example to illustrate this is how Gemini Deep Think performed well enough to earn a gold medal at IMO, but can only read an analog clock correctly 50.1% of the time.<br>None of that explains why enterprise agents underdeliver, though. The short answer is that making AI agents that consistently perform well is very hard.<br>In general, four things explain most of the gap: variance, benchmark quality (datasets & evaluations), agent-specific errors, and alignment. What follows is a data-driven deep dive into each and why there’s such a massive gap between the incredible enterprise AI Agent value we were promised and the little value that has been provided.
On Variance & Reliability<br>Your Agent’s Performance Ceiling and Floor
Solving reliability in a non-deterministic system is inherently a really hard problem. To illustrate this, let’s model an AI agent as a system that performs some number of steps autonomously to complete a task. Let’s assume the probability that any step succeeds is 90%.<br>When your agent attempts a workflow that takes 10 steps, only 34% of your agent runs will succeed on all 10 steps (0.9¹⁰). When that process doubles to a 20-step workflow, the end-to-end pass rate decreases to 12%. A 30-step workflow has a 4.2% success rate.
End-to-end reliability decays as the number of steps grows.
Because every step is now non-deterministic, the overall reliability of the system decreases exponentially with the number of steps in your workflow. This is a big problem that gets exacerbated by complexity.<br>Even if we increase the probability of a single step succeeding to 99%, after 30 steps, you can only expect 73.97% of your end-to-end runs to complete successfully, which is a massive downgrade from the 99% individual step success rate.<br>Of course, real agents aren’t this simple. The calculations above assume that each step is independent of each other. Subsequent steps are not independent. In reality, errors compound (a bad step corrupts context for every step after it). But on the flip side, agents can also recover from their mistakes, you can build guardrails to guarantee specific output formats, etc. This example is just a first-order approximation, modeling an AI agent extremely simply. Unfortunately, what is not just a first-order approximation is the idea this example illustrates: more steps means less reliability, fast.<br>The problem we just described was “intrarun variance”; in other words, the variance within a run.<br>But your agent doesn’t just run once. It will need to solve the same or similar tasks many, many times.<br>Intrarun variance’s counterpart is interrun variance—the variance across multiple runs. Sierra knew this was a problem, which is why when they built τ-bench (one of the industry standard benchmarks for customer support), they made pass^k (pass-to-the-k) a first-party measurement. Pass^k, simply put, is just a measurement for interrun variance: run each dataset task k times; a task only counts as passed if it succeeds all k times. In other words, pass^k asks what your agent can do every single time, not just once.<br>To solidify the issue of interrun variance in a more concrete example, let’s take a look at τ³ bench (an improved τ-bench). For τ-knowledge (banking domain), the highest performing model was gpt-5.2 (high reasoning) with a pass^1 of 25.52% (pass^1 here means they ran each task in the domain once, and 25.52% of them passed).<br>At pass^4, it dropped by about half to 13.40% (pass^4 here means they ran each task in the dataset 4 times and only 13.40% of...