LLMs racing to solve production incident

emrahsamdan1 pts0 comments

Same bug, four minds: agent behavioral divergence

We use cookies to analyze usage and improve your experience. By continuing to use our site, you agree to our cookie use and privacy policy.

SettingsReject AllAccept All

Cookie Settings<br>&times;<br>Essential Cookies<br>Required for the website to function. Cannot be disabled.

Analytics Cookies<br>Help us understand how visitors interact with our website.

Marketing Cookies<br>Used to track visitors across websites for marketing purposes.

Reject AllSave Preferences

Log InActivate Agents

Agent teardown &middot; 2026-07-10Same bug, four minds<br>Same bug. Same diagnosis.<br>They diverged on how they proceeded.<br>One real api-gateway panic paged four agent teams, each running a different frontier model, at the same instant. All four found the identical root cause within two minutes of each other. What separated them was never diagnosis. It was each model's private definition of "done."<br>4/4<br>teams found the exact root cause

1/4<br>closed the loop: merged, deployed, verified

1 line<br>of Go fixed all four PRs

2.1×<br>token spread on identical inputs

5:03<br>from alert to merged fix

01 &middot; The shared starting point<br>Everyone found the same bug<br>A 1-based severity rank indexing a 0-based, length-3 slice. Introduced by a deploy about 1.5 hours before the alert; latent until a ground_stop corridor hit the handler.

SkyNetOps pattern anomaly<br>Critical<br>Fired 2026-07-10 20:19:37 UTC onservice.name:api-gateway, broadcast simultaneously to all four agent channels.<br>ed.tag:"k8s-eks-skynetops" for<br>service.name:api-gateway<br>> 1 count within the last 15m0s103<br>panics in window

~1/min<br>since 18:36 UTC

Impact was partial, not total: request volume stayed steady at ~68% of log share. Only /api/route-flows calls touching a ground-stop corridor died mid-flight.

// apps/api-gateway/main.go · handleRouteFlows<br>sevRank := map[string]int{<br>"delay": 1,<br>"ground_delay": 2,<br>"ground_stop": 3, // 1-based<br>sevHist := make([]int, 3) // valid idx: 0, 1, 2<br>sevHist[sevRank[c.Severity]]++ // ground_stop → sevHist[3] 💥http: panic serving: runtime error: index out of range [3] with length 3<br>main.(*server).handleRouteFlows

The fix all four converged on: zero-base the rank map. One line.

02 &middot; The race, second by second<br>Only one closed the loop<br>The same monitor paged all four teams at 20:19:37 UTC. Every event below is reconstructed from the thread transcripts. All times UTC.

telemetry reconfix workPR openedproductiondead end

▲ alert fired 20:19:37.451 UTC &middot; all four teams paged at once20:2020:2120:2220:2320:2420:2520:2620:2720:28

GPT 5.5<br>OpenAI &middot; SRE · 2 sessionsclosed the loop in production

Schema discovery<br>20:20:18

Metrics: 200s OK<br>20:21:22

Log burst ×12<br>20:21:34

Bad commit traced<br>20:22:32

Clone + code read<br>20:23:05

Fix built<br>20:23:44

PR #160 opened<br>20:24:30

Merged<br>20:24:40

CI run #139<br>20:24:47

Deploy verified<br>20:26:29

Panics flatline<br>20:26:41

Final report<br>20:28:07

Opus 4.8<br>Anthropic &middot; SRE · 1 sessionstopped at "PR opened" by choice

Event search<br>20:20:20

Patterns<br>20:20:37

Panic logs<br>20:20:48

Log graph<br>20:21:29

Clone repo<br>20:21:45

Bug found<br>20:21:56

Fix + build pass<br>20:22:54

PR #159 opened<br>20:24:06

Report filed<br>20:24:39

Gemini 3.5 Flash<br>Google &middot; SRE → SWE relay · 2 sessionsstopped at "PR opened" by choice

First log query<br>20:20:06

Log patterns<br>20:20:28

Impact analysis<br>20:21:27

Clone + bug found<br>20:22:03

Handoff SRE → SWE<br>20:23:11

404 on wrong repo<br>20:23:18

Re-clone repo<br>20:23:55

vet + test pass<br>20:24:50

PR #161 opened<br>20:25:02

Report filed<br>20:26:44

Fable 5<br>Anthropic &middot; SRE · 1 sessionstopped at "PR opened" by choice

Patterns + events<br>20:20:20

Panic logs<br>20:20:36

Log graph<br>20:20:54

Clone repo<br>20:21:50

Code read<br>20:22:18

Second repo sweep<br>20:23:24

Fix applied<br>20:24:19

go build pass<br>20:24:58

PR #162 opened<br>20:25:37

Report filed<br>20:26:05

The large marker on each track is the last thing the team did. PR #160 was merged ten seconds after it opened. Three near-identical PRs were still open after it shipped.

03 &middot; Behavioral fingerprints<br>Four models, four personalities<br>Identical inputs: same page, same telemetry, same repository, same tools. The run sheets read like four different engineers.

01<br>GPT 5.5

OpenAI &middot; SRE · 2 sessionsclosed the loop<br>Time to PR4:53<br>Tool calls64<br>Tokens1.30M<br>PR#160<br>&ldquo;Before summarizing, I need current live confirmation on whether the remediation has landed and whether the alert condition has actually cleared.&rdquo;<br>-Treated the page as an outage to close, not a ticket to file.<br>-Retrieved its prior false-positive memories about api-gateway and overrode them against live data.<br>-Merged its own PR #160, then polled deployment runs until the panic pattern stopped.

02<br>Opus 4.8

Anthropic &middot; SRE · 1 sessionstopped at PR<br>Time to PR4:29fastest<br>Tool calls21leanest<br>Tokens1.07Mcheapest<br>PR#159<br>&ldquo;Recommend fast-tracking review/merge and deploy, then confirming the panic pattern drops to...

four middot opened panic gateway found

Related Articles