Sentinel: an open-source QA agent that reads your code before it clicks anything — simbastack

Sentinel reads the codebase, works out the real business flows, and tests them end to end across the frontend and the backend. It's open source under MIT.

The gap between clicking and understanding

Put a typical AI agent on your app and it opens a page, clicks a few buttons, notices a misaligned element or a console error, and calls the run done. That's useful the way a smoke test is useful. The agent has no model of what your product actually does.

A QA engineer worth hiring doesn't click around. They learn the product first, then reason about it: this is a hotel system, so I need to test a single booking, a group booking, a cancellation that frees the room back up, check-in, check-out, and the night audit, and I need to confirm each one actually persisted on the server instead of trusting that the UI looked happy.

What happened when we gave it a real app and no instructions

We pointed Sentinel at a working full-stack hotel PMS (a property management system): a Next.js frontend, a separate API service, a Postgres database. The PMS is KaribuKit, our own product, which is why we could hand an agent admin credentials for a disposable test tenant. That's also all it got — the repo and those credentials. No test plan, no list of flows.

It read the code, concluded the product was a boutique and safari hotel PMS, and derived nine critical business flows on its own: the full reservation lifecycle, group bookings, the lead-to-proposal pipeline, rate management, guest self-service, mid-stay room changes, the night audit, payment through invoice to refund, and the AI copilot. Cancellations are in there too, though this run folded them in as edge cases and backend checks inside the other flows rather than deriving a standalone flow. That's close to the list a human QA lead would write on day one, and nobody handed it to the agent.

The run itself: plan derived from the repo, top two flows deep-tested twice each, then the vision pass over every screen it visited.

Then it ran the top two of those flows, twice each, and the trace read like watching a person work:

It called GET /api/availability, got a 400, worked out the params it was missing, and retried with adults=2&children=0 to get a 200.

It created a real reservation with POST /api/reservations (201), then fetched it back to confirm it had persisted with the right room and rate.

It walked the status lifecycle, found the check-in endpoint by trial (/checkin gave 404, /check-in gave 400, then a valid call returned 200), and checked the folio.

The folio it was verifying: three room-charge nights at $178, balance due $534. The red toast in the corner is a bug being caught live: "No rooms available" on a reservation that already held its room.

The bugs it surfaced are ones you can't find from the UI alone:

Confirming a reservation came back NO_AVAILABILITY, even though that same reservation already held the room. A backend state-machine bug the UI turned into a misleading "no rooms available" toast on one attempt, and into no feedback at all on another.

The calendar showed a room as available after a booking already existed for it. The API and the UI disagreed, and only checking both layers caught it.

Check-in returned 200, but the guest's registrationStatus stayed NONE on the server: a state transition that only half-completed.

The calendar mid-run, filled with reservations the agent created itself, tentative and confirmed side by side.

How it works

The QA agent's deepest engine, flow, is a pipeline:

Read the repo. A deterministic recon pass (grep and find, no model calls) extracts the structure: frontend routes, API route modules, services, database entities. The model reasons over that digest instead of crawling a monorepo blind, which is slow and misses things.

Derive the flows. Mimo, the Xiaomi model we run for decisions through the pi agent harness, turns the digest into a prioritized list of end-to-end business flows, each with UI steps, backend assertions, and edge cases. The plan is cached per commit, so it only re-derives when the code changes.

Run each flow as an agent loop: the model decides the next action, Playwright drives the browser, and a first-class api_request tool checks server state at each step by running fetch inside the page, replaying the Authorization header the frontend itself sent. The model only ever gets browser and API tools, never your shell or filesystem, and a hard budget bounds each attempt. Past 90 tool calls the action tools refuse to act, and the only move left is finish.

Run it more than once. Autonomous agents are non-deterministic. We measured it on one flow, where one attempt found zero bugs and another found five. So each flow runs twice by default (FLOW_ATTEMPTS, a knob) and the findings are unioned: a bug found by any attempt makes the report, and each flow keeps its worst verdict across attempts.

Grade the...