Validating LLM Code Edits When You Can't Run the Code — Velyr Blog

AI Agents & PR Automation<br>Validating LLM Code Edits When You Can't Run the Code<br>By Velyr Team · Published July 17, 2026<br>TL;DRAn agent that writes code into repositories it cannot execute gets none of the usual safety nets: no build, no tests, no typecheck. What remains is a five-layer stack: constrain which files are touchable, anchor every edit to real file bytes instead of the model's quotation of them, run static checks that only reject provable breakage, validate unparseable formats comparatively (did the edit make the file worse), and put an adversarial second model between the proposal and the write. Every layer fails into a distinct, honest status instead of a retry loop.

There is a class of system that has to write code into repositories it cannot execute. Ours is an agent that opens pull requests against other people's production websites. It runs in a serverless isolate with a wall-clock budget, reads the repo over the GitHub API, and at the moment it writes a change it has no checkout, no node_modules, no build, no test run, and no dev server. Whatever validation happens has to happen before the PR exists, using nothing but the bytes of the files, static analysis that runs in milliseconds, and more model calls.

The customer's own CI may run on the PR afterwards. That is not a safety net, it is someone else's inbox. A PR that fails CI on a syntax error costs the exact trust the whole system depends on. The goal is a change that is provably not-broken before any human sees it.

After a year of production runs, the validation stack has settled into five layers. Each one is cheap. Each one rejects only what it can prove is broken. And they are ordered so that a failure at any point leaves nothing behind in the target repo.

What "can't run the code" actually rules out

It is worth being precise, because each unavailable tool kills a familiar guard:

No npm install. Arbitrary package managers, private registries, native dependencies, and minutes of wall clock in an environment budgeted in seconds. This rules out running anything.

No test suite. Even if one exists (you don't know), you can't run it.

No typecheck. tsc needs the full dependency graph, which needs the install.

No rendering. You cannot boot the app and look at it. (You can screenshot the live production site, which turns out to matter later.)

So the usual ladder (types, tests, CI, staging) is gone at write time. What's left is the file itself, before and after the edit.

Layer 1: constrain the blast radius before validating anything

The first guard has nothing to do with code quality. A model choosing which file to edit is a model that can choose .github/workflows/deploy.yml, and a workflow edit is a secret-exfiltration primitive: CI runs with credentials the agent was never given. So before any content check, the chosen path runs against a denylist:

const FORBIDDEN_EDIT_PATHS: RegExp[] = [<br>/^\.github\//i, // CI can exfiltrate secrets<br>/(^|\/)\.env(\.|$)/i, // .env, .env.local, .env.production<br>/(^|\/)package(-lock)?\.json$/i, // dependency manifests<br>/(^|\/)vercel\.json$/i, // deploy config<br>/\.pem$|\.key$|\.p12$|\.pfx$/i, // private keys<br>/(^|\/)supabase\/functions\//i, // the agent itself (no self-modification)<br>// ...framework configs, lockfiles, Dockerfiles, IaC, migrations

Then the inverse gate: an allowlist of file extensions the pipeline can actually verify. The syntax validator returns ok: true for types it cannot parse (.vue, .svelte), because a validator that blocks everything it doesn't understand blocks too much. That pass-through would be a hole, so the caller closes it: any extension outside the verifiable set is refused outright, with an error that says exactly why ("refusing to open an unverified PR"). A permissive validator plus a strict extension gate is fail-closed. A permissive validator alone is not.

Layer 2: anchor edits to real bytes, not the model's memory of them

The model emits edits as find/replace pairs. The dominant failure mode is not malice or hallucinated APIs, it is transcription drift : the find string is a near-copy of the real code with slightly different whitespace, a straightened quote, or reordered attributes. Applied naively, that either fails or, worse, matches the wrong thing.

The guard works in two steps. First, try the exact substring. If that fails, build a whitespace-normalized version of the file alongside an index map, so every character of the normalized text points back at its position in the original:

// Collapse whitespace runs to a single space AND record, per<br>// normalized char, the original index it came from, so a normalized<br>// match maps back to exact original bytes.<br>function buildNormalized(content: string): { norm: string; map: number[] }

On a unique match, the replacement is spliced into the actual bytes at that anchor , never into the model's quotation of them. The model's find is only ever used to...