Software 2.0: a Security-Critical Project Where I Don't Read the Code – Aaronontheweb

August 10, 2026 •

12 minutes to read

Software 2.0: a Security-Critical Project Where I Don't Read the Code

Software Development

LLM

AI

Software 2.0

ShellSyntaxTree

NetClaw

When can you stop reviewing LLM-authored code? Three properties make it possible - and most projects don't have them.

No Reviews, No Manual Coding<br>LLMs Perform Best on Narrow, Detailed Missions<br>Verification, Not Testing<br>Quick Feedback Loops<br>Should You Be Reviewing LLM-Authored Code?

There’s been lots of rancor online about whether you should read and review LLM-authored code. I wanted to share a project we’ve been using in production for months where I rarely review any of it: ShellSyntaxTree.

ShellSyntaxTree is a parser that attempts to create AST representations of both bash and PowerShell commands using a shared tree representation and parser abstractions, albeit with slightly different grammars to accommodate the quirks between the scripting languages.

SST is an essential tool for powering one of my other projects: Netclaw - a “claw”-style autonomous AI assistant. We use SST to power Netclaw’s approval system for requesting humans to authorize commands: netclaw.dev/architecture/security-model.

SST isn’t a security tool, but it sits inside a security decision. Netclaw auto-approves commands by matching them against pre-approved patterns - git pull, gh pr view - and SST is the layer that extracts those patterns from the raw command string. Get the parse wrong and Netclaw can auto-approve commands the user hasn’t blessed or prompt for commands the user already has.

The screenshot above is a live example of that: SST parsing echo NO_LOG_FILE for the approval prompt, separating the echo verb from the NO_LOG_FILE input. The challenge we’re always working on is determining parsing rules for safely extracting arbitrary command patterns that can be automatically approved later or in some cases, automatically denied / always prompted / etc.

How can we consistently recognize and distinguish the echo command verb from the NO_LOG_FILE input? That’s the kind of improvement we’re always trying to make to SST.

No Reviews, No Manual Coding

Codex has been running headlessly for about 2.5 days on SST with a goal to improve our existing bash and PowerShell syntax representation to include things like loops, inline variables, heredocs, pipe operators, etc.

I spend a lot of time working on the specifications, mining examples and counter-examples out of Netclaw instance logs, and defining some test infrastructure running large corpuses of commands through SST with their expected parse results.

But I feel free to leave the agents implementing SST unattended and trust them to make good decisions for three reasons:

SST has really good constraints that make its mission highly focused, even though the range of scenarios it has to cover is unbounded;

On the spectrum of how “verifiable” a piece of software is, SST sits on the very high end - and it’s partly due to SST’s focus that makes this so; and

SST has an extremely powerful and fast feedback loop because I also own its primary consumer, Netclaw. Any faults undetected via the original verification system get added back into the suite, making it somewhat antifragile.

LLMs Perform Best on Narrow, Detailed Missions

An essential ingredient for having a project where the LLM can competently implement it with little oversight is narrow focus.

I designed ShellSyntaxTree with an LLM-achievable mission intentionally, by making the following design decisions early:

It makes zero promises about safety or about what’s in the environment. “What is this command and how is it structured?” is all that SST is designed to do. Netclaw handles the policy layer by gradually accumulating approval data from the human-in-the-loop over time.

When a command reaches a threshold where it’s too complex to analyze (i.e. relies on external variables, other forms of dynamic execution) it barfs and warns the caller that this can’t be analyzed

Unified consumer-facing model for both Bash and PowerShell parsing + abstract syntax tree representation but allow different grammars for bash and PowerShell.

This acts both as a conformance mechanism for the LLM to bridge SST’s parsing grammars back into a unified representation but also as a pressure-relief valve in the event that PowerShell / bash / anything else we want to analyze in the future have idiosyncratic differences that can’t be easily modeled using a unified model.

This is important: don’t back the LLM into a corner.

Avoiding the question of “is this command safe to run?” and “how is this command connected to every facet of the runtime environment?” keeps SST’s mission very achievable via static analysis, which is the goal.

We’re not trying to understand what the command does, we’re trying to isolate and determine what are the key commands and how do they...