Manuel Schipper

Measuring engineers by their AI exhaust (token burn and lines of code) is a bad idea. Code generated and tokens spent are not the same thing as impact. A staff engineer can spend most of their time on non-coding activities (polishing a vision document, reviewing complex pull requests, influencing other teams, mentoring others) and still have more impact than an engineer who authored twice as many PRs.

In High Output Management, Andy Grove described this as leverage. The output of a leader extends beyond the work they produce directly into what they enable across the organization. Grove calls training one of the highest-leverage activities a leader can perform.

The more senior an engineer is, they are evaluated more heavily based on peer feedback and the impact of the work they led or contributed to, and less so based on the number of impactful PRs. Historically, this created an inverse relationship between seniority and direct technical output. As engineers became more senior, they were expected to find leverage other than coding.

But AI has changed our practice, and I think this is no longer the case.

Nobody knows the right way to code with agents

For a long time, the practice of software engineering was stable enough that a senior engineer could move away from primarily coding without completely losing touch with how software was made. Sure, languages, infra, and frameworks changed every now and then, but the basic principles remained true. The primary interface between humans and software was an IDE and a terminal. Engineers designed systems, wrote code, reviewed pull requests, tested, shipped, and built CI/CD.

Yet now, we have entered a period of actively re-litigating the interface between human intent and software. There is no settled way to build with coding agents, and technical leaders need firsthand experience with the new means of production.

Here’s a non-exhaustive list of open questions I am personally wrestling with:

On code review:

Should engineers read all of the code?

Should we mostly review the tests and resulting behavior?

Is it okay to rely on AI to explain the code changes?

Is one AI code review sufficient, or do we need multiple gates?

On context management:

What goes into an AGENTS.md?

What’s an effective context window?

Should agents be allowed to invoke skills on their own?

On codebases:

Should an LLM maintain a wiki of the codebase?

Should Markdown plans be stored in the repo?

Does codebase curation and organization still matter?

On autonomy:

Should we work with agents as copilots, or should they work unsupervised?

Should we adopt spec-driven development?

Should agents have their own identity or act on behalf of the user?

Should agents focus solely on coding or be connected to external services like GitHub and Jira?

Meanwhile, new tools pop up every day. Here are some tool categories (again, non-exhaustive) that I’ve been experimenting with:

UIs:

Terminals

Terminal multiplexers

Agent GUIs

Diff viewers

Artifact viewers

Agent delegation:

Harnesses

Plugins

Skill packs

Orchestrators

Task managers

Security:

Sandboxes

Pre-tool-use hooks

Credential vaults

Credential proxies

I don’t think we are close to agreeing on a standard way to build production software. These questions cannot be answered by reasoning about engineering principles. They are empirical questions.

You have to run agents against real codebases and find ways to push them into failing. You have to stand up your preferred stack, build your own custom tooling, and be ready to throw it all away and start again three months later. Otherwise, you cannot develop an intelligent position. Secondhand reports, demos, or articles will not suffice.

Seniority should produce more exhaust

An engineer uses coding agents to complete a task. A technical leader now has a broader mandate: figuring out effective ways to leverage coding agents collectively.

Where do agents work well? Where do they fail? What context do they need? What should humans review? Which controls should be deterministic? What should be standardized across the team, and what should be left to individual engineers?

To influence these decisions, you need a lot of firsthand experience. Staff and principal engineers should be running experiments, trying new tools, and pushing models the hardest against real problems. That should inevitably leave a sizable trace of AI exhaust: burned tokens, lines of code, failed prototypes, and abandoned branches.

Here are some conclusions I’ve reached through experimentation:

On context windows: Even the most capable models, such as Fable and Sol, still make a ton of mistakes and ignore instructions. This is especially apparent at longer context windows. Thus, I have concluded that agents have an effective context window. I am still not sure what that is, and I suspect it is not a hard number and instead depends a lot on the task. Nonetheless, I am now capping my context...