We started tracking which AI model wrote every line, and you should too

JER on X: "https://t.co/KbwvXDAO6v" / X Post

Log inSign up

Post

JER

@lifeofjer

We started tracking which AI model wrote every line, and you should too We've been building the same app for over a year with AI agents @atlas_v_erified. Not one model. A bunch of them. Testing models, QA models, building models, design models. Frontend, backend, schema, architecture — all of it. Cursor keeps shipping new ones (theirs and other labs) and we keep trying them. And for most of that year period we had no real way to answer a pretty basic question: which model actually wrote this code or which model created this plan? We had anecdotes. Take for example Opus 4.7. Promising, but we learned after heavy use that it would say the work was done and only complete part of it. Partial implementations. We figured that out fast and moved to a different model. But that's still just a feeling. Anecdote, not hard data. So I built model tracking into the repo. Every time an agent touches a file, it has to leave a byline. Machine-readable. Same session, same sprint. Here is what we record: the coding model that made the edit the planner model, if planning was a different model the exact hunks that changed the plan those changes came from and if it's a bug fix, which earlier change it's fixing (when we actually know) On a normal TypeScript file it looks like this at the top: Here are a few rules that matter when implemented: Newest byline only in the file. Next edit replaces it. Git keeps the history. Hunks don't include the provenance block itself, or the line numbers get useless the second the header moves. If Cursor doesn't expose the real model id / max mode / context window, we write not_exposed. Agents are not allowed to invent it. Planner and coder are separate fields on purpose. A lot of our work is planned by one model and implemented by another. JSON, lockfiles, binaries can't take comments — those go as one line into .cursor/agent-provenance.jsonl. That sidecar is not a loophole for "this .ts file was big." Docs weren't enough Now, here is where it gets interesting. First attempt was the schema and a hard rule in our AGENTS.md (yes, I know that prompts are NOT rules or security or requirements, but it's a fast test). The agents ignored it when it got inconvenient. Left hunks: pending. Skipped headers at the end of large plans and sprints. So I made it into something they can't skip, and it's working great. How we force it Four layers: Always-on Cursor rule — short reminder every turn Cursor hooks — track every edit, nag after writes, and on stop force one cleanup turn if anything is missing Pre-commit — checker blocks the commit CI — same checker on the PR diff The stop hook is the one that actually changed behavior. Agent thinks it's done → we re-check every file from that session → if provenance is missing, it gets forced back in: PROVENANCE GATE: You edited files without valid atlas-agent-provenance/v1 attribution. Do not start new feature work. Fix provenance on every file below, then stop. One loop max. Pre-commit and CI catch the rest. Rules (or .md) are optional to a model. Hooks and CI are not. Why bother Once this is all the way through our codebase, we can actually debug where bugs or architectural changes were planned and implement by model. which coding models create the most rework which planner → coder combos ship clean on the first pass when we fix a bug, can we walk it back to the change_id and the model that introduced it is the "said it was done, only did half" failure mode concentrated in specific models Publicly available I really wish I had this data back to the start of this project. It would be incredibly valuable to us. Imagine running a plan to review ALL code generated by a specific model (in loops), or tracing back every issue and running data on which models produce the least issues (even segmenting by back-end, front-end, db schema, etc). So, I'm going to drop the public repo here so anyone can utilize this, and I have submitted this to Cursor Marketplace for anyone to one-click add (if they approve, 🤞 @cursor_ai). Drop a comment below with what you are building using Cursor (@cursor_ai). Would love to see what is being built. Public Repo https://github.com/atlas-verified/agent-model-provenance

span:not(:empty)~span:not(:empty)]:before:content-['·'] [&>span:not(:empty)~span:not(:empty)]:before:px-1 [&>span:not(:empty)~span:not(:empty)]:before:shrink-0">6:32 PM · Aug 12, 202621Views

span:not(:empty)~span:not(:empty)]:before:content-['·'] [&>span:not(:empty)~span:not(:empty)]:before:px-1 [&>span:not(:empty)~span:not(:empty)]:before:shrink-0 min-w-0 overflow-hidden">JER

@lifeofjer

33m

Add to Cursor here:

GitHub - atlas-verified/agent-model-provenance: Cursor plugin: force model bylines (coder + planner...

From github.com

Log in or sign up for X See what’s happening and join the conversation Continue with phoneContinue with AppleContinue with Google or Log in with username or...

We started tracking which AI model wrote every line, and you should too

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

NHS apologises and admits Palantir have access to identifiable patient data

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills