Version control for everything - Ty Overby
Blog<br>Reading List
AI assisted agentic coding has reached escape velocity, but non-programming use cases haven’t seen<br>the same degree of adoption. I believe that the main reason for this is the lack of version control.<br>Imagine using claude-code outside of a git repository. Even for small things like refactors, using<br>AI would be very stressful and error prone:<br>It would be near-impossible to track the changes that the LLM made.
Auditing the<br>LLM generated code is useful in the moment to ensure that changes are reasonable before moving on<br>to another task, and in the future when you want to understand why some code was written<br>The LLM could put the codebase in a bad state and you’d have no way of reverting
This is true even if the LLM is incredibly smart and didn’t make any “mistakes” -<br>the human prompter forgetting to tell it about a design constraint could be bad<br>enough<br>There’s no split between the “development branch” and “prod”<br>You can’t parallelize development by having multiple LLMs work on different branches<br>Even when I pay Claude to work on a small script, I always create a new git repo just to make my<br>life easier. But outside of coding, it’s nearly impossible to find tooling that has the same<br>guardrails and affordances.<br>Case study: software development outer loop<br>Managing the software development process is hard. We use issue trackers and pull requests to manage<br>work, people write documentation and communicate over email, instant messaging, and in meetings.<br>Keeping all of the information in these channels synchronized and up to date is a full time job. In<br>this scenario, let’s say we’re concerned with the following systems:<br>github issues (read/write)<br>pull requests (read/write)<br>google calendar (read/write)<br>google docs (read/write)<br>gmail (read)<br>slack (read)<br>and you’re interested in using an LLM to find places where some information hasn’t made its way from<br>one service to another
e.g. update an issue with new information after an email<br>conversation<br>. This task is hard in isolation
though I think that<br>today’s LLMs could do it<br>but the biggest issue is that none of these services have<br>built-in mechanisms that would allow the LLM to propose an action to be reviewed by a human.<br>Option 1: a proxy layer<br>Without changing any of the underlying services, you could imagine building a proxy to add a “pull<br>requests” layer that would allow staging changes across multiple underlying services and allow<br>review before publishing the changes. An agent would act through this proxy, which would aggregate<br>the mutations until someone could review, approve, and publish them.<br>This is challenging for a few reasons:<br>Building one-off systems like this is time consuming, it’s tied to the specific workflow and<br>complexity grows as you need to integrate more services.<br>“Revert” would probably be out of reach. Underlying services might not provide functionality<br>necessary to implement “undo"
especially when reverting a change that has already<br>had other changes stacked on top of it<br>Detecting and resolving merge conflicts in the underlying services isn’t always possible.<br>There’s no atomicity. If someone clicks the “publish” button and one service rejects the change<br>for any reason, all the previous changes to other services are already out in the world.
This is especially troubling if the already-published changes can’t be<br>reverted<br>The layer on top of the underlying services prevents you from seeing what the whole state of the<br>world would look like if you were to merge the change.
Imagine having to do code<br>review, but could only look at the diff instead of being given the diff and the ability to see<br>the entire contents of the codebase before and after the diff is applied.<br>Option 2: put everything else in git<br>If you’re ok with leaving github, google docs, etc, then you could move this functionality into git.
Jane Street famously<br>does code review by embedding code review comments directly in the source code as code comments,<br>and this workflow decision makes it trivial to involve LLMs in code review because everything in the<br>process is tracked with version control.<br>Why not put issues alongside the<br>codebase? Pull request and code-review metadata in source control? Design docs from Google Docs to<br>checked-in markdown? It would be ideal if all of these were stored in the same repository as the<br>code itself so that changes to code, issues, and docs could be made in a single atomic update<br>instead of having to coordinate across services.<br>In my view, the main obstacle here is that without serious dedication, the user experience for<br>humans would be a major downgrade. This isn’t insurmountable, but it would be a lot of work.<br>A better world for LLMs is a better world for me<br>Although I’ve framed this blog post as “things...