Making AI Code Review Measurable | azuanMaking AI Code Review Measurable<br>How I built a repeatable evaluation workflow for AI code review using historical pull requests, coding agents, and cost tracking.<br>July 2, 2026 · 9 min · Azuan<br>Code review has always been one of the bottlenecks for software engineers. At loveholidays, we have been using a lot of AI in our daily workflows. Whether a PR is written with the help of AI or not, someone still needs to review it.<br>Then I read a thread by Intercom on how they use AI not just for code review, but also to approve it. Sounds risky, but also very interesting.

Disclaimer: This post is not an endorsement. This was just an experiment inspired by the thread and my own interpretation of how a system like that might work. It was also only possible because OpenAI gave us a three-month trial. During the process, I burned through a few billion tokens. It was a great learning experience, but definitely not cheap.<br>1. Why build this myself?#<br>There are already a lot of code review tools out there. I&rsquo;m sure some of them are great, and some may even be better than what I built. But I have always had a love and hate relationship with code review tools, even before AI. They are often hit or miss. And when the cost is high, it is hard for me to recommend a tool broadly without first understanding how useful it actually is.<br>I also wanted something that could run locally inside my existing workflow and use my existing ChatGPT Codex subscription, whether personal or enterprise. I really, really did not want to pay another subscription.<br>2. Coding agent#<br>Almost all coding agents have a non-interactive mode. With Codex, you can trigger it via codex exec. I ran all sorts of automation with this approach.

I decided to use codex exec to run the code review. Using only the git diff would be a lot faster and cheaper, but I personally do not like that approach. Context matters a lot.<br>When I review a pull request, I rarely look at the diff in isolation. I open related files, check how the surrounding code works, look for similar patterns, and think about how the change could break existing behaviour. I wanted the AI reviewer to be able to do the same.<br>Another reason I liked this approach is that it fits into my existing workflow. The agent can use the same skills, MCP servers, commands, and project context that I already use day to day. That makes the review less like a generic external tool and more like an extension of how I already work.<br>The basic flow looks like this:<br>Clone the pull request into a new git worktree location.git worktree makes it easier to run multiple evaluations in parallel.

Run codex exec with a custom prompt, rules, and output format.Run this in a sandbox, your own machine, or any environment you are comfortable with, as long as you understand the risks and limitations.

Ask the agent to return a structured JSON output.<br>Example output:<br>"verdict": "rejected",<br>"reviews": [<br>"file": "src/file_1.ts",<br>"line": 42,<br>"comment": "This will break feature A.",<br>"suggestion": "Consider rewriting it this way."

With this approach, Codex can inspect relevant files and gather context before performing the review. Because the output is structured, I can also build a simple web interface around the system.

Mock review#<br>The next step was to answer a question. How can I test the code review system quickly?<br>Similar to the Twitter thread, I decided to use past pull requests as the source of truth. This is obviously simplified, but a pull request has a known lifecycle. At a specific commit, the expected outcome is usually clear. Should this be approved, or should it be blocked until the author makes changes?<br>With that in mind, I used previous pull request data as fixtures for mock reviews.<br>Pull RequestCommit hashExpectedTagsNotePR#1aaaApprovedbookingPR#2bbbRejectedbookingPR#2cccApprovedbookingPR#3dddRejectedpaymentCaused an incidentSome examples:<br>PR#1A 10x engineer requested a review and everything was LGTM.

PR#2The author requested a review at commit bbb, and reviewers left comments.<br>The author updated the pull request at commit ccc, and it was finally approved.

PR#3This pull request was originally approved, but it accidentally caused an incident.<br>In the mock review, I want this PR to be rejected. Thanks to my team lead, who asked about this scenario.

This is also where feedback from my colleagues helped a lot. They pointed out scenarios that I had missed, especially around the kind of issues that should block a pull request versus comments that are merely nice to have. That made the fixture list more useful, and made the evaluation better reflect real review expectations from the team.

Comparing evals#<br>My next question was what prompt would produce the most reliable review for this type of pull request.<br>With the coding agent and mock review setup in place, I could test different prompts and compare the results.<br>Prompt A:<br>You are a generic senior software engineer. Review this PR and...