How to build GitHub code review agent - Saba

Saba

SubscribeSign in

How to build GitHub code review agent<br>Build your own code review agent in under 30 minutes

Saba<br>Jul 09, 2026

Share

There are a lot of code review tools popping up lately. People assume these tools are very complex, while some might be, building one you can use on day to day basis is pretty simple and straightforward. In this post I will take you through the whole process.<br>High Level Architecture

On a high level, a code review agent needs to:<br>Detect when its mentioned on Pull Request

Clone the codebase in temporary directory

Read the diff between base and head branch

Start the session with relevant context and tools

Let the agent inspect relevant files, reason about changes and produce feedback

Use GitHub comment tool to leave comments directly on the pull request

We will configure GitHub Application that gives our workflow permissions to detect the bot mentions, clone repositories and post reviews on the pull request. Agent with file system , memory and tools, including bash, memory and comment(calling GitHub api to post a review), and finally a GitHub integration and a workflow that ties all of this together.<br>Prerequisites

Before we get started we need to create a GitHub App. This is the least fun part of setup, so it’s better if we do this one upfront.<br>Go to GitHub and open:<br>Settings → Developer Settings → Github Apps → New Github App<br>For personal accounts the link is: https://github.com/settings/apps/new<br>When GitHub asks for Webhook settings, clear the Active checkbox, we will do polling for simplicity of the setup.<br>Next, set following Repository permissions:<br>Contents: Read Only

Issues: Read & Write

Pull Requests: Read & Write

Metadata: Read Only

And finally, create the app.<br>Once created, note the App Id down. Head to the App Settings page and generate the Private Key. GitHub will download the .pem file automatically, Finally click Install Application and add it to the repository you want to review pull requests on.<br>At the end of this set up you should have:<br>App Id

Private Key(.pem file)

Application installed on the repository you want to review

Setting up an agent

For building an agent we will use open source project i am working on called Wolli, it’s built for long running, purpose built agents which lets you launch minimal agents and extend them over time by talking to them. To get started run<br>npm i -g wolli<br>wolli

This will guide you through the onboarding, I recommend using your existing OpenAI Subscription rather than API keys, since running agents could cost a lot of money.

Once you have configured the model, you will want to close the interactive terminal and create an agent with the following command<br>wolli new code-reviewer

This creates an agent and runs it in the background as a daemon, you can always type Wolli in terminal to inspect the agents you have created and stop/delete them. here is what we have created so far

Now we want to add GitHub integration, which will hold the credentials, listen to events and expose actions like addComment to workflows. you can think of this as 2 part system where integration defines capabilities and workflow decides which of those capabilities get used by an agent, here is an example workflow

Luckily for us, Wolli comes with built in plugin that predefines integration, workflow and a tool that lets agent leave a comment. Here’s what plugin looks like on high level:

If you wanna do a deep dive into GitHub plugin, you can visit this page . to install a plugin you can run the following command<br>wolli code-reviewer plugins install "$(npm root -g)/wolli/built-in/plugins/github"

When asked about private file, I recommend providing it as:<br>!cat /Users/saba/Downloads/path-to-key.2026-07-08.private-key.pem

rather than copy-pasting its contents, since it will not accept multi line input and you would have to manually remove new lines.<br>once that’s done, you will want to restart the agent by:<br>wolli restart code-reviewer

Here is what full sequence looks like in terminal:

This is what our setup looks like now:

We are almost done, just missing one more piece, a system prompt. We have wired up everything but we have not told an agent what it’s supposed to do, it will be very confused when it gets a GitHub event. To prevent that, you can run:<br>wolli code-reviewer

This will open an interactive chat interface with an agent, you need to tell it what you want it to do. I provided my prompt below, you can customize it to your preferences, maybe tell it what kind of code you like, what you don’t like.<br>I have already configured built in github integration.

You are a github code reviewer.

When mentioned on pull request you should review the diff.

You should focus on real issues such as bugs, edge cases, security, performance, meaningful code duplication.

For each problem explain the problem why it matters and suggest concrete fix.

Keep in mind, you are free to have a conversation...