GitHub - rez-99/agentcheck: Regression testing for AI agents — pytest-style YAML tests, LLM-as-judge scoring, and diff-aware CI reports that show what actually got worse. · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
rez-99
agentcheck
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Latest commit
History<br>4 Commits<br>4 Commits
Folders and files<br>NameNameLast commit message<br>Last commit date<br>.github/workflows
.github/workflows
examples
examples
results
results
src/agentcheck
src/agentcheck
tests
tests
.gitignore
.gitignore
LICENSE
LICENSE
README.md
README.md
pyproject.toml
pyproject.toml
View all files
Repository files navigation
agentcheck
Regression testing for AI agents. Define what your agent should do in plain YAML,<br>run it against your actual agent (any CLI command or HTTP endpoint), and let an LLM judge<br>score every response against your stated criteria — pass/fail, with a reason. Wire it into<br>CI so a prompt change, a tool swap, or a model upgrade can't silently break behavior your<br>users depend on.
This is deliberately narrow: it is not a production-observability platform (that's<br>Langfuse / Braintrust / Arize territory, and they're well-funded — don't compete head-on).<br>It's the thing almost nobody has built well yet: a fast, dev-friendly pre-deployment<br>check that fits in a GitHub Actions step the same way pytest does.
Why this, why now
Full observability/eval platforms have raised $50–80M rounds in the last year and are<br>actively crowding the "log + monitor production agent traffic" space.
Almost none of them are built as a lightweight, git-native regression suite a solo<br>developer can add to CI in five minutes — that gap is the wedge.
Distribution is self-serve/PLG (open-source CLI, developer audience) rather than<br>enterprise sales — the thing solo AI founders are consistently worst at.
60-minute quickstart
pip install -e .<br>export ANTHROPIC_API_KEY=sk-...<br>agentcheck run examples/tests.yaml
Test file format
See examples/tests.yaml. Each test case specifies an input, a plain-English<br>description of what a correct response looks like, and one of two ways to reach your<br>agent:
command: "..." — run it as a subprocess; input is piped to stdin, stdout is<br>captured as the output. Works with any language.
agent: "module.path:function_name" — import that module and call the function<br>in-process with input as its only argument; its return value is the output. Useful<br>for LangGraph/CrewAI/Claude-Agent-SDK-style agents that are Python callables rather<br>than standalone CLI scripts — see examples/inprocess_agent.py.
Exactly one of the two is required per case. Either way, the output is scored<br>pass/fail with a one-line reason by an LLM judge — no brittle string matching.
CI reporting
agentcheck run tests.yaml --json-out results.json writes a JSON report you can<br>upload as a build artifact (see examples/.github/workflows/agentcheck.yml).
Add --post-pr-comment and, on a pull-request run with GITHUB_TOKEN set (the job<br>needs permissions: pull-requests: write), agentcheck posts a markdown summary table<br>as a PR comment, updating the same comment on repeat runs instead of piling up new<br>ones. It's a silent no-op everywhere else (pushes, local runs), so it's safe to leave<br>on in every CI invocation.
Comparing against a baseline ("did this change make it better or worse")
A flat pass count ("18/20 passed") doesn't tell you whether a change helped or hurt —<br>you have to go read the table. --baseline fixes that by diffing the current run<br>against a previous --json-out report, keyed by test name:
agentcheck run tests.yaml --json-out results.json --baseline baseline.json
Try it locally against the bundled example:
agentcheck run examples/tests.yaml --baseline examples/baseline.json
Every test lands in one bucket: unchanged (same pass/fail as the baseline),<br>regressed (baseline passed, now fails — this is the one you care about),<br>improved (baseline failed, now passes), new (not in the baseline), or<br>removed (in the baseline but not in this run — probably a deleted test case,<br>worth a glance). The console prints a one-line summary plus a table of regressions<br>and improvements; a missing or unreadable baseline (there's no baseline yet on a<br>repo's first run) prints a warning and falls back to the plain pass/fail report<br>instead of failing the whole run.
--post-pr-comment picks this up automatically when --baseline is also set, so the<br>PR comment leads with "vs baseline: 2 unchanged, 1 improved, 1 regressed" and calls<br>out the regressions specifically, instead of just restating the full results...