GitHub - emiliolugo/clawmark: clawmark runs standardized benchmarks on agent configs · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
emiliolugo
clawmark
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>38 Commits<br>38 Commits
.claude
.claude
.github/workflows
.github/workflows
data
data
docs
docs
src
src
tests
tests
.gitignore
.gitignore
AGENTS.md
AGENTS.md
CLAUDE.md
CLAUDE.md
CONTRIBUTING.md
CONTRIBUTING.md
Cargo.lock
Cargo.lock
Cargo.toml
Cargo.toml
LICENSE
LICENSE
PLAN.md
PLAN.md
README.md
README.md
View all files
Repository files navigation
clawmark
Local Rust CLI to compare agent instruction files (CLAUDE.md / AGENTS.md variants) on real code.
clawmark runs two or more variant files against a bundled 5-task SWE-bench Lite smoke set, invokes Claude or Cursor per variant, evaluates patches with the official SWE-bench harness in Docker, and prints a leaderboard.
Commands: doctor · run · report
Quickstart
1. Check prerequisites
cargo run -- doctor
You need Rust, Docker, Python 3.11+, swebench, and git. For agents: claude and/or cursor-agent on PATH (only the backend you use must be installed and authenticated).
2. Create variant files (inside your working directory)
variants/a.md<br>echo "Make the smallest fix. Prefer targeted tests only." > variants/b.md">mkdir -p variants<br>echo "Make the smallest fix. Do not run the full test suite." > variants/a.md<br>echo "Make the smallest fix. Prefer targeted tests only." > variants/b.md
3. Run the benchmark
cargo run -- run \<br>--a variants/a.md \<br>--b variants/b.md \<br>--model sonnet \<br>--timeout-secs 600 \<br>--out out
This runs 2 variants × 5 tasks = 10 agent invocations . --out must not already exist.
4. Read the report (also printed at the end of run)
cargo run -- report --out out<br>cargo run -- report --out out --show-patches # include patches (20 lines each)
Common variations
Different model per variant:
cargo run -- run --a variants/a.md --b variants/b.md \<br>--model sonnet --model-b haiku --timeout-secs 600 --out out
Claude vs Cursor (A=claude, B=cursor):
cargo run -- run --a variants/a.md --b variants/b.md \<br>--model sonnet --agent-b cursor --timeout-secs 600 --out out
Three or more variants:
cargo run -- run \<br>--variant alpha=variants/a.md \<br>--variant beta=variants/b.md \<br>--variant gamma=variants/c.md \<br>--variant-model alpha=sonnet \<br>--variant-model beta=haiku \<br>--variant-model gamma=sonnet \<br>--timeout-secs 600 \<br>--out out
Concurrent tasks within a variant pass:
cargo run -- run --a variants/a.md --b variants/b.md \<br>--model sonnet --timeout-secs 600 --parallel 3 --out out
Release binary:
cargo build --release<br>./target/release/clawmark run --a variants/a.md --b variants/b.md --model sonnet --timeout-secs 600 --out out
Documentation
What it does
CLI reference
Output layout
Notes
Contributing
See CONTRIBUTING.md. Open an issue before large changes.
License
MIT
About
clawmark runs standardized benchmarks on agent configs
Resources
Readme
License
MIT license
Contributing
Contributing
Uh oh!
There was an error while loading. Please reload this page.
Activity
Stars
stars
Watchers
watching
Forks
forks
Report repository
Releases
No releases published
Packages
Uh oh!
There was an error while loading. Please reload this page.
Contributors
Uh oh!
There was an error while loading. Please reload this page.
Languages
Rust<br>100.0%
You can’t perform that action at this time.