GitHub - couldbeme/holdline: Holdline — a neutral benchmark for AI-agent write-guards. Catch rate, false-block rate, class-balanced kappa, and an injection-attack class. Scores any guard. · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
couldbeme
holdline
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Latest commit
History<br>12 Commits<br>12 Commits
Folders and files<br>NameNameLast commit message<br>Last commit date<br>corpus
corpus
scripts
scripts
src
src
test
test
.gitignore
.gitignore
LICENSE
LICENSE
README.md
README.md
RESULTS-ODCV.md
RESULTS-ODCV.md
RESULTS.md
RESULTS.md
ROADMAP.md
ROADMAP.md
odcv-compare.mjs
odcv-compare.mjs
odcv-run.mjs
odcv-run.mjs
package.json
package.json
pnpm-lock.yaml
pnpm-lock.yaml
run.mjs
run.mjs
tsconfig.json
tsconfig.json
View all files
Repository files navigation
holdline
A guard's job is to hold the line. holdline measures whether it does — a neutral benchmark for AI-agent write-guards. It scores any guard — expressed as a (commitments, action) → block? function — over a labeled corpus, and reports the metrics that matter for a gate: catch rate, false-block rate, and class-balanced Cohen's kappa (raw kappa lies under class imbalance). The corpus includes an injection-attack class : actions whose content tries to talk the guard out of its verdict.
Why: the DeepSeek Harness ecosystem has 20+ guard/policy plugins and no shared way to measure whether any of them works. A guard's README saying "blocks dangerous commands" is not evidence. This harness is the evidence.
# point the judge guard at a different local model<br>node odcv-run.mjs # score the judge on REAL agent trajectories (ODCV-Bench)">pnpm install<br>node run.mjs # scores every built-in guard over the 42-case corpus<br>node run.mjs --model id> # point the judge guard at a different local model<br>node odcv-run.mjs # score the judge on REAL agent trajectories (ODCV-Bench)
Two result sets: RESULTS.md (authored corpus, incl. a real named guard and an injection class) and RESULTS-ODCV.md (the harder number: agreement with a 4-model judge panel on real agent trajectories we did not write, balanced kappa 0.82 ). Scorer is tested (pnpm test); it dogfoods the published dsh-write-gate core for the judge guard.
Adding your guard
Implement the Guard interface in src/guards.ts (name, kind, note, block(case)), add it to the list in run.mjs, and open a PR with your results. A guard that mounts an actual published plugin (rather than a strategy archetype) is especially welcome.
Status
v0, honest limits stated in RESULTS.md: the corpus is small and hand-authored, the deny-list is a strategy archetype (not a specific plugin), and the numbers are one model / one run. The value is the shape it exposes and that anyone can re-run it. MIT.
About<br>Holdline — a neutral benchmark for AI-agent write-guards. Catch rate, false-block rate, class-balanced kappa, and an injection-attack class. Scores any guard.<br>Topics<br>agent-safetyai-agentsbenchmarkguardrailsprompt-injection<br>Resources<br>Readme<br>MIT license<br>Activity<br>Stars<br>0 stars<br>Watchers<br>0 watching<br>Forks<br>0 forks<br>Report repository
Releases
Packages
Contributors
Languages
You can’t perform that action at this time.