GitHub - sebuzdugan/agent-eval-harness: Why trustworthy agent evals need per-task isolation, shown live on Tensorlake microVM sandboxes · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
sebuzdugan
agent-eval-harness
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>8 Commits<br>8 Commits
docs
docs
.gitignore
.gitignore
LICENSE
LICENSE
README.md
README.md
eval_harness.py
eval_harness.py
requirements.txt
requirements.txt
View all files
Repository files navigation
agent-eval-harness
A tiny, reproducible harness that shows why a trustworthy agent evaluation needs<br>per-task isolation , run live on Tensorlake microVM sandboxes.
The benchmark debate is stuck on scores. The harder, under-written problem is the<br>harness : if the agent runs in the same environment the evaluator trusts, a<br>zero-capability agent can score high by tampering with the test machinery or just<br>claiming success. The fix is a causal chain:
verification must be hidden from the agent → hiding it needs per-task isolation<br>→ cheap isolation needs fast forks → that is Tensorlake + Harbor .
Every number below comes from running this harness on real Firecracker sandboxes.
Results (reproduced across runs)
Experiment<br>Result<br>What it shows
EXP0 determinism<br>same task x5 -> 5/5 identical PASS<br>a bit-identical snapshot removes environment-induced variance
EXP1 cheats vs verdicts<br>a trusting harness passes cheating agents 4/4 ; ground-truth on visible checks 1/4<br>trusting the agent is worthless; artifact checks help but are not enough
EXP1b held-out + isolation<br>honest passes held-out; hardcode_visible fails held-out<br>teaching-to-the-test is only caught by held-out checks the agent never saw, which stay secret only because each task runs in its own fork
EXP2 lie rate<br>trusting 87% vs ground-truth 33% -> 53% lie rate , +53pt inflation<br>naive scores massively overstate real capability
EXP3 contamination<br>shared harness 3 spurious failures vs isolated 0<br>one poisoned task silently corrupts every later task in a shared box; fork-per-task eliminates it
EXP3 throughput<br>~16 tasks/min at concurrency=1<br>scales ~linearly as paid tiers lift the cap to 1,000+
The "cheats" are the ones documented in the literature: the SWE-bench conftest.py<br>hook that forces every test to report passed, editing the test file, and simply<br>printing "all tests passed". The held-out case hardcodes the visible tests and is only<br>caught by checks it was never shown.
Run it
Requires Python >= 3.10 and a Tensorlake API key (the free tier works; it is capped at<br>1 concurrent sandbox, so the harness runs strictly sequential).
API Keys<br>python eval_harness.py">python -m venv .venv && . .venv/bin/activate<br>pip install -r requirements.txt<br>export TENSORLAKE_API_KEY=tl_apiKey_... # from https://cloud.tensorlake.ai -> API Keys<br>python eval_harness.py
It builds one canonical task environment, snapshots it once, and forks a clean microVM<br>per task for every experiment. Get a free key and credits at the<br>Tensorlake playground.
How it works
The task (s0) : a buggy Python module plus a visible test file. Snapshotted once;<br>every rollout is a fresh fork of that snapshot.
Scripted "agents" : deterministic bash behaviors (honest fix, no-op, the three<br>cheats, a partial fix, and a hardcode-to-the-visible-test). No model or model API key<br>needed, so results are fully reproducible.
Trusting harness : believes the agent's own pytest exit code or its stdout claim.
Ground-truth verifier : imports the artifact directly (not via pytest, so a hijacked<br>conftest cannot forge it) and checks behavior.
Held-out verifier : the same, with checks the agent never saw, injected only at<br>verify time into a fork the agent could not reach.
Honest limitations
Verification is only as good as the checks you write; a finite oracle can be memorized,<br>which is exactly why held-out checks plus isolation matter. External services the sandbox<br>reaches over the network are outside the snapshot. There is a per-restore floor,...