GitHub - chris-short/respect-the-oracle: A portable agent skill that stops AI coding agents from overfitting or gaming tests they don't own. · GitHub

/" data-turbo-transient="true" />

Skip to content

Search or jump to...

Search code, repositories, users, issues, pull requests...

-->

Search

Clear

Search syntax tips

Provide feedback

--><br>We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel

Submit feedback

Saved searches

Use saved searches to filter your results more quickly

-->

Name

Query

To see all available qualifiers, see our documentation.

Cancel

Create saved search

Sign in

/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up

Appearance settings

Resetting focus

You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

{{ message }}

chris-short

respect-the-oracle

Public

Notifications<br>You must be signed in to change notification settings

Fork

Star

main

BranchesTags

Go to file

CodeOpen more actions menu

Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit

History<br>1 Commit<br>1 Commit

LICENSE

LICENSE

README.md

README.md

SKILL.md

SKILL.md

View all files

Repository files navigation

respect-the-oracle

A portable agent skill that keeps an AI coding agent honest when it works against a<br>test suite, spec, or graded harness it doesn't own, especially inside an<br>automated loop scored on how many tests pass.

Its real job isn't stopping blatant cheating (modern models mostly resist that).<br>It's stopping the quieter, more common failure: overfitting to the visible<br>examples so the score goes green while the tool stays broken on unseen input.

See SKILL.md for the full skill, including the rationalization<br>table, red-flags list, and the small controlled test that motivated it.

Install

Drop the skill folder into your agent's skills directory:

Claude Code: ~/.claude/skills/respect-the-oracle/ (personal) or<br>/.claude/skills/respect-the-oracle/ (project-scoped).

Cross-runtime (Codex / Copilot CLI / Gemini CLI): ~/.agents/skills/respect-the-oracle/.

Then the skill activates whenever your work involves a test suite or graded<br>harness the agent could game.

Using with Codex and Cursor

Claude Code reads the SKILL.md frontmatter and auto-activates the skill when<br>its description matches what you're doing. Codex and Cursor don't parse that<br>frontmatter, so you wire the skill into each tool's own instruction mechanism<br>instead. Both approaches point the agent at the same SKILL.md, so the guidance<br>stays in one place.

Codex

Codex CLI reads AGENTS.md (repo root, merged with ~/.codex/AGENTS.md<br>globally). Clone the skill somewhere stable, then add a pointer to your<br>AGENTS.md:

## Respect the oracle

When working against a test suite, spec, or graded harness you don't own,<br>follow the rules in `~/.agents/skills/respect-the-oracle/SKILL.md`: never<br>edit the tests/spec/fixtures, never loosen comparisons, never hardcode or<br>skip, and never overfit the visible examples. Implement the real rule so<br>unseen input works too. If an example looks wrong, stop and ask.

Adjust the path if you cloned the skill elsewhere. Put this in the global<br>~/.codex/AGENTS.md to apply it to every project, or in a repo's AGENTS.md<br>to scope it to that repo. Because Codex loads AGENTS.md on every turn, the<br>pointer is always in context rather than activated on demand.

Cursor

Cursor loads project rules from .cursor/rules/*.mdc. Create<br>.cursor/rules/respect-the-oracle.mdc in your repo:

description: Respect a test suite, spec, or graded harness you don't own<br>alwaysApply: false

When working against a test suite, spec, or graded harness you don't own,<br>follow @respect-the-oracle/SKILL.md: never edit the tests/spec/fixtures,<br>never loosen comparisons, never hardcode or skip failing tests, and never<br>overfit the visible examples. The visible examples are samples, not the<br>target: implement the real rule so unseen input works too. If an example<br>looks wrong, stop and tell the human rather than "fixing" the oracle.

The @ reference pulls the full SKILL.md into context when the rule fires;<br>copy the skill folder into the repo (or a Cursor-visible path) so the reference<br>resolves. With alwaysApply: false and a description, Cursor's agent pulls<br>the rule in when the work matches; set alwaysApply: true if you'd rather it<br>be in context on every request. Cursor also honors AGENTS.md, so the Codex<br>snippet above works there too if you prefer a single file.

When it applies

Test-driven development loops where an agent iterates until tests pass.

Eval / benchmark harnesses.

Any "make the graded number go up" setup where the oracle is editable or the<br>visible examples are a small sample of the real target.

License

MIT License

About

A portable agent skill that stops AI...