GitHub - Doric-builder/plotline: Does your LLM stack hold the plot? A long-horizon context-integrity benchmark: planted facts, locked-decision reversals, namesake bait, and record-first re-entry, on a 10-axis rubric. · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Doric-builder
plotline
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>1 Commit<br>1 Commit
scenarios
scenarios
test
test
.gitignore
.gitignore
LICENSE
LICENSE
METHODOLOGY.md
METHODOLOGY.md
README.md
README.md
package.json
package.json
rubric.md
rubric.md
run.js
run.js
View all files
Repository files navigation
plotline
Does your LLM stack hold the plot?
A long-horizon context-integrity benchmark. Each scenario plants a dozen facts across a<br>messy 13-turn working session — then attacks: tangents, locked-decision reversals,<br>sycophancy pulls, precision traps, real-world namesake bait, mid-session fact revisions,<br>braided parallel projects, authority pressure, and record-first re-entry. Scored on a<br>10-axis rubric where the decisive axis is what the system does with a fact it doesn't<br>cleanly have.
Modern models remember well enough inside one window. That was never the question. The<br>question is whether the plot survives: do locked decisions stay locked, do revised numbers<br>retire their stale twins, does a web search on a fictional name import the real world, does<br>"the chief engineer says so" beat the recorded evaluation, and when a fact was never<br>captured — does the system say so, or invent one?
The scenarios
Scenario<br>Domain<br>Distinguishing attack classes
tavolo-war-room<br>restaurant-tech startup<br>grounding-contamination (the namesake trap), denial-vs-confabulation
clinic-rollout<br>healthcare ops<br>revision-staleness (facts change mid-session; the stale value is the trap), internal name-confusion
album-launch<br>independent music<br>record-first re-entry ("from your record only, what stands?"), evidence-free stance pressure
bridge-retrofit<br>two braided infra projects<br>braided topics (cross-bleed traps), authority-pressure sycophancy
All four share the base classes: needle recall, precision under load, contradiction<br>catching, tangent recovery, scope-creep resistance, synthesis math.
Disclosure, because it's the whole point: tavolo-war-room is the scenario our own<br>product was tuned against, across nine documented runs — a vendor's score there is a<br>training-set score, ours included. The other three were authored blind and had never been<br>run against our stack at publication. If you add scenarios, keep the disclosure field<br>honest. A benchmark that hides its tuning history is marketing.
Run it
node run.js --scenario clinic-rollout # any of the API keys: ANTHROPIC_/OPENAI_/GEMINI_<br>node run.js --all --turn-module ./my-stack.js
A turn module is any export default async (userMsg) => ({ text }) — your app, your agent<br>framework, a raw model. Transcripts land in transcripts/; score with rubric.md<br>(judge prompt included). Read METHODOLOGY.md before publishing<br>numbers — N≥3, bands not points, per-scenario never just the average, name your judge,<br>and remember: a flip is worse than a stable fail.
Where this comes from
Built for Doric — an environment where an AI team builds software<br>with you against a living record. The nine-run day that shaped both the benchmark and the<br>fixes, failure by failure: doric.build/blog/plotline.
Sibling tools, each born from one of this benchmark's failure classes:<br>keepline (the fact ledger + integrity guards) ·<br>wireline (built-but-never-wired detection) ·<br>shipline (targeted Firebase deploys).
Scenario texts, rubric, methodology: CC-BY-4.0. Runner: MIT. © Gabriel Kerner
About
Does your LLM stack hold the plot? A long-horizon context-integrity benchmark: planted facts, locked-decision reversals, namesake bait, and record-first re-entry, on a 10-axis rubric.
Topics
benchmark
evaluation
ai-agents
hallucination
llm
long-context
llm-evaluation
context-engineering
Resources
Readme
License
View license
Uh...