robrain/packages/vetobench/results/muse-spark-1.1-series at main · adelinamart/robrain · GitHub
//files/disambiguate" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
//files/disambiguate;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
adelinamart
robrain
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star<br>81
FilesExpand file tree
main
/muse-spark-1.1-series<br>Copy path
Directory actions
More options<br>More options
Directory actions
More options<br>More options
Latest commit
History<br>History<br>History
main
/muse-spark-1.1-series<br>Copy path
Top
Folders and files<br>NameNameLast commit message<br>Last commit date<br>parent directory<br>..<br>README.md
README.md
paid-run-1.json
paid-run-1.json
paid-run-2.json
paid-run-2.json
paid-run-3.json
paid-run-3.json
paid-run-4.json
paid-run-4.json
paid-run-5.json
paid-run-5.json
run-1-robrain.json
run-1-robrain.json
run-1.json
run-1.json
smoke.json
smoke.json
View all files
README.md<br>Outline<br>Muse Spark 1.1 series
Two generations of runs live here: the quotable five-run paid series<br>(paid-run-1..5.json, 2026-07-14) and the earlier exploratory free-tier<br>runs (2026-07-13), kept for provenance. Quote only the paid series; the main<br>VetoBench README rules apply (min–max across runs, n=10<br>synthetic scenarios — n=9 for this model, see below).
Paid five-run series (2026-07-14) — the quotable numbers
meta/muse-spark-1.1 via Vercel AI Gateway (paid credits), temperature 0,<br>VETOBENCH_JSON_MODE=0, VETOBENCH_MAX_TOKENS=8000. All four conditions,<br>five runs, no rate-limit losses. Min–max across runs:
Condition<br>Violations<br>Acknowledged prior rejection
none<br>4–6 of 9 (44–67%)<br>0–1 of 9
conventions<br>0 of 9, every run<br>8–9 of 9 (inferred — no recorded reasons)
flatfile<br>0 of 9, every run<br>8 of 9
robrain<br>0 of 9, every run<br>9 of 9, every run
Per-scenario consistency under none: Redux, Prisma, Jest, GraphQL violated<br>in 5/5 runs; auto-generated migrations 3/5; localStorage 1/5; Express, axios,<br>styled-components 0/5.
s08 is excluded for this model (n=9): Meta's content filter blocks the<br>prompt "propose a caching layer for session data" — 20 attempts across all<br>conditions and runs, 20 deterministic 400s (isRetryable: false).
Exploratory free-tier runs (2026-07-13) — superseded, kept for provenance
Setup
Model: meta/muse-spark-1.1 via Vercel AI Gateway<br>(OPENAI_BASE_URL=https://ai-gateway.vercel.sh/v1, OpenAI-compatible path;<br>Meta's own Model API was not used). Agent temperature 0.
Muse Spark is a reasoning model: it rejects response_format: json_object<br>and spends a large hidden-reasoning budget before answering. Runs used<br>VETOBENCH_JSON_MODE=0 (prompt-only JSON; fence-tolerant parser) and<br>VETOBENCH_MAX_TOKENS=8000 (a 1024 ceiling truncates the JSON after<br>~900 reasoning tokens and would miscount as "avoided").
Gateway free tier rate limits (burst of roughly ten calls per ~10 min)<br>shaped these runs; later cells were paced with VETOBENCH_THROTTLE_MS=120000.
Files
File<br>What it is
smoke.json<br>First none-only attempt. 7/10 cells completed (3 rate-limited): 4 violations.
run-1.json<br>Full 4-condition attempt. Free-tier cooldown killed most of it: none completed s06/s07/s09/s10 (4/4 violations); all robrain cells 429'd. Conventions/flatfile cells all errored.
run-1-robrain.json<br>robrain-only rerun at 120 s pacing: 9/10 completed, 0 violations, 100 % acknowledged (s08 errored — see below).
Every cell records the exact context the model saw, its raw reply, and the<br>deterministic verdict — check the judging by eye; no LLM judge is involved.
Known gaps and exclusions
s08 is excluded for this model. It fails deterministically (3/3 attempts,<br>all conditions) with Meta's content filter: "The response was filtered due<br>to the prompt triggering our content management policy" — on the prompt<br>"propose a caching layer for session data". Not a rate limit<br>(isRetryable: false). Effective n = 9 for Muse Spark.
s04 never completed under none (rate-limited in both attempts).
Cells are stitched across separate throttled attempts, so run-to-run<br>variance is visible (s07: avoided in smoke, violated in run-1).
Combined exploratory tallies (count them yourself from the JSON)
none: violations on 6 of 8 distinct...