GitHub - PhillipChaffee/big-pickle-swe-atlas: big-pickle (OpenCode Zen stealth model) scores 50.8% on Scale AI's SWE Atlas Codebase QnA — full results, verifier logs, and reproduction configs · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
PhillipChaffee
big-pickle-swe-atlas
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>2 Commits<br>2 Commits
results
results
run_config
run_config
.env.example
.env.example
.gitignore
.gitignore
LICENSE
LICENSE
README.md
README.md
preflight.sh
preflight.sh
View all files
Repository files navigation
Big Pickle on SWE Atlas — Codebase QnA
Task Resolve Rate: 50.8% (63/124) — big-pickle, the free stealth model on OpenCode Zen, evaluated on Scale AI's SWE Atlas Codebase QnA benchmark using the mini-swe-agent scaffold.
Run on 2026-08-11 with the official open-source harness, task data, and judge model.
Result in context
Against the official SWE Atlas QnA leaderboard (updated 2026-07-28):
Model (scaffold)<br>Task Resolve Rate
Opus 5 (Claude Code, xHigh)<br>63.17
Opus 4.8 (Claude Code, xHigh)<br>57.26
big-pickle (Mini-SWE-Agent) — this run<br>50.81
GLM 5.2 (Mini-SWE-Agent)<br>48.12
GPT-5.6-Sol (Codex, xHigh)<br>46.00
GPT 5.5 (Codex, xHigh)<br>45.43
Within the Mini-SWE-Agent scaffold class — the apples-to-apples comparison — this run outscores every entry on the official leaderboard , and it also tops the Codex-scaffold GPT entries. Only the two Claude models running on their native Claude Code scaffold score higher. Note the caveats below before treating this as a leaderboard-equivalent number.
By language
Language<br>Resolved<br>Rate
TypeScript<br>18/31<br>58.1%
Python<br>16/29<br>55.2%
Go<br>19/38<br>50.0%
10/26<br>38.5%
By category
Category<br>Resolved<br>Rate
Code Onboarding<br>17/28<br>60.7%
Architecture & system design<br>23/44<br>52.3%
Root-cause analysis<br>17/37<br>45.9%
Security<br>5/11<br>45.5%
API & library usage / integration<br>1/4<br>25.0%
Method
Everything follows Scale's published protocol as closely as budget allowed:
Tasks: all 124 Codebase QnA tasks from scaleapi/SWE-Atlas (Apache-2.0), unmodified — including Scale's shipped mswea_qa_config.yaml agent configuration (system/instance templates, step_limit: 250).
Harness: Harbor v0.18.0 with Modal sandboxes, per the SWE-Atlas README.
Scaffold: mini-swe-agent pinned to 2.4.6 — the same minimal bash-only scaffold Scale uses for non-first-party models on the leaderboard.
Model: big-pickle via OpenCode Zen's OpenAI-compatible endpoint (https://opencode.ai/zen/v1), litellm route openai/big-pickle. Total consumption: 674M input / 4.3M output tokens, at $0 (the model is free during its stealth period).
Judge: claude-opus-4-5-20251101 — the exact judge model Scale specifies — accessed through Anthropic's OpenAI-compatible endpoint (https://api.anthropic.com/v1) with EVAL_MODEL overridden to the bare Anthropic model ID.
Scoring: the benchmark's own rubric-based verifier, unmodified. A task resolves only if every scored must-have rubric passes.
Caveats
Read these before quoting the number:
Single trial per task (-k 1). The official protocol runs 3 trials and reports the mean. At n=124, the single-trial standard error is ≈ ±4.5 points — comparable to the leaderboard's own reported error bars (±5).
Reduced sandbox resources. Tasks declare 16 CPU / 16 GB; this run used 4 CPU / 8 GB to fit a personal budget. Slower command execution can only depress an agent's score (via command timeouts or OOM kills), not inflate it. Empirically it appears to have had no effect here: a scan of all 124 agent trajectories found zero command timeouts and zero exit-137 kills — no command ever hit the 900s ceiling or the memory limit.
Self-reported. Scale did not run or verify this evaluation. The full per-task verifier logs in this repo allow independent auditing, and the run is reproducible from the configs here plus the public SWE-Atlas repo.
Model identity unknown. big-pickle is officially unconfirmed; leaked provider errors and API response signatures suggest it is currently served by DeepSeek infrastructure. The underlying model may change without notice, so this result is a snapshot of whatever was behind the alias on 2026-08-11.
Data exposure. OpenCode states that prompts to big-pickle during its free period may be used to improve the model. The benchmark's task content (already public, canary-marked by Scale) was necessarily sent to that endpoint.
Two resolved tasks had unscored rubrics. On task-...ba9ad (5 of 11 rubrics) and task-...baa1d (1 rubric), the judge returned...