Session SWE Bench - SpecGoogle ドキュメントを使用して公開済み<br>不正行為を報告する詳細

Session SWE Bench - Spec<br>5 分ごとに自動更新

Session SWE Bench

Evaluating coding agents in real sessions<br>Quality and cost, measured the way people actually work: a multi-turn session in a real harness. Open discussion draft, feedback welcome.<br>Goal is to answer<br>“What’s the quality vs cost spread for MY typical session when using Claude Code or Codex?”<br>Why<br>A. Benchmarks score one isolated task (but real work often happens in long-running sessions)<br>SWE-bench Verified runs 500 single tasks, one per environment.<br>DeepSWE runs one long task through mini-swe-agent.<br>Neither matches real use .In most real sessions, people often start with one chunky task and then keep going with follow ups and other related smaller tasks within the same session.

B. Benchmarks use lightweight test harnesses (but we use extensively designed coding harnesses)<br>In the real world we use Codex or Claude Code, where context is managed differently compared to the lightweight test harnesses. This can change and impact both the output quality as well as the cost of sessions.

What to measure<br>Cost. Dollars, how many turns it took to finish<br>Quality. How good is a specific model x harness combination

Plan<br>Build sessions. One heavy opening task, then related follow-up tasks from the same repo<br>Run in real harnesses. Codex, Claude Code at default settings (for example GPT-5.5, high effort), native tools and pricing included.<br>Grade the overall session output. We run pass/fail on each task in the session. Tasks are weighted by difficulty. For fail scenarios, grade for progress.

Open questions<br>Session realism.<br>Are 10 SWEBench-Verified tasks threaded together a good proxy for a realistic session? >> Tasks pulled from the same repo to maintain subject matter continuity,<br>How should we order them? >> Should we open with a big difficult task upfront followed by medium and small tasks<br>What should be the delay between task starts? >> For realistic sessions, the time to the next query isn’t always consistent. Sometimes you need longer to think or you are just away from your desk. This impacts cost because of cache TTLs.

References: SWE-bench, DeepSWE (Datacurve), mini-swe-agent.