Reinforcement Dreaming: Evidence-Guided Search over Executable Futures

survivingdreams.com

Abstract

Most coding agents work in a straight line: plan, build, test, finish. If they make the wrong assumption early, they can waste hours going down the wrong path while still sounding confident about their progress.

We propose Reinforcement Dreaming (RD), a different way to run long-horizon agents. Instead of committing to one path, RD explores multiple executable futures of the same environment. We start with software repositories.

A Dream Run holds the goal, constraints, budget, and state of the task. From there, it can create separate candidate futures, or dreams, inside isolated sandboxes. Each dream can edit real code, run commands, test ideas, and record what actually happened.

The system decides where to spend more compute based on evidence, not on how confident the model sounds. It can keep pushing the main path, open a small side dream to answer a question, explore a completely different approach, combine useful findings, or return to an earlier checkpoint.

State is stored outside the model context, which lets a run continue for six to twelve hours even as the model's context is refreshed.

A candidate only becomes a Surviving Dream after a separate process verifies its claims, checks the important constraints, and records any remaining risks.

Within a single run, RD is an inference-time search and compute allocation system. It is not reinforcement learning. Over many runs, however, the evidence collected could be used to train better value models, improve how compute is allocated, and eventually fine-tune the system.

This paper describes the architecture, algorithms, and evaluation plan, along with the obvious ways it could fail: high cost, bad evaluators, differences between sandbox and production, or simply being beaten by a cheaper approach.

objective · constraints · budget

DREAM RUN

primary dream<br>coherent main implementation

shadows + alternatives<br>targeted uncertainty

evidence ledger<br>receipts · checkpoints · hypotheses

evidence guides allocation

SURVIVAL PIPELINE

clean<br>re-execution<br>semantic<br>review<br>adversarial<br>review<br>hard<br>gates

Surviving Dream<br>verified claims · disclosed risks

human adopts · defers · discards

Architecture overview. A durable Dream Run explores executable futures and allocates compute from recorded evidence. Independent gates decide what may be presented to a human as a Surviving Dream.

1. Introduction

Consider a concrete task: reduce the 95th-percentile latency of a service's request router by 30% without changing its public API.

A coding agent gets several hours to work on it. Early in the run, it decides serialization is the bottleneck and starts changing code around that theory. Three hours later, a benchmark points somewhere else: lock contention in the connection pool. By then, the context is full of serialization work, the summaries say progress is good, and the useful checkpoint from three hours ago is gone.

The problem is bigger than one bad guess. Long-running agents tend to commit to the first plausible path, mix their own narrative with what the environment actually proved, lose important state as contexts refresh, and keep working because a process is still running rather than because the work is still useful. Agent-computer interfaces made repositories practical environments for language-model agents [20], and execution-based benchmarks made their output easier to check [21], but most systems still push one main trajectory forward with retries.

Reinforcement Dreaming (RD) is an architecture for doing something different. It keeps several possible futures available and lets evidence decide which ones deserve more work.

Software repositories make this unusually practical. A worktree and its toolchain can be copied into an isolated sandbox where an agent edits real files, runs real commands, and sees real results without touching the original repository. Different theories about the task can therefore be tested in code instead of argued about in text.

A long run also needs an identity that survives the model working on it. RD stores the objective, constraints, budget, hypotheses, checkpoints, worktrees, open questions, and recorded outcomes outside any individual context. A model context becomes a temporary worker inside a larger Dream Run and can be refreshed without restarting the task.

Anything that touches the environment goes through an instrumented execution layer. Commands, results, commits, artifacts, benchmarks, and environment information are written to an append-only evidence ledger. Strategic decisions are then made from those records rather than from the model's description of its own progress.

Agent summaries are claims. Executed outcomes are evidence.

A Dream Run usually has one primary implementation. When the system reaches an important uncertainty, it can open a small shadow dream to answer one question, fork a larger...