Blog - Retriever: A Programming Framework for Closed-Loop Robot Agents<br>← Project Technical Blog<br>Retriever: A Programming Framework for Closed-Loop Robot Agents<br>A programming framework for building closed-loop robot systems with explicit timing and data handoff, so perception, reasoning, and control can run together at mismatched rates.<br>Technical write-up Closed-loop robot agents Multi-rate control

Retriever is a programming framework for general-purpose robot agents. It supports closed-loop systems that combine slow VLM reasoning, medium-rate skill execution, and high-rate control in a single program — each module running on its own independent clock, from seconds down to milliseconds. Timing and data handoff are explicit parts of the program, not buried inside callbacks or middleware behavior.

To show what this looks like in practice, we begin with Retriever-0 — a concrete closed-loop agent we built and deployed with the framework.

Retriever-0: a closed-loop manipulation agent

Retriever-0 is a bimanual manipulation agent built entirely inside a single Retriever pipeline. It runs a VLM planner, an execution monitor, a VLA skill policy, and a joint controller in one coordinated closed loop — each module on its own clock, from seconds down to milliseconds.

Retriever-0 is evaluated on two families of long-horizon tasks that require combining partial observability, conditional replanning, and dexterous manipulation:

Retrieval from a deformable bag. The agent reaches into a cloth bag, locates a target object under uncertainty, and extracts it while continuously adapting its grasp strategy as the bag deforms under manipulation.

Spice search and seasoning. The agent searches a set of drawers for a target spice under partial observability, updates its belief based on what it finds, and completes a multi-step seasoning task using retrieved tools. The planner branches conditionally on inspection outcome, so the execution trace varies depending on which drawer holds the target.

A representative trace from the spice search task:

OpenDrawer(TL) → InspectDrawer(TL) → CloseDrawer(TL) → OpenDrawer(TR) → InspectDrawer(TR) → Pick(spice) → Season(food) → CloseDrawer(TR)

These tasks require a VLM planner reasoning over seconds, a VLA skill policy running at ~2Hz, and a joint controller running at ~200Hz — all exchanging information inside one loop. The execution monitor watches skill progress and triggers replanning when needed; the VLA receives active skill commands and emits action chunks for the controller to execute. No module is isolated; the loop is closed.

Task familyRetrieval under occlusion<br>The agent searches inside a deformable container, updates belief from new observations, and adapts the manipulation strategy as the scene changes.

Task familyConditional search and use<br>The agent inspects candidate locations, replans based on what it finds, and completes a multi-step task using the retrieved object.

The Retriever-0 pipeline: slow VLM planning (with belief and memory), execution monitoring, medium-rate VLA skill execution, and high-rate control — all running simultaneously at mismatched clocks inside one program.

# Retriever-0 pipeline — adapted from the actual Python code (names simplified; port mappings omitted)

wrist_cams = WristCameraFlow() @ Rate(hz=30) # 4× wrist cameras (left/right, top/bottom)

head_cam = HeadCameraFlow() @ Rate(hz=30) # stereo head camera

goal = GoalFlow() @ Rate(hz=1)

belief = BeliefMemoryFlow() @ Trigger("inspection_done")

planner = VLMPlanFlow("gemini") @ Trigger("replan_request")

monitor = ExecMonitorFlow() @ Rate(hz=10)

vla = VLASkillFlow("pi05") @ Rate(hz=2)

ctrl = ControllerFlow() @ Rate(hz=200)

teleop = TeleopFlow() @ Rate(hz=100)

with Pipeline("Retriever-0") as pipe:

# goal → planning and monitoring

goal.then(planner, sync=Latest())

goal.then(monitor, sync=Latest())

# perception → belief → planning

wrist_cams.then(belief, sync=Latest())

belief.then(planner, sync=Latest())

head_cam.then(planner, sync=Latest())

# planning ↔ execution monitoring (replanning loop)

planner.then(monitor, sync=Latest()) # plan chunks

monitor.then(planner, sync=Latest()) # replan requests

# monitoring ↔ skill execution

monitor.then(vla, sync=Latest()) # active skill command

vla.then(monitor, sync=Latest()) # progress p ∈ [0,1]

# operator override

teleop.then(monitor, sync=Latest())

# skill → control (action chunking handled as sync policy)

wrist_cams.then(vla, sync=Latest())

vla.then(ctrl, sync=Latest())

# Message passing backend supports in-process, Python multi-processing, and Rust-based dora-rs

retriever.init(backend="dora")

pipe.run(duration=300.0)

Each module is a Flow — a Python class with a step() method — attached to a clock that says when it runs: @ Rate(hz=2) fires on a fixed schedule, @ Trigger(...) fires on an event. The pipe.then(a, b, sync=Latest()) calls wire the graph and declare how each edge samples upstream data at runtime. The rest of this...