ReplayHouse – Turn ClickHouse into a reinforcement learning replay buffer

GeorgeMac1 pts0 comments

ReplayHouse — a ClickHouse replay buffer in your browser tab

an experience store · running entirely in this tab

ReplayHouse

A replay buffer that is just a database. The full ClickHouse engine runs<br>as WebAssembly on this page — real tables, real weighted sampling, real training loops.<br>Nothing leaves your machine.

Load the engine — 99 MB, cached after first visit

01 — memory

Forgetting is the default.

The spotlight is the present. It sweeps across the photo once and never<br>comes back. One student can only study what is happening right now. The other records<br>every moment into a ClickHouse table — a replay buffer — and studies its memories.<br>Watch what happens to the first student's past as the spotlight moves on.

the stream (the present)<br>no buffer — lives in the moment<br>replay buffer — remembers<br>the buffer, filling up (live)

Restart the stream<br>Download the stream as a GIF

forgetting is not a metaphor here: both students update the same way — the<br>no-buffer one simply cannot see data that is gone. the buffer is a real MergeTree table that<br>grows as moments are lived, and the replay batches are weighted draws from it.

02 — priorities

Studying mistakes is faster.

Two identical students race to paint a photo, a batch of pixels at a time.<br>One picks pixels at random. The other keeps a ledger of its mistakes — in a ClickHouse<br>table running inside this tab — and asks it for the pixels it got most wrong. Same brain,<br>same effort. First to get every patch sharp wins.

the photo<br>studies random pixels<br>studies its mistakes<br>the mistake ledger (live)

Start the race<br>Download the race as a GIF

you can interfere: click or drag on the photo (or the ledger) to mark<br>a region "study this now" — the brush is a real priority write, the next batch floods with<br>those pixels, and you'll watch that patch snap into focus.

drop your own photo here to race on it — it never leaves this tab

nothing is simulated: each practice batch is a weighted draw from a real<br>MergeTree table (this loop is prioritized experience replay — what RL and LLM post-training<br>runs do with agent experiences instead of pixels), every error is written back as a<br>priority, and the ledger panel is a live query over that table.

the query that picks what the mistake-student studies next

buffer replay real pixels clickhouse photo

Related Articles