The Art of Not Buying Disk - Marcos Sánchez
I have released rpkv v0.1.0: key-value<br>reads over Redpanda topics without duplicating values. That is the whole<br>pitch, and “without duplicating values” is not a feature. It is a grudge.
The economics of 2026 hardware
You may have noticed – perhaps while weeping at a checkout page – that<br>RAM is currently priced like it is mined by hand, and NVMe has joined it<br>in the stratosphere out of solidarity. The industry’s advice is timeless:<br>buy more anyway. Storage is cheap, they say, in articles written when it<br>was.
My homelab did not get the memo. It has the disk it has, and everything<br>I care about lives in Redpanda topics – if it is not in a topic, as far<br>as I am concerned it does not exist. And I am not buying more disk to<br>store the same bytes twice.
The problem
A compacted topic already holds the latest value for every key. What it<br>cannot answer cheaply is “latest value for key X” – the log knows, but<br>only if you replay it, which is less a query than a pilgrimage.
Every respectable answer is some flavour of “copy every value into a<br>second store”. Kafka Streams materializes your topic into RocksDB. A<br>cache materializes it into RAM – see above, re: weeping. Fine<br>engineering, all of it, billed in gigabytes I already spent.
The grudge, implemented
rpkv is a pure-Go sidecar that keeps only an index: key -> (partition, offset), in embedded Pebble.<br>14.2 bytes per key measured, whether the value is a heartbeat or a novel.<br>Reads fetch exactly that one record from the log over the plain Kafka<br>protocol. The values stay where they were. The index is a projection:<br>destroy it and it rebuilds from the topic at ~996k keys per second,<br>faster than I can regret deleting it.
The trade-off, printed on the tin: every read pays a broker round-trip,<br>11.1 ms p50 measured. If you need microseconds, buy the RAM and my<br>condolences. Big values, modest read rates, a disk budget held together<br>by spite – that is the sweet spot, and it is shaped like my rack.
Compaction and retention are handled by a read verification protocol,<br>not by optimism: fetches are never trusted blindly, and superseded<br>pointers wait for the index to catch up. When retention deletes a<br>record, rpkv answers 410 and moves on – the topic is the system of<br>record, and if the log decided those bytes were not worth keeping, who<br>am I to argue. Details in the repo, which for once has more tests than<br>excuses.
On the icon
The logo was drawn by ChatGPT: a little panda – as the Chinese call the<br>red one – sitting on a stack of red boxes you have seen before.<br>Infringing one trademark is carelessness; infringing two in one icon is<br>curation. Any resemblance is a matter between my lawyer and theirs –<br>only one of whom exists.
This is how it has always worked. Rick Houlihan called it data<br>pressure in his re:Invent<br>2018 talk: data presses against the technology that holds it until<br>something gives, and what gives is usually somebody too stubborn to buy<br>more hardware. rpkv is my small entry in that tradition, and I hope it is<br>useful to somebody else’s stubborn rack.
So: go install, point it at a topic, and stop paying for your data<br>twice.