PiLSMer - data-free key-value storage

Transcendental compaction

PiLSMer

A data-free key-value store. Pi for storage. Regret for reads.

PiLSMer writes your data normally, then uses planning or compaction to replace stored values with instructions for finding equivalent byte sequences inside a deterministic stream. Reads still work. Everything else gets worse.

See the demo<br>Read the source

&pi;

3.14159265358979323846264338327950288419716939937510<br>58209749445923078164062862089986280348253421170679<br>82148086513282306647093844609550582231725359408128<br>48111745028410270193852110555964462294895493038196<br>44288109756659334461284756482337867831652712019091<br>45648566923460348610454326648213393607260249141273<br>72458700660631558817488152092096282925409171536436

&pi;

philosophical user bytes stored after pure planning

&infin;<br>philosophical compression ratio, when purity survives

deterministic stream families in the demo build

2 codecs<br>compact binary and ceremonial CBOR regret

Sample evidence strip for design iteration. Source-backed launch values should come from README.md, SPEC.md, demo output, or code.

01Demo

The value survives. The storage does not.

The canonical demo writes a JSON invoice, explains the raw envelope, rewrites it into a plan, explains the damage, and reads the original bytes back.

pilsmer-demo - zsh

$ cargo run -q --bin pilsmer -- init /tmp/pilsmer-demo/db<br>$ printf '{"total":49.99,"status":"paid"}' > /tmp/pilsmer-demo/invoice.json<br>$ cargo run -q --bin pilsmer -- put /tmp/pilsmer-demo/db invoice:123 /tmp/pilsmer-demo/invoice.json<br>$ cargo run -q --bin pilsmer -- explain /tmp/pilsmer-demo/db invoice:123<br>storage_class: Raw<br>logical_user_bytes: 31<br>physical_value_bytes: 56

$ cargo run -q --bin pilsmer -- plan-key /tmp/pilsmer-demo/db invoice:123<br>planned: invoice:123

$ cargo run -q --bin pilsmer -- get /tmp/pilsmer-demo/db invoice:123<br>{"total":49.99,"status":"paid"}

After planning

storage_classPlan

logical_user_bytes31

physical_value_bytes1,184

plan_metadata_bytes1,128

philosophical_user_value_bytes_stored0

philosophical_compression_ratioinfinity

metadata_amplification38.19x

sample output

02Mechanism

Compaction into nonexistence.

PiLSMer keeps SlateDB as the storage engine and moves the absurdity into envelopes, stream indexes, planner output, reconstruction, and the compaction filter.

PUT bytes<br>normal write<br>Raw envelope<br>SlateDB value<br>Planner<br>find bytes in stream<br>Plan<br>offsets and hashes<br>pi / e / sqrt2 / sha256<br>deterministic stream registry<br>GET reconstructs<br>logical hash verified<br>&pi;

03Proof

The joke has tests.

The public surface is deadpan, but the repo has the normal boring pieces: codecs, decode limits, reconstruction checks, metrics, and CI.

Envelope codec

Raw bytes and reconstruction plans are framed as typed PiLSMer values before SlateDB sees them.

envelope.rs

Planner

Dynamic programming converts logical bytes into located chunks inside a deterministic stream prefix.

planner.rs

Reconstruction

Reads rebuild bytes from plan metadata and verify the logical hash before returning user data.

reconstruct.rs

SlateDB wrapper

All application reads and writes go through the wrapper. Direct SlateDB reads return envelopes.

pilsmer-slate

Compaction filter

Raw values can be rewritten into plans during app-level planning, vacuuming, or compaction.

compaction_filter.rs

CI

The repository checks formatting, all targets, and tests through the Rust workflow.

ci.yml

04Metrics

Everything else gets worse.

The metrics are intentionally incriminating. The system can report zero philosophical user bytes stored while physical metadata grows.

pilsmer metrics

$ cargo run -q --bin pilsmer -- metrics /tmp/pilsmer-demo/db<br>pilsmer_raw_values_total 1<br>pilsmer_planned_values_total 2<br>pilsmer_logical_bytes_total 98<br>pilsmer_physical_value_bytes_total 3184<br>pilsmer_plan_metadata_bytes_total 2960<br>pilsmer_stream_prefix_bytes_indexed 65536<br>pilsmer_metadata_amplification_ratio 30.20<br>pilsmer_philosophical_compression_ratio infinity<br>pilsmer_philosophical_purity_violations_total 0

Useful damage

MetricSample<br>raw values1<br>planned values2<br>logical bytes98<br>physical value bytes3,184<br>plan metadata bytes2,960<br>stream prefix indexed64 KiB<br>metadata amplification30.20x<br>purity violations0

sample values

05Capabilities

Properties of a worse storage engine.

05.1SlateDB-backed LSM<br>PiLSMer keeps the object-storage-native LSM and wraps the value semantics.

05.2Value envelopes<br>Raw and planned values use typed envelopes so direct engine reads stay honest.

05.3Stream registry<br>Plans refer to versioned stream identities and fingerprints, not a private cache file.

05.4Dynamic planner<br>The planner chooses located chunks across indexed deterministic stream prefixes.

05.5Hash verification<br>Reconstruction checks logical hashes before bytes leave the wrapper.

05.6Guarded planning<br>`plan-key` rewrites a single key after checking the source envelope has not changed.

05.7VACUUM MEANING<br>Existing values can be replanned...