Compaction in Streambed – Streambed<br>Skip to content
CTRL K
Docs
Getting Started
Commands
Configuration
Maintenance
Architecture
Blog
Compaction in Streambed<br>Implementation<br>What’s next
Why I Built Streambed
LightDark<br>System
Light
Dark
System
Blog<br>Compaction in Streambed
Compaction in Streambed
August 9, 2026·Vignesh (Viggy) Ravichandran
Streambed is getting more mature. Big data has a small problem or more precisely small file problem.<br>Every flush (not every single DML) creates a new parquet file. Typically that means hundreds and thousands of small files for a single table. Two challenges with that:<br>It impacts the latency (specifically query read)<br>Maintenance (S3 scales but it’s not like infinite)<br>To solve that we introduce, maintenance and maintenance compact command which does metadata maintenance, small file compaction.<br>Implementation
Reads multiple small files and rewrites them as a large parquet file. Using sqlite for compare and swap. Explored S3 conditional rewites but we will introduce once streambed support multiple hosts. For single host, sqlite row lock is sufficient. Implemented as a separate process to keep the sync process clean and also reliable (avoid co-ordination b/w multiple routine). Eventually should merge it as part of the sync daemon.<br>What’s next
Support compaction in MOR equality delete, optimize COW with key-range pruning,
Why I Built Streambed