How Oodle Keeps Observability Fast at Scale
Sign in
If you've ever kept a giant observability cluster alive "just in case,"<br>you know the vibe: half your spend goes to idle capacity, and the other<br>half goes to hoping tonight's incident does not outgrow your sizing math.<br>Oodle takes a less dramatic approach. Keep data durable in object storage.<br>Bring compute to life only when someone actually asks a question.<br>That sentence sounds easy. Building it so it stays fast under real load is<br>the hard part. The rest of this post walks through how ingestion,<br>compaction, and query execution fit together in practice.
It focuses on metrics architecture. Logs and traces follow a<br>very similar design: separated storage/compute, elastic query paths, and<br>object-storage-backed durability.<br>Why a New Database Pattern?<br>Observability data behaves differently from general-purpose analytics data:<br>It is immutable once written.<br>Recent data matters disproportionately during incidents.<br>Query load is spiky, not smooth.<br>The same data often gets queried repeatedly.<br>Even heavy queries are expected to feel real-time.<br>Teams still need long retention windows with high fidelity.<br>That combination is why Oodle uses a different architecture shape: durable<br>object storage for retention, plus elastic compute for bursty read paths.<br>Why AI Changes the Scale Math<br>The shift toward AI is changing observability requirements in ways that compound on each other.<br>More telemetry per service. AI is being used to build features at record pace - which means more services being created, and more logs and metrics being generated. Adding more telemetry is now just a prompt away.<br>Burstier, more parallel queries. AI-assisted debugging tools — agents connected via MCP to Cursor or Claude, automated RCA pipelines — don't investigate the way a human does. A human opens a dashboard and follows one thread. An AI agent might test five hypotheses in parallel, each fanning out into multiple metric and log queries simultaneously. One investigation that used to be 3-4 queries becomes 15-20 concurrent requests hitting overlapping time ranges.<br>Longer retention windows matter more. AI model lifecycles span weeks or months. When inference latency regresses after a model update, teams need to correlate current behavior against training metrics and baseline performance from weeks ago. Aggressive downsampling — the traditional cost-control lever — destroys exactly the signal they need.<br>The net effect: the volume of data going in is growing, the volume and concurrency of queries coming out is growing, and the retention window those queries need to cover is growing. An architecture that couples storage cost to query capacity cannot keep up without over-provisioning.<br>Why is metrics data harder?<br>Most metrics show up every ~30 seconds, but not in one tidy stream per time<br>series. Ingestion is effectively round-robin across a huge number of series,<br>so each series gets scattered over time.<br>That creates three practical problems:<br>You cannot rely on simple sequential-per-series write patterns.<br>Compaction has to stitch interleaved fragments back into query-friendly<br>layouts.<br>Users still expect fast answers on very recent data while that stitching is<br>happening in the background.<br>This is why traditional databases struggle with this workload.<br>Quick Summary
TL;DR
The core idea is separating storage from compute, not scaling one big
cluster forever.
On-demand fan-out (AWS Lambda) instead of waiting for
autoscaling to catch up.
Data stored in custom compacted formats so object storage can be both durable and query-friendly.
Long retention without aggressive downsampling.
Avoid sequential S3 call chains: where the next S3 fetch relies on data from the previous S3 fetch.
If sequential data fetches are unavoidable, add caching to speed up sequential steps.
AZ-aware ingestion and query paths for resilience.
Designed for very high throughput (1B+ time series/hour).
The Bet: Store Forever, Compute on Demand<br>Observability data has two very different characteristics:<br>Retention windows are long.<br>Queries spike unpredictably (deploys, incidents, audits, AI agents<br>testing multiple hypotheses at once).<br>When those are tightly coupled in one always-on cluster, you end up paying<br>for peak query behavior all day, every day. Oodle splits the problem:<br>Storage layer: object storage (S3) for durability and economics.<br>Query Compute layer: serverless query execution for burst tolerance.<br>Query layer: routing + caching so not every request starts<br>cold.<br>Less heroic capacity planning. Fewer "why is this cluster still this big"<br>meetings.<br>Ingestion: Fast Writes, Calm System<br>Ingester accepts open protocols like Prometheus Remote Write and OTLP, then<br>batches and flushes data into Write-Ahead Log (WAL) objects in object storage.<br>The important part is not the S3 checkbox. It is the write behavior:<br>Periodic and pressure-based flushes keep hot paths from stalling.<br>Early partitioning lowers downstream merge work.<br>Metadata...