Observability ingestion on S3 alone · TripleCloud Blog

Can you build observability ingestion on S3 alone - no Kafka, no disks, no coordination layer?<br>Yes - you can run observability ingestion on S3 alone, with no stateful layer to operate: no Kafka, no local disks, no coordination service. We replaced an Apache Kafka + Flink + OTel pipeline (~$700–800/month at 10 MB/s) with one engine where the data, the WAL, and the Apache Iceberg catalog all live in S3, with durability backed by S3. Here's how it works.<br>June 22, 2026<br>10 mins to read

Share

Self-hosted observability sooner or later runs into the problem of storing state. Query load, CPU, and data volume can all be handled by scaling out, but the stateful layer is something you have to operate by hand. At first it's almost unnoticeable: a disk degrades here, replication falls behind there, a recovery hangs somewhere else. As the data grows, incidents stop being one-offs and start to recur. At some point your observability stack - whether it's Grafana Loki, Elastic, or ClickHouse - starts demanding the same attention as a full-blown database that you're on the hook for.

Kubernetes operators cover some of these cases, but operating the state is still on you. Managed solutions take that burden away and bring their own: rising costs, ingestion-pipeline constraints, and limits on retention and cardinality.

But if you'd rather not sign up for the constant operational grind - or live with the constraints of managed solutions - it's worth asking: can we take the stateful part out of operations entirely?

Storage and formatStorage and format

The first candidate for offloading storage responsibility is object storage - specifically Amazon S3. As an object storage backend, S3 gives you what local disks can't: fault tolerance and practically unlimited scale, with no storage to manage yourself. It isn't free, though: data-access latency goes up, and you pick up separate costs for API operations. For OLTP workloads that's a dealbreaker. For observability workloads - which are dominated by sequential writes and analytical reads - these trade-offs are often acceptable.

At first glance, this problem is already solved. Loki, for example, uses S3 as its primary storage. But according to Loki's public documentation (v3.6.x) at the time of writing, Loki doesn't remove the stateful layer: ingesters buffer writes in memory and in a WAL on a local disk, and only then flush them to S3. So a stateful component remains, and it has to be operated. On top of that, Loki only covers part of observability - you still have to assemble the other, separate pieces to get a complete stack (traces, metrics, events). Loki does guarantee durability for acknowledged writes through its WAL and replication factor, but that durability rests on local disks and replication, which you have to maintain.

Once you treat S3 as your storage, the next question is what format to store the data in. We need a model that:

works efficiently with an append-only workload;

lets you add new data atomically;

filters data well at large volumes.

Apache Iceberg fits these requirements well. Iceberg stores data as immutable Apache Parquet files and adds them through atomic commits, so readers always see a consistent snapshot. A separate metadata layer prunes files by their statistics before the data itself is ever read, and those statistics can be extended to match an observability filtering profile.

WritingWriting

Iceberg handles storage and reads, but writing to it is more involved. Before a commit, the data has to be gathered into batches, sorted, have its statistics computed, written to Parquet, and committed atomically. That's an asynchronous process. The client, meanwhile, expects a fast acknowledgment that its data has been accepted and won't be lost. This creates a mismatch: writing to Iceberg is slow, whereas ingest has to be fast and must not lose acknowledged data.

The standard move is to put an ingestion layer like Kafka in front of Iceberg. But that just pushes the stateful part into a separate layer again. We tried a Kafka + Flink + OpenTelemetry collector setup, and in our configuration it cost us ~$700–800/month at around 10 MB/s of traffic (storage not included). In other words, the problem doesn't go away - it just moves to another component.

We wanted a simpler design:

accept observability records and persist them to a WAL in S3 ;

acknowledge the client only after the data is safely stored in S3;

asynchronously collect data from the WAL, group it into batches, and write it to Iceberg.

This raises the central question: how do you distribute that processing across workers without standing up a separate coordination layer ? One option is centralized orchestration - a control component that sees the whole pipeline state and makes the decisions: which WAL files to process, who performs the next step, and when a commit can happen. The catch with this approach is that you need leader election...