The Observability Pain Cycle Observability The Observability Pain Cycle<br>Nikola Petkovic · Aug 7, 2026 · 7 min<br>read

If you used an observability product, you have probably experienced some version of this. The vendor asks you to ingest everything up front, and charge you the moment this huge volume of telemetry is ingested. Only then do you get a chance to explore that data and get some value from a small portion of it, when someone builds a dashboard, writes an alert, or goes exploring in the middle of an incident. So the spend is at full height from the first minute, regardless of when and to what extent you start to use it later. Fashions changed along the way: new protocols appeared (OTel), novel telemetry databases built on object storage and columnar layouts, AI happened. But the store-everything-up-front philosophy stayed firmly in place, protecting the vendor’s interest.

As a result, a large amount of telemetry noise sits around, eating your bill and slowing down your queries. Whether you are getting enough value for your spend is a second-order concern for the vendor. Their main worry is that the invoice doesn’t cross your pain point. And when it does, they hand you the tools to analyze and cut some telemetry. A couple of months later the cycle repeats: costs climb, and you are back to hunting for telemetry to trim. It is always the pain of a high bill that triggers the trimming exercise. This pattern is so ubiquitous that it deserves a name: the Observability Pain Cycle . It is the natural consequence of the store-everything-up-front philosophy that mainstream observability platforms were built on, and have done everything in their power to keep it that way.

How store-everything-up-front became the default

The model was reasonable when it was introduced. On physical servers running monoliths, telemetry was low-volume and hand-instrumented. The challenge was to instrument and collect, not to store and query. “Send us all your telemetry and query what you need” was sound: storing some unused data cost a bit more, but it was acceptable.

A couple of things changed around a decade ago. Cloud, containers, microservices, and distributed tracing drastically increased the telemetry volume. And producing telemetry stopped being a concern. On the contrary, it became very easy to emit. Auto-instrumentation, built-in library metrics, exporters, and eBPF help you emit telemetry, often without writing instrumentation code at all. The store-everything-up-front model was never revisited. However, the cost became a problem impossible to ignore. So the vendors built some tools to “help” you when the bill starts to hurt.

The direct consequence of store-everything-up-front is a large amount of noise: telemetry you never asked for. It brings no real value, and it serves mainly that the vendor gets to charge you for it, telling you that “you never know whether you might need it eventually”. But the volume is not a joke anymore, an average vanilla Kubernetes cluster produces almost 100,000 time series out of the box.

How it works

Let’s first take a look at what the system architecture looks like and how data flows through it. The architecture is sane, but the vendor-favoring defaults result in the customer pain points (shown in red).

The Observability Pain Cycle<br>High cost drives the telemetry filtering.

Workloads apps & services Agent Filter forward ALL by default Observability Platform Store Everything Upfront Analysis Tools What costs this much? drives $$$$$ Store all telemetry Dashboards slow queries Alerts<br>Telemetry data flow Control flow

This is how the story typically unfolds:

You onboard by following the vendor’s documented path. Enable the integrations, turn on auto-instrumentation, scrape the whole cluster. The agent sees every metric, log line, and span at the source, which would be the right place to decide what to keep. But it ships with pass-through as the default. And if you flip it, there is a punishment: you can no longer put together dashboards and alerts, or browse through the telemetry that is available at the source.

Everything gets stored, only a small part gets used. All telemetry lands in the backend, while dashboards and alerts (once they are created some time later) read a small, stable subset of it. Nobody tracks which stored series feed a panel or an alert, which have never been read, or which data would bring no practical value in possible future investigations.

The bill becomes a visible pain point that no longer stays under the radar. An engineer is assigned to find out where the money went, and always finds the same things: metrics nobody reads, debug logs kept at full retention, cardinality bombs, namespaces onboarded and forgotten.

The trimming is coarse. Drop these log levels, sample these traces at 10%, exclude these metric prefixes. In the absence of a clear link between stored signals and the value they bring, hunting for what to trim becomes a hard and error-prone...