Kubernetes SLOs with Mesh Metrics: 30-Day Implementation Playbook

Skip to main content

Pricing<br>Resources

Academy

About

Contact usDocs

Get Service Mesh Certified with Buoyant.<br>Enroll now!

Blog home<br>SLOs from Mesh Metrics: A 30-Day Implementation Playbook

Linkerd Production Readiness Pre-Launch Checklist

Download checklist

Heading

Relevant articles<br>Linkerd vs Istio

eBPF, Sidecars, and the<br>Future of The Service Mesh

A Kubernetes engineer’s guide to mTLS

Mesut Oezdil

August 5, 2026

Linkerd

If you run Kubernetes in production, you’ve probably heard the term Service Level Objective (SLO) in countless planning meetings over the last 2 years. But have you ever defined one? When I have, it was on a gut feeling: I never had a metric I trusted enough to defend when the page went off at 2 am, and the thresholds were never really agreed on. The great news is, if your services are meshed, you already have the metrics.<br>This is a hands-on playbook. You’ll learn which five SLOs to define first for east-west service traffic and how Linkerd’s golden metrics map to each one. You’ll get the exact PromQL (the Prometheus query language) for every one of them, how to alert on them without paging yourself into oblivion, and the order to roll it all out over your first month.<br>Everything here was run on a live single-node K3s v1.34.6 cluster with Linkerd edge-26.6.3, the OpenTelemetry Demo as the workload, and a bring-your-own Prometheus and Grafana. Every query and every number below came off that cluster.<br>What an SLO actually is (and isn’t)<br>Let’s start with a Service Level Indicator (SLI). An SLI is a number that measures one service health dimension, like the fraction of requests that succeed. A Service Level Objective (SLO) is a target for that number over a window, like 99.9% success over 30 days. A Service Level Agreement (SLA) is an SLO with financial penalties attached to it, like a cloud provider owing you service credits when monthly uptime drops below 99.9%.<br>The job of an SLO is narrow: it gives the on-call engineer a number to defend and a threshold to page on. That’s it. It is not a dashboard with every metric you have, and it is not an aspiration to 100%. An SLO you can’t measure is a wish, and an SLO you set to 100% is a pager that never sleeps.<br>Where the metric comes from<br>The reason my team struggled to define an SLO was the practical wall right behind the concept. We wanted to track availability, but we didn’t have a metric for it. Instrumenting every service to emit a clean success-rate counter is a project no one is looking forward to.<br>Meshing removes that work for east-west traffic. The moment a workload is meshed, its Linkerd microproxy starts recording every request it handles, and 2 metrics carry everything the SLOs need:<br>response_total, a counter of responses with a classification label of success or failure, which gives you request rate and error rate.<br>response_latency_ms_bucket, a latency histogram, which gives you p50, p95, and p99.

The data flow. A meshed pod runs your app container next to a linkerd-proxy sidecar. The proxy exposes response_total and response_latency_ms_bucket on its admin port. Prometheus scrapes the proxy through a PodMonitor, recording rules turn the raw counters into SLIs, and those SLIs feed both the Grafana panels and the burn-rate alert.‍<br>Here’s a single inbound response_total series from orders-api on the live cluster, with a few labels trimmed for readability:<br>response_total{<br>workload="orders-api", namespace="slo-lab", direction="inbound",<br>classification="failure", status_code="500", tls="true",<br>srv_name="all-unauthenticated", target_port="80"<br>}Two things to keep in mind: The tls="true" is mutual TLS between meshed pods, which you get for free. Meanwhile classification="failure", backed by the status_code="500", is the load-bearing label: the proxy already decided this was a failed request, so your availability SLI is one division away, and the collection already happened.<br>There’s a second benefit, but less obvious than the dashboard. A number you trust changes the conversations you have. When you can point at a real success rate, you can defend a rollback in a review, push back on a latency target nobody actually measured, and make the case for reliability work with data instead of a hunch. The metric is the argument.<br>If you want a full overview of what mesh-derived metrics do and don’t cover versus your application’s own telemetry, take a look at my earlier post, OTel and mesh-derived metrics. This post takes the metrics as given and turns them into SLOs.

Prometheus table view of ‘response_total{workload="orders-api"}’, showing the success and failure counters with ‘status_code’ and ‘tls="true"’.The 5 SLOs to ship first<br>Five SLOs cover the questions an on-call engineer asks. For each one, we cover what it measures, the exact PromQL, a starting threshold, and why that threshold. Every query groups by a workload label that we relabel onto the proxy metrics, so...