. -->

How we build and evaluate our MCP server for SRE agents | ClickHouse<br>Skip to content

Open searchOpen region selectorEnglish<br>Japanese<br>Korean<br>Chinese<br>French<br>Spanish<br>Portuguese<br>Arabic

48.9kSign inGet Started

->Scroll to top<br>BackBlog<br>Engineering<br>Copy pageCopied!More actionsView as Markdown Open this page in Markdown<br>Open in ChatGPT Ask questions about this page<br>Open in Claude Ask questions about this page<br>Open in v0 Ask questions about this page

How we build and evaluate our MCP server for SRE agents

Brandon Pereira<br>Jul 22, 2026 · 19 minutes read

Last month at Open House in San Francisco, we introduced the ClickStack MCP server, a set of purpose-built observability tools that give AI agents structured, high-level primitives for investigating production incidents instead of hand-assembling raw SQL. On stage, we shared early results from benchmarking the ClickStack MCP against a generic ClickHouse SQL interface: 18% more accurate root cause and remediation outcomes, 26% fewer tool calls, and 2.4x more consistent results run over run.

Those numbers prompted plenty of questions about how we measured them, so this post explains exactly what we did.

Shipping MCP (Model Context Protocol) tools that agents use to investigate production incidents introduces a new kind of risk. The same question asked twice doesn't always produce the same answer, and a refactored tool, a renamed parameter, or a change in query logic can quietly degrade investigation quality in ways you won't catch without a structured way to measure them.

We needed to answer two things. First, whether each new version of the MCP produced better investigative outcomes than the last. Second, whether the MCP actually outperformed an agent with direct SQL access to ClickHouse. To do that, we built hdx-evals, a reproducible benchmarking framework that seeds identical synthetic telemetry, runs Claude agents against each configuration, and scores the results blindly across implementations. Because the framework runs every combination of MCP and model, a useful side benefit fell out of it: when a new model ships, we can point hdx-evals at it and see how it performs on the same scenarios before deciding what to run in production.

hdx-evals is open source and lives in the HyperDX repository. If you want to follow along with the code, see github.com/hyperdxio/hyperdx/tree/main/packages/hdx-eval.

ClickStack MCP vs ClickHouse MCP #

Before we turn to the benchmark scenarios, we’ll briefly revisit why we built a dedicated ClickStack MCP alongside the ClickHouse MCP. If you’re already familiar with the distinction, you can skip ahead to the incident scenarios.

The ClickHouse MCP gives agents direct access to ClickHouse through SQL. This is flexible, but it leaves the model to discover the schema, construct each query, and assemble multi-step investigation workflows itself. In observability investigations, this can lead to extra tool calls, inefficient queries, high-cardinality results, and inconsistent approaches between runs.

The ClickStack MCP instead provides higher-level semantic tools for common observability tasks, such as finding recurring event patterns, comparing time windows, identifying outliers, and moving between logs and traces. These tools still execute SQL underneath, but they package the query logic and return structured results that are easier for an agent to use. The aim is to improve accuracy and consistency while reducing tool calls and common query errors. The scenarios below test whether those advantages hold across different types of investigation.

Incident scenarios #

A benchmark is only as good as the incidents it tests against. If they're too clean, every agent looks brilliant. If they're too unrealistic, the results tell you nothing about how the MCP performs in production.

So each hdx-evals scenario simulates a real incident: tens of millions of synthetic logs and spans, seeded deterministically so every run starts from identical data, with a planted anomaly buried inside realistic noise, plus distractors: secondary issues timed and worded to look like the real problem. An agent that jumps to the first error it sees should fail. An agent that reasons through the data should not.

The data is completely synthetic rather than derived from the OpenTelemetry Demo, existing products, or recorded telemetry. Public datasets often contain well-known and documented failure modes that may already appear in a model’s training data. Reusing them could allow an agent to recognize a familiar incident instead of investigating it through the available tools, so generating original scenarios helps us test the agent’s investigative ability rather than its prior knowledge.

Traces and logs are generated deterministically in TypeScript from a seeded Pseudo-Random Number Generator (PRNG) and written directly into ClickHouse tables that mirror ClickStack's real OTel schema. Topology varies per scenario, from a single api-server up to 25...