What the Official MCP Conformance Suite Tells You (and What It Hides) | Barefoot Digital

What the Official MCP Conformance Suite Actually Tells You (and What It Hides)

I build MCP servers for logistics and fintech integrations, and I have the same fear every other MCP author has: a server can look fine in local testing, then break silently under a newer client, a spec revision bump, or a dependency update. The protocol is still young, so the surface area changes fast.

The obvious place to look for guardrails is the official conformance suite: @modelcontextprotocol/conformance v0.1.16. So I pointed it at the smallest real server I had running — a minimal TypeScript SDK server over Streamable HTTP with two or three tools — to see what it would actually report.

What the numbers look like on a normal server

The suite ran roughly 30–32 scenarios. Eight passed. About twenty-two failed.

That sounds catastrophic if you read it quickly. It is not catastrophic. Most failures were the suite testing capabilities my server had never declared.

For example, I do not expose the resources capability, so every resources-* scenario failed — not skipped, failed. Same for prompts-*: no prompts declared, all prompt scenarios failed. There were also fixture-style scenarios for content types normal tools simply do not return: image content, audio content, embedded resources, sampling, elicitation, logging. My server is not broken; it just does not implement those features. The suite has no capability awareness, so it runs every scenario regardless of what the server says it supports.

The headline result makes a valid production server look roughly 75% broken when it is fine.

A representative failure read something like this in the per-scenario output:

resources/list: FAILED<br>Expected: resource list contains fixtures<br>Got: method not found / capability not supported<br>Spec ref: https://spec.modelcontextprotocol.io/spec/2025-03-26/server/resources/

I am reconstructing the message from memory; the exact wording is whatever the suite printed. The important part is the signal shape: the failure is real against the spec, but irrelevant to my server's declared surface.

Mechanical gotchas that make the report worse

Two things about the CLI output make the noise harder to interpret.

First, the process exits 0 even when scenarios fail. You cannot use the exit code in CI to fail a build. The real signal is in the per-scenario checks.json files the suite writes out.

Second, the "Total" line counts individual checks, not scenarios. So the numbers look larger and more alarming than the scenario count. If you expect 30 scenario results and see a total in the hundreds, you think something is exploding. It is just a different unit of counting.

What it cannot test

The CLI is HTTP-only for the transport in the version I used. I got no stdio coverage from it, so any server primarily meant for local stdio clients would need a different harness.

Also, the newest supported spec version in the suite lagged the latest spec draft at the time I ran it. That is not a complaint — keeping a test suite current with a moving draft is hard — but it matters if you are trying to validate against bleeding-edge behavior.

What it does well

It is worth being fair. The suite has genuinely useful parts.

Per-check spec references. When a check fails, it links to the exact spec section. That is excellent DX; you can go from failure to spec line quickly.

JSON output mode. The machine-readable output is clean enough to wrap in your own reporting.

GitHub Action. You can drop it into a workflow without inventing your own runner.

Expected-failures baseline. There is a YAML file you can use to gate regressions: declare the failures you expect, and the suite flags only new ones. This is probably the right way to use the tool against a server that does not implement the whole kitchen sink.

Failure-message clarity. The messages are readable and point at the right contract.

For SDK or server authors who need to ask "does my implementation conform to the MCP protocol spec?" the suite is a reasonable starting point. That is its question.

The bug it masked

Here is the part that surprised me. My demo server had a classic lifecycle bug: it shared a single server instance across sessions. The first scenario passed, then every subsequent scenario started returning fetch failed errors. The suite kept going and reported a cascade of transport failures, but it never surfaced a clear "your server crashed mid-run" signal. A human has to recognize that the first failure was the real one and everything after it was collateral.

That is a common real-world failure mode — state leakage between clients — and the conformance suite made it look like a long list of unrelated protocol failures instead of one server crash. That matters because the suite is positioned as a correctness check, but it can obscure operational problems that production monitoring would catch...