v0Log inSign up

← BlogIf you run an OpenAI-compatible gateway long enough, you drown in traces that look different and are the same program. Same JSON schema. Same “return the answer span.” Same “normalize this issuer name.” Different companies, amounts, and sentences — identical contracts. That is the bet behind Seldon: find those programs, then compile the repetitive ones into cheaper deterministic pipelines. This post is about the first half of that loop — Trace Audit — and what we learned when we stress-tested it at scale.

Compilation only works if we can answer a precise question first:

*:first-child]:mt-0 [&>*:last-child]:mb-0"><br>Which traces are the same replaceable program — not merely the same topic?

We built that answer for Trace Audit v0, measured it on 12,517 traces, and kept the mistakes that taught us more than the models did. Audit is only half the story: once clusters exist, Seldon treats them as specs for a restricted form of program synthesis . We sketch that compilation step below; a dedicated follow-up will go deep on the synthesizer. For the framing we already published, see Program synthesis, and why compiling LLM calls into ETL is one.

TL;DR. Hard contract blocks first, then block-local DBSCAN on a short deterministic feature string (embedding_text). We hit 1.000 / 0.9999 / 1.000 pair precision / recall / purity — with about $0 model cost. Live summarizers (GPT-4.1, GPT-5.6-luna, Phi-4) did not help clustering; they hurt purity. Use LLMs for labels and dossiers after clustering, not as the clustering signal.

Why this matters if you ship LLM products

Most teams start with a frontier model because it is the fastest way to prototype extraction, classification, and normalization. Then volume shows up, and a large share of calls turn into the same shape over and over:

document / ticket / record<br>→ same prompt template<br>→ same response_format / output shape<br>→ different payloads

You are paying reasoning prices for ETL wearing a prompt — often without lineage, without a ranked view of which workflows dominate spend, and without a safe path off the model.

Seldon’s loop is deliberately boring:

See it — gateway traffic → contract-compatible clusters → workflow map ranked by savings. (this post)

Compile it — search a typed operator library for a cheaper pipeline that reproduces the cluster. (teased below; deeper post coming)

Trust it — shadow validation on your traces; automatic LLM fallback; demote when quality slips.

Trace Audit is step one. Get the buckets wrong and everything downstream is fiction.

The wrong instinct (and why it’s everywhere)

The fashionable pipeline for “understand my LLM traffic” looks like this:

Summarize-then-embed clustering pipelinePretty-print the traceLLM summaryEmbeddingCluster by similarity<br>It is intuitive. Summaries are readable. Embeddings are familiar. Product-analytics stacks popularized the pattern.

For a compiler , it is the wrong objective function.

We do not want “these traces talk about invoices.”

We want “these traces are the same compilable extraction program .”

Trace ATrace Bresponse_format: json_schemafree-form proseoutput: {vendor, invoice_number, total}output: a paragraph→ ETL / structured-extraction candidate→ leave on the LLM

Topic similarity merges them. A compiler must not.

Conversely, two span-extraction calls with different entities should cluster: same roles, same mode, same output shape — different payloads.

*:first-child]:mt-0 [&>*:last-child]:mb-0"><br>Program identity ≠ document similarity.

The bigger picture: audit → compile → runtime

Before we zoom into clustering, here is where Audit sits in the product.

Seldon stack from app to hybrid runtimeApp calls OpenAI-compatible base_urlGateway: route, log, cost analytics (dayone)Trace Audit: clusters, families A–G, ROImap (this post)Compiler: cluster = spec, typed operators =search, held-out traces = verifierHybrid runtime: cheap compiled path whenconfident; frontier LLM fallback<br>Day one you change base_url and get a normal gateway. Over time, repeated traces author free I/O specs. Audit finds the same program hiding under different payloads. The compiler tries to synthesize a cheaper implementation of that program. Runtime keeps the LLM as a safety net — compilation is partial on purpose.

That framing is deliberate program synthesis, not “ask an LLM to write a pipeline and hope”:

Classical synthesis ideaHow Seldon uses itSpecThe clustered traces (programming-by-example)Search spaceFixed, typed operator library (families A–G + adapters)Proposal priorFrontier model at compile time only — suggests decompositionsAcceptanceEmpirical: match the LLM oracle above threshold on held-out tracesFailure modeRefuse to compile; leave the call on the frontier model

Three departures from textbook synthesis matter in production: acceptance is statistical , not a proof; the LLM oracle is noisy (sometimes the compiled path is more correct on deterministic families); and the I/O contract...