Topping the BIRD-INTERACT benchmark with SLayer | Motley SLayer v0.9 is live: the open-source semantic layer built for AI agents. View on GitHub →

Product SLayer<br>Motley Cloud

Solutions For builders<br>For data teams

Pricing<br>Security<br>Docs<br>Blog<br>About<br>Book a demo<br>Get started

← All posts Beating the BIRD-INTERACT benchmark: can you get much better than plain Claude by adding a semantic layer?<br>Egor Kraev · June 22, 2026<br>The best way to make a product good is to use it in a wide variety of cases. At Motley, we had already learned a lot from our customers and design partners while building SLayer, the open-source semantic layer for agents. The next natural place to look for that variety was text-to-SQL benchmarks.

TL;DR

With the Claude SDK and SLayer (the open-source semantic layer we developed), an agent reached a 75.3% pass rate on BIRD-INTERACT (a-interact, mini-interact), against the official best of 36.33% on the comparable Lite leaderboard.

The biggest lever was not the semantic layer, but the agent harness . The Claude SDK alone took a raw-SQL agent to 71.7%, while the PydanticAI-based setup yielded a much more modest improvement.

A substantial share of the benchmark’s gold answers are wrong, so we built an annotation agent and report results against both the original and the annotated answers.

SLayer still added an edge on top: 75.3% vs 71.7% against the original gold answers, and 83.7% vs 79.0% against the corrected (“annotated”) ones.

The SLayer ingestion ran fully autonomously for this run, so the numbers would probably be higher in a human-curated setup.

If you’re unfamiliar with SLayer, here’s what it does: sitting between the agent and the database, it holds the definitions the agent queries against: semantic models of your tables and fields (hand-written or auto-ingested from the schema), metric definitions like “net revenue”, and free-text knowledge attached as memories. Instead of writing raw SQL, the agent queries through structured JSON that SLayer compiles into SQL. A text-to-SQL benchmark exercises a large part of that job, answering business questions over an unfamiliar database with a knowledge base attached, so it was a natural way to stress-test SLayer.

So we ran one. With the Claude SDK and SLayer, an agent reached a 75.3% pass rate on BIRD-INTERACT (a-interact, mini-interact), compared to an official best of 36.33% on the comparable Lite leaderboard 1. Boom. Headline number. The reality is more subtle, and the process to get there more interesting.

This post is the first in a series: here we cover how we got there and what the run taught us. Later posts go deeper on the individual lessons.

The benchmark maze

Text-to-SQL benchmarks are a fascinating, messy domain. Even if you just look at the BIRD family, it amounts to almost 20 different sub-benchmarks.

First there is the original BIRD, with its full and dev subsets. Then there is LiveSQLBench, with Base-Full, Base-Lite, Large-Full, and Large-Lite variants (and the occasional SQLite variant thrown in). Then there is BIRD-INTERACT with Full, Lite, and mini-interact variants, and each of those comes in c-interact and a-interact flavors, with separate leaderboards depending on whether the agent was budget-constrained. And then there is BIRD-CRITIC, with its own sub-benchmarks.

Laid out, the family looks like this:

BenchmarkVariantsNotable flavorsBIRD (original)full, devsingle-shot text-to-SQLLiveSQLBenchBase-Full, Base-Lite, Large-Full, Large-Liteadditional SQLite variantBIRD-INTERACTFull, Lite, mini-interact c-interact / a-interact, budget-constrained vs notBIRD-CRITICown sub-benchmarksSQL debugging / critique<br>That is close to 20 distinct leaderboards before you have written a line of code.

We wanted to run SLayer in conditions close to how it would actually be used, which meant the Claude SDK for the agent and Opus as the LLM. That made a single full benchmark run cost on the order of $1000 for the smaller benchmarks, more for the larger ones. Running, and re-running to correct the issues that inevitably arose, across all the flavors was not an option. We had to settle on one to start with.

Why a-interact, mini-interact

We chose BIRD-INTERACT, a-interact flavor . What sets BIRD-INTERACT apart is that it tries to simulate a real working environment rather than a one-shot question. Each database is coupled with a hierarchical knowledge base, metadata files, and a function-driven user_sim simulator, so the agent can solicit clarifications, retrieve knowledge, and recover from execution errors on its own, with no human in the loop. The a-interact flavor takes this furthest: it is an open-ended agentic setting where the model decides for itself when to question the user simulator, when to explore the database, and how many turns to take before submitting a final answer. That is the closest of all the flavors to how agents actually operate in the real world, which is why we picked it.

We chose mini-interact because it is...