Text-to-SQL Fails Because of Your Warehouse, Not the Model :: mamonas.devm.
mamonas.devJul 15, 2026tech6 min read<br>Text-to-SQL Fails Because of Your Warehouse, Not the Model<br>I’ve been experimenting with pointing LLMs at databases through MCP servers and asking them questions. The results followed a pattern. First, the model would simply make up tables and columns that didn’t exist, then write plausible-looking SQL against them.<br>Giving it tools to investigate the schema fixed the first failure: it could list tables and describe columns, and it stopped making up structure it could now look up. But a subtler failure was left behind. When it wasn’t clear what a column meant, the model didn’t ask and didn’t flag it. It picked an interpretation and ran with it.<br>That second failure is the one worth writing about. The models are decent at this point, and I don’t think a better one is what’s missing. What’s missing is on our side: the context that tells the model what the data actually means.<br>The Benchmark Cliff⌗<br>For years, text-to-SQL progress was measured on Spider, an academic benchmark with clean schemas, a handful of tables per database, and readable column names. The top system passed 91% execution accuracy and the leaderboard was frozen in early 2024, effectively solved.<br>Then the same lab released Spider 2.0, built from real enterprise warehouses instead: BigQuery and Snowflake databases that often exceed 1,000 columns, where answers can take over 100 lines of SQL. GPT-4o scored 86.6% on academic Spider and 10.1% on Spider 2.0, the same model against real schemas.<br>BIRD, another major benchmark, sampled 500 of ChatGPT’s failures and categorized them: 41.6% linked to the wrong tables or columns, and another 40.8% misunderstood the database contents or generated schema items that don’t exist. Over 80% of failures are schema grounding. SQL syntax barely registers as a category. The failures aren’t in the SQL. They’re in the model’s picture of what the database means.<br>The Warehouse You’d Actually Point It At⌗<br>Ask two teams at the same company for last quarter’s revenue and you can get two different numbers, both correct by their own definition. One excludes returns, one doesn’t. One counts bookings, one counts recognized revenue. The metric name is shared; the logic behind it is not. That logic usually isn’t in the warehouse at all, it’s in a Slack thread from 2023 and the head of the analyst who built the dashboard.<br>The physical layer is no better. A warehouse that has been through a few years of team changes looks something like this:<br>SHOW TABLES LIKE 'customers%';<br>-- customers<br>-- customers_v2<br>-- customers_final
The migration behind customers_v2 got most of the way done before priorities shifted. Columns are the same story: amt and amt_2 sit next to each other, no comment on either. A model with schema access can see all of this. It cannot see which table is canonical, because nobody wrote that down anywhere a query can reach.<br>A Hacker News commenter describing their experience with Databricks Genie summed up the trap: “I have hundreds of tables designed by several different teams. I do have decent documentation on the tables but if I had a nice, organized data model I wouldn’t need an AI assistant.” Text-to-SQL works best on exactly the warehouses that need it least.<br>Same Model, Different Context⌗<br>The evidence I trust most comes from cases where the model stays fixed and only the context changes.<br>Pinterest built text-to-SQL internally and found the bottleneck wasn’t SQL generation, it was finding the right table among hundreds of thousands. Adding table documentation to their search index moved the right-table hit rate from about 40% to 90%, with the model unchanged.<br>BIRD ships each question with an optional sentence of human-written context about the data. That one sentence lifts GPT-4 by about 20 points. The part I find most convincing: it lifts human accuracy by the same 20 points, from 72% to 93%. When missing context hurts experts as much as it hurts the model, the problem isn’t intelligence. The information just isn’t there.<br>Even the vendor numbers say this. Snowflake’s engineering blog reports raw GPT-4o at 51% on their real-world evaluation, and 90%+ once a human hand-builds a semantic model describing the data. They’re selling Cortex Analyst, so it’s a vendor benchmark. But what it concedes is the point: the model alone is a coin flip, and the entire difference is metadata someone wrote.<br>The Agents Got Good, Though⌗<br>The counterargument has real numbers behind it. Spider 2.0 scores went from around 10% in late 2024 to claimed 90%+ on its Snowflake track by mid-2026, and dbt’s benchmark measured raw text-to-SQL accuracy nearly doubling between 2023 and 2026 from model progress alone, 32.7% to 64.5%. If the warehouse were the whole...