Does your AI data assistant know when it does not know?

Silent failure · text-to-SQL

Does your AI data assistant know when it does not know?

Silent failure in AI answers over a data warehouse, and how to find out whether yours has it.

Ask a modern AI data assistant a question about your warehouse and you usually get a clean answer: a number, a sentence of explanation, sometimes the SQL it ran. It looks right. That is exactly the problem.

A real one

The public New York City taxi dataset records a tip on every trip, and how each trip was paid. For cash trips the tip is handed over in the car, so the meter never sees it: the tip column is zero for essentially every cash ride. Ask an assistant “what is the average tip on cash trips” and it will run a correct-looking AVG(tip_amount), return a small positive number, and say nothing else. The query ran. The number is formatted. It is also wrong, because the tip is not recorded for cash at all, and averaging the zeros answers a different question than the one you asked. Nothing in the answer tells you any of that.

That is a silent failure : an answer that runs without error, looks plausible, and is wrong, with no signal attached that it should not be trusted. It is not a crash, which you notice. It is not “the model is inaccurate,” which you can benchmark. It is the failure you do not see, and it is the one that reaches a slide.

It gets worse as models improve

The comforting assumption is that better models fix this. They do not. What improves fastest is not accuracy, it is your trust in the output. A model that is right nine times in ten is more dangerous than one that is right four times in ten, because at nine you stop checking. So the failure that matters is not “the model cannot answer.” It is “the model cannot answer, and answers anyway.”

I measured the two halves of that. On one subscription-style warehouse, across eight languages, three runs each, I put the same set of questions, each with a verified answer, to two frontier models. They got the same number of answers right. The difference was entirely in what they did when they were unsure: one model declined twenty times out of seventy-two; the other declined once, and produced a confident, plausible, wrong number for many of the rest. Same knowledge, opposite disposition. No leaderboard shows that gap, and it is the only thing that decides whether the system is safe to put in front of people who cannot read the SQL underneath it.

The wrong number that looks ordinary

Silent failure is not only about unanswerable questions. It hides in definitions too. Ask “how many customers made a second purchase within thirty days of the first.” The obvious first query counts thirty days from the signup date. The correct one counts from the first purchase. On a real retail warehouse those two queries returned two different integers, both valid SQL, and nothing in either answer saying which reading it used:

The assistant answered95

Verified answer621

Anchored to signup, not first purchase6× too low

A number like that does not announce itself. It goes into a board pack as retention, and the meeting moves on.

Why the tools you already have do not catch it

Three kinds of tooling sit near this problem and none of them closes it.

Your dbt tests check the data: uniqueness, not-null, referential integrity, freshness. They pass on every example above, because none of them is a broken row. The query was fine. It answered the wrong question.

Eval and observability platforms measure the model’s text: latency, cost, consistency, generic hallucination. They cannot decide whether a number is factually right, because that needs the metric definition, the grain, and the data model.

Semantic layers define what a metric means. Useful, and necessary, but nobody independently checks that the definition matches the data, or that the assistant actually used it.

The gap in the middle needs data engineering, metric knowledge, and a way to prove the right answer, all at once. That is where the wrong numbers live.

How to find out whether you have it

You can test the cheapest half yourself, today, for nothing. The open tool quaesitor-zero reads your schema and writes a set of questions the data cannot answer: an attribute that was never recorded, a period outside the data, a segment where the value is structurally absent, like those cash tips. You ask your assistant those questions the way you normally would, paste the answers back, and it scores one thing: did the assistant decline what it could not know, or did it invent something. Nothing is sent anywhere. The tool has no network access, no telemetry, and it never uses a model to judge a model. You read every answer yourself, which is the entire point.

That measures whether the system says “I do not know” when it should. It does not tell you whether the answers it does give are...