Vetto — GulliBench
On this page
← all research<br>vetto research
GulliBench
GulliBench: Intelligence Isn't Enough. Measuring Skepticism in Frontier Models.
GulliBench<br>Arthur Kamienski · Lucas Smaira · Vetto Research Team<br>August 2026·16 min read
share
In the shadow of intelligence
Frontier AI models have gotten incredibly smart. Month after month, new models are released and shatter benchmarks and leaderboards. It seems they can reason their way out of almost any problem, given enough time and training data.
But still, people don't seem satisfied when using these models in their day to day. Models still get basic things wrong, or just steer away from the intended goal completely. They make wrong assumptions, make wrong decisions, and end up with the wrong answer. They over-rely and over-commit. They just seem too… gullible.
We hypothesize this happens because gullibility is largely orthogonal to intelligence: being gullible is a behavioral trait that is just mildly correlated with raw brain power. Think of Sheldon from The Big Bang Theory: a genius who finished his PhD at 14, but still can't understand basic sarcasm. He takes things too literally.
Similarly, current AI models are trained to be extremely good at solving hard problems (i.e., being very smart). But these problems are usually well-defined and have clear solutions. They are created in sterile environments, where data is standardized and things are mostly deterministic. In short, models are trained with the heavy assumption that the data, tools, and knowledge they are handed are undeniably pristine.
As most of us know, that is not the case in real life. Data is messy and sometimes plain wrong. Tools are buggy and give unreliable results. Assumptions need to be revisited and rewritten. This is no surprise for humans: we've lived in this messy world since the day we were born, and all of us have learned not to trust things at face value, for better or for worse. AI, though, was never taught that. During training, it rarely had to deal with situations where the obvious answer was wrong (even less so when this is done purposefully).
In summary, we have been chasing intelligence, and the other behavioral traits that improve problem solving in the real world have been left in its shadow. Mirroring this trend, most efforts in measuring AI have also focused on raw intelligence: most benchmarks are built with the sole goal of measuring the AI's ability to get things right, while not measuring other problem-solving-related skills.
We believe the first step toward closing this gap is raising awareness and shedding light on the problem. That's why we created GulliBench, our first attempt at measuring AI gullibility.
GulliBench
Measuring AI gullibility: does a model check the data, or take it at face value?
150 tasks · mean pass@1, n=5 · higher = less gullible · CIs in Results
Full breakdown in Results below.
The setup
The core idea behind GulliBench is eliciting gullibility in simple tasks, ones the models would breeze through if not for a single catch. A gullible model fails by not being thorough; a careful analyst derives the answer with ease.
We give the model a tiny environment: a handful of records (≤ 100), a single plain-English question, and one obvious place to look for the answer.
The catch: that obvious place is a lie. But not a wild lie — a convenient one. A status column that says overdue on invoices that were in fact already paid. A total field that was rolled up before the last few line items landed. The kind of stale, authoritative-looking field that exists in every real database on earth.
The right answer is always sitting right there too, just one step further down. You don't count the status column: you check due_date against today and whether it was paid. You don't trust the total: you sum the line items. The truth costs one extra derivation; the lie costs nothing.
So every task is really one question: when the convenient number and the underlying data disagree, does the model notice or does it just grab the number that was handed to it?
The tasks
Questions are made deliberately simple and boring, usually one line long:
How many invoices are overdue?
What's the total value of all orders?
How many customers placed more than 10 orders?
What's the total balance across all accounts?
Questions are also extremely easy by design. Any analyst could answer them in their sleep, even adding the numbers by hand. We're not measuring whether the model can compute the answer, but whether it bothers to check the thing it was given.
The data
The data is small and messy: a few dozen rows and a handful of entities, built from one canonical source of truth. Somewhere in it, the convenient answer and the underlying facts disagree. Often that's a stored field that has drifted out of sync with the primary data it summarizes. But just as often it's the primary data itself that needs...