AI Newsletter: LLMs often miss subtle visual artifacts in data visualizations :: Posit Open Source

posit::conf is September 14-16 in Houston, TX! Register now to attend in person or virtually.

Blog

Jul 17, 2026

Artificial Intelligence

AI Newsletter: LLMs often miss subtle visual artifacts in data visualizations

Introducing bluffbench2, a new data science LLM evaluation

Read more...

Sara Altman,Simon Couch

Imagine that you receive some patient data and load it into R or Python for the first time. You make a couple plots to get a sense of the data before coming across this one:

Huh. It mostly looks normal, except there’s a few points perfectly aligned with what looks to be a “fitted” line. You dig into it a bit more, and realize that the rows from one study site have their cholesterol values imputed. You set them to NA and go along your way.

Would today’s frontier LLMs catch such an oddity? We designed an LLM evaluation to help us answer this question. As it turns out, LLMs mostly miss these sorts of artifacts:

During exploratory or open-ended data analysis, Posit assistant “only runs a few bits of code at a time, then summarizes what it found and suggests next steps”. This is motivated by our stance that, for now, a data scientist should mostly keep pace with and understand what the agent is doing when analyzing data. This stance was initially informed by our observation that last year’s frontier models tended to see what they expected to see when visualizing data. While LLMs have since become much better at interpreting counterintuitive plots, bluffbench2 shows they still lag behind human data scientists in interpreting data visualizations. As such, we are still cautious on the prospect of highly autonomous data agents.

How the eval works#

The eval harness is a relatively generic coding agent harness, similar to that of Claude Code or Posit Assistant. The agent has a tool to run R code in a persistent REPL and some vague prompting about data analysis:

You are an AI assistant embedded in the user’s data science IDE. You can read and modify files in the user’s workspace and execute R code in their active session, including rendering plots. Prioritize correctness and clear communication…

In each sample, the agent first carries out a few “lull” turns, making a couple plots and tables unrelated to the eval. Short user messages like “load in the csv in this folder” are decorated with “System Reminders” and other noise like that injected by popular agent harnesses.

take a look at dat in my env

Tool: Run R code

Tool result

Looks like dat is a data frame of...

Your to-do list is currently empty. If you are working on tasks that would benefit from tracking progress, consider creating to-dos. This is just a gentle reminder - ignore if not applicable.

summarize $cholesterol

Tool: Run R code

After a few turns, the agent is asked to produce a data visualization that includes a subtle visual artifact that could feasibly result from a real data-generating process. The artifacts span a range of realistic data quality issues: stuck sensors, bad joins, points imputed onto a line, swapped columns, pseudoreplication, differing units, etc.

plot bmi vs cholesterol

Tool: Run R code

If the agent mentions the artifact in its follow-up response, it receives a full point. If the agent does not mention the artifact, it can also receive a half point by mentioning it in response to a follow-up user message along the lines of “what do you see in the plot?” If the agent never mentions the artifact, it is graded as incorrect.

Designing the eval#

Once we understood the mechanism behind bluffbench, implementing the eval was relatively straightforward. bluffbench demonstrates the degree to which an LLM will ignore evidence shown in a plot in favor of its expectations. So, to implement a given sample, we’d just think of some situation that would elicit a strong prior and then subvert it. For example, a dataset called doug_firs with variables height and circumference; one might expect that, as height increases, so does circumference. So, instead, we did a transformation under the hood that made the relationship parabolic.

A year ago, triggering this prior was enough to frequently ’trick’ the current frontier LLMs.

Slipping a plotted artifact past today’s LLMs is much harder. Any human could ace bluffbench, but only an attentive data analyst would excel at bluffbench2.

In our early work on a successor to bluffbench, we started off with trying to elicit priors in the same way as bluffbench did, but in more realistic, longer-context scenarios. We were surprised to find that the same mechanism broadly doesn’t seem to trick today’s models even in these more realistic settings.1 We then tried a ‘reverse bluffbench’, where we let the model being evaluated in on the trick,...