Weighing smoke: why GEO dashboards are mostly useless — BTG

Skip to content

Weighing smoke: why GEO dashboards are mostly useless

By Iain, 05 Jul 2026

The promise of a tool that gives you similar search visibility to the pre-agent world is understandably seductive. But before you eagerly hand over your cash for a platform, consider whether an afternoon’s work a month might be a more cost-effective alternative.

I recently wrote about the GEO chimera, the cottage industry promising to optimise your brand into AI answers. My argument was that the levers being sold mostly did not exist. Another part of the industry is selling gauges rather than levers. Estimates of the spend on AI visibility tracking now run past $100 million a year, across hundreds of dashboards charting your brand’s presence in ChatGPT the way Moz once charted your rank in Google. The pitch is falling on the fertile ground of brand marketers standing on the precipice of everything they previously relied upon, staring into the coming void.

There is a post-millennial feel to all of this. The Hollywood screenwriter William Goldman is famously quoted as saying, “nobody knows anything.” The lesser-known part of his quote is, “every time…it’s a guess and, if you’re lucky, an educated one.” That is agentic search visibility in a sentence.

Most current tools run prompts daily, average the runs, report a visibility percentage with a trend line and a confidence interval. The averaging is not the problem per se. The problem is everything it is averaging, because the prompts in the measurement basket are generally steered by the analytics vendor, the query volumes weighting them are modelled fiction, the query surface being sampled is not one a human uses, and nobody has shown that the resulting score predicts a single outcome or conversion that is important to a business. Yet these tools are multiplying like vape shops on a dying high street.

The truth is the things we’re desperately reaching for do not really exist. There are minimal mechanisms to improve your visibility in search agent responses. And there is no way to reliably track and measure that visibility. After a quarter century of SEO and SERPs, this is a bitter pill that many are still unwilling to swallow.

The tasty candy currently being offered as an alternative is what I call precision laundering . Run a hollow measurement often enough that the noise cancels, and out comes a seemingly stable, decimal-pointed number that passes for knowledge. But while averaging buys a number, it cannot buy validity.

The dice cup

In January, Rand Fishkin published a detailed study of the search agent tracking industry. With Patrick O’Donnell of Gumshoe.ai, he recruited 600 volunteers to run twelve brand-recommendation prompts through ChatGPT, Claude and Google’s AI, 2,961 times in total, across categories from chef’s knives to cancer hospitals. Fishkin went in expecting to prove that tracking search agent responses was pointless. The results were stranger than that.

Ask a search agent for brand recommendations a hundred times, and the odds of getting the same list twice are under one in a hundred. The same list in the same order runs closer to one in a thousand. The brands change, the ordering changes, even the number of items changes. Fishkin’s conclusion on position tracking was blunt. Any tool that gives a “ranking position in AI”, he wrote, “is full of baloney.”

This is not one maverick standing alone. An arXiv study of repeated GEO measurements found that the set of brands returned for an identical prompt overlapped only 45-59% between runs, with wide variance, and concluded that single observations of search agent visibility are misleading and that visibility must be treated as a probability across repeated runs. A second paper makes the statistical point directly. Citation metrics from generative engines are random variables, not fixed values. Ahrefs, coming at it sideways, found that Google’s AI Mode and AI Overviews cite different sources 87% of the time for the same query.

This churn cannot be controlled for. A language model does not look up an answer. It generates one token at a time, choosing each word from a range of plausible options with a built-in measure of randomness (known as “temperature”). That randomness is a vital setting because it is what makes the output read as fluent rather than canned. Ask the same question twice, and the answers drift apart on their own.

Add web search and the dice get rolled twice more: once when the model breaks your question into its own set of search queries (referred to as the “fan out”), and again when it picks which of the pages it fetched are worth citing. Even the settings meant to hold steady wobble once a system is live, where requests are batched together, silently routed to whichever copy of the model is free, and swapped onto new versions partway through whatever you thought you were measuring.

So the variance Fishkin measured...