Of Hammers and Nails: What AI Can and Cannot Do for a Data Analyst
Adam Cassar
SubscribeSign in
Of Hammers and Nails: What AI Can and Cannot Do for a Data Analyst<br>The LLMs are genuinely useful. They are also given more credit than they're due.
Adam Cassar<br>May 21, 2026
Share
Every few years, a new technology arrives with the same promise: this one will transform the organisation, eliminate the grunt work, and make whole categories of expensive people redundant. AI, and the large language models driving its current moment, is the latest. In data and analytics, the claims have been particularly bold — the well-prompted chatbot will soon replace the analyst, we are told. Having spent the past year rolling AI tooling out across a large organisation, the reality is more interesting, and more mixed, than that.<br>Where it genuinely helps is in writing code<br>Start with what works, because something genuinely does. AI tools have made writing code significantly faster. That matters more than it might sound. In teams that haven’t yet built mature data assets (data models), coding and data preparation is the job — easily 80 to 90 percent of what analysts actually spend their time on. Anything that speeds this up is a meaningful productivity gain.<br>Why does this matter so much? Because the single biggest factor separating a data team that delivers real value from one that simply costs money is the quality of its underlying data assets. Teams with clean, well-structured data models are dramatically more productive than those without. AI tooling can get organisations to that point faster, and that is a genuine advantage.<br>AI needs a lot of prep work on your data, and even then gets things wrong too often<br>The bigger claims for AI in analytics centre on its ability to answer ad hoc data questions — essentially, a tireless analyst available around the clock, queryable in plain English. The idea is appealing. The reality is not yet there.<br>The issue isn’t that AI can’t get the right answer. Sometimes it does, and well. The problem is consistency. Without very carefully built data architecture — clean models, good documentation, detailed instructions — the error rate is extremely high. And even with all of that in place, published results from companies that have done this work carefully show accuracy rates around 86 percent. That sounds reasonable until you think about what it means in practice - one in six answers is wrong. And you probably don’t know which ones. Would you be comfortable making a serious business decision knowing there was a 1/6 chance (at best) it was off? A human analyst getting things wrong that often would be a serious performance problem. The technology, held to the same standard, doesn’t pass.<br>It’s also worth noting that we already have a technology which answers business metric questions with near-perfect accuracy and never hallucinates. It’s called a dashboard. A well-built dashboard suite covers most of the questions stakeholders actually ask, reliably and cheaply. The case for replacing it with an AI query interface is weaker than the hype suggests.<br>Giving LLMs enough context to analyse metric changes well is often more laborious than doing the work yourself<br>This is arguably where AI disappoints most, because the use case sounds so compelling. The typical ask is something like: produce a report on how X performed and why.<br>Getting numbers is straightforward in a mature data setup — they’re already in your dashboards, or you can query them quickly. The hard part is the commentary: explaining why the numbers moved. In practice, that means sitting in meetings where teams discuss what drove their results, and chasing down the people who actually know. That knowledge lives in people’s heads, not in databases.<br>For AI to write that commentary usefully, someone has to first write down all that context in a form the model can use. And the effort of doing that turns out to be roughly similar to just writing the report yourself. Without automated ways of capturing that context the time saved is surprisingly small. Things like integrations into team Slack channels and structured meeting notes can help, but it’s both imperfect and still suffers from the hallucination problem.<br>What you often get instead is work that’s slightly better rather than significantly faster. The models do often spot things that humans miss, even if the overall error rate is meaningful, and it handles the mechanical parts of the drafting well. Those are genuine benefits. But the transformative reduction in analyst workload that has been predicted? That hasn’t materialised — at least not yet.<br>The bottom line<br>AI tooling has made real contributions to data work: faster code writing, quicker data asset development, and a useful drafting assistant for analytical writing. These gains are worth having.<br>What it hasn’t done is replace the core of what good analysis requires — judgement, context, institutional knowledge, and the ability to tell a...