Self-Service Analytics Is Not Self-Service. But Don’t Tell Anyone

Data Analysis Journal

SubscribeSign in

Self-Service Analytics Is Not Self-Service. But Don’t Tell Anyone - Issue 329<br>What Anthropic reveals about the data infrastructure, expertise, and human judgment behind successful self-service analytics.

Olga Berezovsky<br>Aug 19, 2026

Share

Welcome to the Data Analysis Journal, a weekly newsletter about data science and analytics.

Subscribe

A quick note: I have a 20% discount for my readers for the upcoming dbt Summit in Las Vegas on Sep 15–18! If you work with dbt, analytics, or ETL, this is a must-attend event for anyone working with data. Expect breakout sessions, hands-on labs, and time to connect with fellow analytics engineers.<br>Hope to see you there! Use code Ext-OlgaB at checkout.

In early June, Anthropic published a guide - How Anthropic enables self-service data analytics with Claude, and the analytics world was never the same.<br>For a few weeks, the post was everywhere - in Slack conversations, internal discussions, and all over my LinkedIn feed.<br>I know it’s been 2 months since that article was published, and many people have probably forgotten about it and moved on. Well, I haven’t.<br>This was one of those rare historic moments when an AI lab officially confirmed something data teams have been saying for so long: you cannot simply point a model at a warehouse and expect reliable answers.<br>It admitted that out of the box Claude never achieved more than 21% accuracy in answering analytical questions, even after verifying Claude had read all the materials and content available. In another test, even after providing Claude with all pre-existing analytical content - all the SQL queries, notebooks, dashboards, etc, it improved accuracy by only 1%.<br>So what did? Data governance, encoding analytical workflows and business context as skills and human validation. Anthropic keeps human sign-off on anything reaching leadership and tests against golden datasets.<br>It basically confirmed you need a village of analysts to make self-service work and keep it going.<br>After yet another piece of LinkedIn cringe that made very little sense, I decided to feature an Anthropic article today to (a) celebrate its unusual transparency and (b) level set expectations for where analytics stands and why so many self-service initiatives will likely fail.<br>Feel free to forward this to your CEO if someone still thinks your data team is slow and behind.

What Anthropic proved

Anthropic clearly said - connecting Claude directly to a data warehouse through MCP gives it access to the data, but access alone does not make its answers trusted or reliable . Claude still does not know which of several similar fields contains the approved metric, which business rules it should apply, or whether a table was retired last quarter. MCP does not teach Claude how the company defines revenue, customers, activation, or churn.<br>Anthropic’s data shows that typically models perform only at 21% accuracy when first introduced to a new business domain. It eventually can get improved up to 95%, but only after a senior data team organized the business context, corrected mistakes, and continuously tested the system against a known set of questions. That means, 95% is not the expected accuracy of AI analytics out of the box. It takes a village to get there .<br>In Anthropic’s setup, the semantic layer provides trusted definitions the AI needs before it queries the data. The problem with that is most companies do not have a semantic layer, and building one can take ages. And if you already have one, supporting, maintaining, testing, and verifying it also takes resources.

How Anthropic enables self-service data analytics with Claude<br>Connecting an AI model directly to the warehouse (and this is what most of us are doing now) is not enough. Reliable analytics still requires shared definitions, human review, testing, and ongoing maintenance.<br>I love this summary from Ayush Gupta, Genloop CEO:<br>Anthropic’s self-service analytics post is the clearest public case any AI lab has made that you cannot point a model at a warehouse and get reliable answers. Five thousand words in, they say it directly: pointing Claude at a warehouse “can create a false sense of precision.” The rest of the post is the infrastructure required to undo that. It is the most honest public account of what production-grade agentic analytics actually demands. It is also a near-perfect illustration of why most companies will not be able to follow the same path.

The team behind “lean” self-serve analytics

Let me start with this: the company that wrote about how it successfully implemented self-serve analytics had at least 30 full-time data scientists and analysts (plus an unknown number of data and ML vendors). And it is hiring more, clearly aiming to expand its analytics team.<br>That may not seem like a lot for a large company. At the time, those 30 roles represented approximately 2% of Anthropic’s...