Where’s the holistic AI productivity data? – Rachel Andrew
Skip to the content
Hello
About
Blog
Now
Books
Writing
Projects
Contact me
Search
Bluesky
Search for:
Search for:
For most of my career I ran a very small company. When you run a tiny company your resources (both time and money) are limited, and you want to use them on the things that will have the most impact. You have to quickly stop doing things that aren’t cost-effective, to avoid "throwing good money after bad". Ideally, you do a small trial of something new and measure the results before rolling it out more widely, to avoid going all in on something untested. As tech news starts to publish stories about how companies are realising that AI costs more than the humans it was supposed to replace, I’m wondering why it took them so long to figure this out. I’ve spent the last two years watching companies large and small diving headlong into AI. Rarely do I see an attempt to measure the actual costs, financial and otherwise, of that decision.
It’s certainly possible, for a skilled person, to speed up certain processes while also maintaining quality with the use of an agent. I’ve a number of examples that have been successful, and enabled improvements across content sets that would have been hard to justify the work on otherwise.
However, as I document this work I realise how the success of it relies on the things I know. I can spot when the AI tool goes off track, I review its work in the way I’d review the work of a very junior writer. I couldn’t just hand this stuff off to anyone, and them be able to replicate what I can do in terms of the quality of the end result. When you do that, what you get is something that looks on the surface like the same output, but is a pastiche of the result when someone with actual knowledge is behind it.
The same thing seems to be playing out with agentic coding. You can get yourself something that looks like a functioning application. However, without a great deal of knowledge about how to build a functioning application, what you have is often just a reasonably functional mockup. At best you’ve got a handy personal tool that should never escape into production.
Individual productivity enhancements have a ceiling, what you can do with the tools is limited by the need to review the output. As everyone talks about productivity, I’m just not seeing any real research that demonstrates AI is measurably increasing productivity when you take a holistic view.
Individual AI productivity gains
Individually it’s clearly possible to use an LLM to increase productivity. As I’ve already described, a skilled individual can selectively introduce an AI tool to perform specific (usually rote but not quite scriptable) tasks. There are improvements to be had there, but they are similar to the bump you get when you finally figure out how to use a spreadsheet properly, or learn how to automate tasks with some simple coding. If you can already do those things, then AI use can, in some circumstances, automate some additional tasks or make it quicker to create those automations.
This level of improvement is appearing in research data, for example the London School of Economics found in their report Bridging the Generational AI Gap: Unlocking Productivity for All Generations that professionals using AI save an average of 7.5 hours per week. I have a theory that in many cases for non-coders, AI has just solved coding’s image problem, and these gains could have been achieved without AI.
However, another way someone might report increased individual productivity is by shifting the work onto someone else. That might be another person or team—writers end up fixing slop drafts and having to correct obvious errors, code reviewers wade through Pull Requests, and QA teams spend more time dealing with bugs. It also might be your reader or user who now has to wade through paragraphs of slop, is misled by inaccurate documentation, runs into bugs in your app, or finds it inaccessible to them. In this case you might feel more productive, but all you’ve done is move the work around, make someone else’s job or experience measurably worse, and reduce quality.
It’s for this second reason that a holistic approach needs to be taken to truly assess productivity across an organisation. If we look at specific individuals or even teams, we’re likely to miss task reallocation based on AI use.
AI as a forcing function for accessible data
In addition to the issue of task reallocation, there’s another reason why it’s hard to quantify how useful AI actually is. AI tooling has forced a lot of data to become available and easily consumed. This makes it easier to perform non-AI automations.
People who refused to write documentation in the past are now churning out skills, which are documentation. We can use these to easily identify the process needed to achieve tasks. Identifying repetitive processes is the first step of any automation...