The Trust Ceiling: The invisible limit on what AI is allowed to do
William Tunstall-Pedoe on AI
SubscribeSign in
The Trust Ceiling: The invisible limit on what AI is allowed to do<br>Trillions of dollars in value - and any real prospect of AGI - sit behind a line the industry can't scale its way past. The first of two essays describing humanity's path to trustworthy AI.
William Tunstall-Pedoe<br>Jul 20, 2026
Share
Over the past decade, artificial intelligence has advanced into territory long assumed to be permanently human. Machines learned to speak, to hear, to see, to answer our questions; then, most strikingly, they mastered language itself. Systems like ChatGPT, Claude and Gemini now converse fluently, draw on more of humanity’s written knowledge than any person could absorb in a thousand lifetimes, and increasingly reason, using language as a substrate for thought.<br>For the first time we can plausibly imagine the arrival of what some call Artificial General Intelligence (AGI), defined in OpenAI’s charter as “highly autonomous systems that outperform humans at most economically valuable work”. In practice: systems that automate large parts of every business, and can act as ever-present assistants in our daily lives for almost everything we want to do.<br>But despite this extraordinary progress, something fundamental is broken.<br>The systems we are building today are powerful, but they are, this essay argues, intrinsically untrustworthy.
A sample of recent mainstream coverage of AI failing in deployment. The four headlines are four different symptoms of the same underlying cause, which this essay describes.<br>Hallucinations, lack of accuracy, inconsistency and the inability of such systems to reliably explain their results are holding back AI adoption behind a ‘trust ceiling’ where trillions of dollars of value are stuck. The AGI prize might be imaginable but who is going to allow AI to fully automate their business or to take control of their personal life when at every step the AI system might do huge damage to their brand or reputation, cost them money or tell them something that is outrageously untrue? What regulator is going to allow AI systems to become widespread in an industry when every decision they take is a black box and subject to unpredictable error? The “...most economically valuable work” from the AGI definition will remain in human hands until this issue is addressed.<br>If we can solve this trust problem while preserving and extending the enormous capabilities we have developed, humanity has a path to reaping these extraordinary benefits.<br>This essay explains the technology issues behind this challenge and its companion essay, published alongside it (and linked at the end), describes the path that will get us there. It is not the path the industry is on - and without change, the industry’s path will fail.<br>Two Paradigms of Software
To understand why, it helps to step back and look at computer software itself. Broadly speaking, there are two fundamental paradigms.<br>The first is what we might call symbolic software. This is software from the traditional world of programming: algorithms, data structures, and databases. This is the majority of software even in use today. In this world, fully correct is the standard. There is a definition of what the software should do, and the software either meets it or it doesn’t. If a spreadsheet adds two cells together we expect it to produce the right answer every single time. If it does not, we call it a ‘bug’: the publisher finds it, fixes it, and knows it will stay fixed, because the same inputs always produce the same behaviour. When a software system that helps with the company payroll retrieves data from a database we expect it to never get a payment wrong. These systems are typically deterministic, consistent, explainable, and fully controllable, and when they fail, the failure is an exception to be eliminated rather than a permanent property of the system.<br>The second paradigm is statistical machine learning, and in particular the one machine learning method that has come to dominate: large neural networks (often referred to as ‘deep learning’). These systems are not programmed in the traditional sense. Instead, they learn from data. They are large mathematical formulae and begin with randomly initialised parameters and the parameters are then improved systematically through a training process, exposing them to data, until the formulae stop improving and they generate useful results.<br>The results however are never guaranteed. These formulae are approximations with performance always below 100%. Best guesses. However, they do very important things that symbolic software cannot.<br>What makes modern AI so powerful is that these deep learning systems improve as we scale them - a trend that has survived every prediction of its end. More data for training, more compute, larger models - and performance continues to increase.<br>When neural networks first saw...