Please Stop Talking About AGI - by Jack Morris
Token for Token
SubscribeSign in
Please Stop Talking About AGI<br>Why I think Yann Lecun was right about LLMs (but perhaps only by accident)
Jack Morris<br>Feb 21, 2025
58
Share
It’s become very popular over the last few years to speculate how close society might be to Artificial General Intelligence (AGI). What AGI actually means is murky, and often-debated, but mentioning AGI is usually a good jumping-off point for discussions of future artificial intelligences’ capabilities. Many following the field maintain AGI timelines, rigorous guesses for the probability of this mythical intelligence to emerge at future points. Those in the know might ask you for your timelines over coffee, classifying them as “long” – that it might be a decade or two before AIs are smart enough to take all of our jobs – or “short” – that it could happen any day now.<br>This isn’t the most useful way of thinking about the progression of AI capabilities. The existence of a timeline implies AGI has a rigorous definition and can be measured. It also implies that AGI is inevitable, the only question being when it will arrive.<br>What I see is not a march towards complete general intelligence, but rather a trend of increasing AI productivity per unit of human input. This trend holds across many disparate applications. Our AIs can label more data, write more code, do more math, as well as drive cars and pilot planes for longer with less intervention from us. It may be possible that we’ll never reach a point where AIs can run forever, uninterrupted, without human guidance. Rather we’re pushing the boundary of how much we can get for what we give.<br>Instead of talking about the mythical final frontier of AGI, I think we should start thinking more realistically and measuring the ratio of human input per useful AI output.
What will the future trend of human input per AI output look like?<br>Imagine for a moment the curve of how much we input have to provide for a unit of economic value the computer produces, and how this has changed over time. A very rough estimate is pictured above; one important open question is whether we’re approaching some unknowable carrying capacity, or if this figure will eventually decay to zero. (If this happens, it means that computers will be able to produce economic value with zero human input. This would be a frightening outcome.)<br>To understand what I mean better, let’s take a trip back in time to 2017…<br>We’ve seen this before (in self-driving cars)
If you’re new to the AI field, you should know that before language models, there was a previous AI craze circa 2017: the rise (and fall?) of the self-driving car.<br>If you’re not new to AI, let me remind you.<br>Around that time, several companies declared that within a year they would have Fully Self-Driving cars. Billions of dollars were raised. Millions of miles were driven. Many companies were founded, some of which eventually went bankrupt.<br>And years later, we’re still not quite at FSD. Teslas certainly can’t drive themselves; Waymos mostly can, within a pre-mapped area, but still have issues and intermittently require human intervention.
In 2016, Tesla CEO Elon Musk promised that a Tesla would drive itself fully autonomously from Los Angeles to New York City by the end of the year. That still hasn’t happened. (Teslas are still sold with an optional “Full Self-Driving” subscription)<br>In response, the field has moved on from speculating the exact point cars will be fully-self driving. People instead discuss miles-per-disengagement (or miles-per-human-intervention). How far can the car drive without a human getting involved? This new lens gives us something that we can measure and track over time. Better technology gives us more miles driven per necessary human action.<br>What does the future look like for FSD? A recent report said Teslas can drive thirteen miles per human intervention; this estimate feels a little low to me, but still seems pretty good. We can certainly drive this number up with bigger models, faster inference, more data, and improved overall engineering.<br>A crucial question is whether with current technology, the miles-per-intervention number is bounded by some theoretical limit we don’t understand. We don’t know whether our models will keep getting better forever (approaching infinite miles driven with no interventions) or if there really is some amount of human intervention that will always be necessary.<br>Why Yann Lecun was wrong (kind of)
Now let’s apply this idea to today’s AI craze: language models.<br>A few years ago, Meta’s Chief AI Scientist Yann Lecun gave a talk about how language models won’t give us a direct path to human-level intelligence. He argued that because language models generate outputs token-by-token, and each token introduces a new probability of error, if we generate outputs that are too long, this per-token error will compound to inevitable failure.
Yann presenting his “unpopular opinion”:...