The Death of the Chatbot - by Damir Marusic

SubscribeSign in

Tuesday Notes<br>The Death of the Chatbot<br>Say goodbye to your little friend.

Damir Marusic<br>Jun 17, 2026

10

Share

I’ve been using AI a whole lot lately, at work and at home. I’ve spent an inordinate amount of time trying to get it to be as useful as possible. I’ve mostly been using it for ideating and sharpening arguments on tight deadlines, and I’ve had some success. Nevertheless, though I’ve fully accepted that AI is a game changer and will profoundly reshape the economy, I now believe that the practice of chatting with your AI is not long for the world. The chat box will be remembered as a primitive interface for a technology in its infancy, akin to what punch cards were to early computers.<br>Chatbots are incredibly limited. That “oh shit” feeling we got when we first started playing with them is… an illusion. Or better put, a misdirection, largely due to the language we use to talk about them. That language has us asking the wrong questions. We have been focusing on what these things are rather than what they’re for.

Subscribe

It’s become fashionable among the Silicon Valley set to mock the famous “stochastic parrot” paper, which insisted that large language models are nothing more than probabilistic language simulators. Understanding is definitionally beyond them, the paper argues, because statistically inferring the next unit in a series does not generalize to a conception of the world. Mastery over any domain of knowledge — which the models demonstrate in some areas — should not be taken as evidence of understanding. And without understanding, there’s no “there” there.<br>Critics counter that the problem is muddled concepts — that the truly stupefying leaps that have emerged in the AI space are proof that we don’t really know what we mean when we talk about understanding. After all, at scale, these models do seem to be able to behave as if they understand. LLMs trained only on text seem to build internal representations of things they were never explicitly taught.<br>That’s not nothing. But it’s a lot less than it might seem.<br>I’m sure we’ve all noticed how even the most advanced models tend to get less sharp during the course of a long conversation. It was more obvious in earlier models, but it’s still abundantly apparent today. The term of art is “context rot,” and it gives us insight into why these things are the way they are.<br>The “context window” is, basically, the bot’s attention span — the space it has for parsing what data you give it. You type something in, it “reads” it, and responds, using the complex probabilistic math that lets it infer, word by word, what is an appropriate answer. But the bot doesn’t really “remember” your conversation. It’s quite literally “born” anew with each turn in the chat, spawned fresh and fed back a transcript of what was said up to that point. Rinse and repeat.<br>Rot itself is due to two things. One is an observed property — that the statistical inferences the model draws degrade as the text it’s working with gets longer. Models have been observed to focus on whatever comes at the very beginning and the very end of a long stretch of text, and tend to lose track of what’s buried in the middle. That’s partly because of training — they’ve learned from experience that the important stuff usually sits at the edges of a text — and partly due to how the math works.<br>The other is that engineers have tried to stretch limited context windows — in some setups by having the model summarize the conversation before reading it back. That leads to elisions and omissions that over time compound, leading the model to get more lost and forgetful the longer you’ve been at it.<br>The models have no sense of time and no real memory. But most important, they have no sense of truth, which flows from the other two facts. The model has no way to check what it says against the world. It has only ever learned what true-sounding text looks like — the shape of a well-formed claim. Its builders know this, which is why you get a comical automatic apology blurted out when you catch it making stuff up.<br>With all this laid out, it’s hard to defend these things being conscious in any recognizable form. And without consciousness, how can one have understanding?<br>Let’s try to put the thorny question of consciousness aside, the critic responds. Maybe the error is in thinking one needs consciousness to have understanding. Books, after all, contain plenty of understanding (the good ones, anyway). If a book can hold understanding without being conscious, why not grant the same to a system that does something a book could never do — talk back and engage in a stimulating dialogue?<br>Books do in fact contain understanding, but they do not understand themselves. They encode the understanding of an author. And that’s a critical insight when transferred to AI. AI, too, encodes human understanding — more of it than any single human mind could possibly hold. And...