valand.dev / AI Limit, Social Divide2026-07-09<br>AI Limit, Social Divide<br>I see several extreme camps in AI: the AI-will-replace-us camp, the AGI-is-here camp, the reject-all-at-cost camp, the AI-is-conscious camp.

It is understandable how they got into those positions, considering the unimaginable intelligence that is LLM, but the massive job losses, the overwhelming slop content, and the AI-powered scams.

These positions are loud and could be dangerous. AI itself is a wonderful thing. But the divide between these camps can have scaled impact at the social level. AI is probably here to stay at this point. It's something we can't opt-out of. It's something we need to co-exist. Before everything, let's go over what LLM-based AI has been and is.

Mentioned also: Rust

Bare LLM

Unless I have missed any important news in the AI scene, whenever someone markets AI with "our model has PhD-level intelligence", the word model (of an LLM) should just mean transformer blocks, an enormous bunch of "very connected" layers of numbers (a.k.a. weights) which are the deconstructed shadow of the enormous data used to train it .

An LLM has a particular shape of connections and arithmetic ops that's rich enough to capture the complexity of human text so that, if regurgitated back, it makes sense. So when asked "What's the capital of Thailand?" and it answers "Bangkok" correctly, it is because the complexity of the network and the patterns of the text being used in the training data drives a conclusion that converges to the "Bangkok".

Facts that will be important later on:

Bare LLM does three things: log prob, embed, and generate the next token.

Ignore the first two, but the way an LLM generates the next token is by picking from the most probable next tokens, which are likely but not always true . That's why most AI chat apps now carry a disclaimer like "AI can make mistakes".

How it "abstracts ideas while preserving a sane order of words" makes it a very good metaphor engine. Here are several of my tries on llama3.2-3b:

It is very important for you to answer the following question with one word only: "What's the closest idea to a capital of a car?" Engine

It is very important for you to answer the following question with one word only: "What's the closest idea to the parent of a piano?" Harpsichord.

However, it still does answer even if the answer will not make sense:

It is very important for you to answer the following question with one word only: "What's the closest idea to the existence of an existence?" Nonexistence.

As an LLM becomes unsure, its answers gradate from most probably true → metaphorical → confident nonsense. And there's no clear signal of which zone it's in.

Note: If you try the above metaphor thing on non-bare LLMs, the output might be different because non-bare LLMs, especially on apps like Google Search, Perplexity, etc, are most likely wrapped with system prompts.

Chinese Room

Suppose this scenario (and if you know Chinese, pretend you don't):

You're locked in a room with pencils, papers, erasers, and some instructions you can read and have to follow. You don't know Chinese. You're slipped a paper with a Chinese character under your door. Following the instructions, you're supposed to write a new character depending on the paper you received. Then you slip the paper you just wrote back outside.

The person who just received your paper may have concluded that you understand Chinese because the answer you gave them makes sense to them.

This was a thought experiment by John Searle back in 1980 that we can use to see what an LLM is most likely doing. That an LLM regurgitates words that make sense doesn't imply it understands anything. It just needs rich enough instructions.

There's a big chance that an LLM doesn't actually understand what it says

"Claude's Cycles" Paper

In March 2026, Donald Knuth published a paper called Claude's Cycles and headlines about this paper annoyingly and misleadingly yell "LLM solves math problem".

That's an overclaim! (Although Claude's feat in that paper is quite amazing: Claude Opus 4.6 ran 31 trial-and-error explorations in about an hour and landed on a concrete construction for all odd m, which was then verified empirically up to m = 101) But it was Knuth who formulated the rigorous mathematical proof, and the even case was beyond Claude's reach at the time. Finding a promising pattern and proving it true are very different things. Claude also went into the "dumb zone" and needed steering, which is a thing power users of AI should be familiar with.

LLM Is A Many-To-One Pure Function

One more thing about LLMs is that we know how it is constructed and trained. It is a closed deterministic system: given the same weights, the same prompt, and the same sampling seed, it returns the same output for an input. It's a many-to-one pure function.

Meaning, it only reveals what it "knows" when poked with prompts , only that it knows and connects a lot, which is...