Eron on X: "https://t.co/fQgIxVyHZJ" / X<br>Post

Log inSign up

Post

Eron

@0xEronn

LLMs do one thing: predict the next word. Here's how that creates something that feels like thinking<br>Every time you send a message to ChatGPT, Claude or Gemini the same thing happens. The model looks at everything you wrote and makes one decision: what word probably comes next. Then it does it again. And again. Until the response is complete.<br>That's it. That's the whole trick.<br>And somehow from that one absurdly simple task repeated billions of times on an incomprehensible amount of text something emerges that can write code, explain quantum physics, translate poetry and hold a conversation that feels remarkably like talking to a person.<br>Here's what's actually happening inside.<br>The number that puts everything in perspective<br>GPT-3 trained on enough text that if a human tried to read all of it without stopping - no sleep, no breaks, twenty four hours a day - it would take over 2,600 years.<br>Larger models since then trained on significantly more.<br>This matters because the model didn't learn language the way you learned it. Nobody sat down and wrote rules. Nobody explained grammar or meaning or context. The model was shown an almost incomprehensible volume of text and given one instruction: get better at predicting what comes next.<br>Everything it knows about language, facts, reasoning, tone and personality emerged from that single optimization target applied at a scale that is genuinely difficult to hold in your head.<br>What a parameter actually is<br>The model's behavior is determined entirely by numbers. Hundreds of billions of them. These are called parameters or weights and they're the only thing that makes one model different from another.<br>At the start of training every parameter is set to random values. The model produces pure gibberish. Then training begins.<br>The process works like this. Take a piece of text. Feed all of it except the last word into the model. Look at what the model predicted would come next. Compare that to the actual last word. Measure how wrong the model was. Then adjust every one of those hundreds of billions of parameters slightly in the direction that would have made the model less wrong.<br>Do this for trillions of examples.<br>The math involved is staggering. If you could perform one billion calculations per second and you tried to replicate the computation required to train the largest language models it would take you over 100 million years. This is only possible because of specialized chips called GPUs that run enormous numbers of operations simultaneously.<br>After training is complete those hundreds of billions of parameters encode - in a form that nobody fully understands - something about how language works, what things mean, how concepts relate and what kinds of responses are appropriate in what kinds of situations.<br>How a single prediction becomes a conversation<br>When you send a message to a chatbot here is exactly what happens.<br>The system takes the full context of your conversation, adds a description of the AI's role and purpose, and feeds all of it into the model. The model looks at this entire block of text and predicts the most likely next word. That word gets added to the text. Then the whole thing runs again. And again.<br>Each prediction is probabilistic. The model doesn't pick the single most likely next word every time. It samples from a distribution. The most likely words get picked most often but occasionally a less likely word gets selected. This is why asking the same question twice gives you different answers and why responses feel natural rather than mechanical. Pure maximum-likelihood prediction produces stilted repetitive text. Sampling from the distribution produces something that reads like a person wrote it.<br>The temperature setting you sometimes see in AI tools controls this distribution. Low temperature makes the model more deterministic and focused. High temperature makes it more varied and creative and eventually incoherent if pushed too far.<br>The architecture that made everything possible<br>Before 2017 language models processed text word by word from start to finish. Each word was handled sequentially. This created a fundamental problem with long texts because information from the beginning of a passage had to travel through every intermediate step to influence the end.<br>In 2017 a team at Google published a paper called Attention Is All You Need. The architecture they introduced - the transformer - changed everything.<br>Transformers don't read text sequentially. They take in the entire input at once and process it in parallel. Every word can directly attend to every other word simultaneously. The beginning of a long passage influences the end as directly as words that appear right next to each other.<br>The key mechanism is called attention. Every word in the input gets represented as a long list of numbers. Then attention lets every word examine every other word and update its own representation...