In Which I Lose My Mind Over Embeddings (HPLM Chapter 2) · Maayan Roth
In Which I Lose My Mind Over Embeddings (HPLM Chapter 2)
July 31, 2026
Chapter 2 of<br>The Hundred-Page Language Models Book<br>is when things started to get really exciting for me.
Chapter 2 – Language Modeling Basics
Chapter 2 introduces the fundamental building blocks of machine learning with natural language<br>input and outputs: tokenization, embeddings, and model evaluation frameworks. Tokenization is the<br>process of converting input words into “tokens” that are recognizable by language<br>models, and embeddings convert those tokens into dense numerical representations. Together, they<br>tackle the problem of efficiently representing arbitrarily large vocabularies in machine-readable<br>formats.
I had heard the word “embeddings”, of course, and I understood, at a high level, that<br>embedding models are something important for LLMs, and are used to capture semantic similarity<br>between words. More than that, I could not have told you. HPLM explains the concepts beautifully:
Imagine that you have a language with a vocabulary of 10000 words. (Note that this is a very small<br>language. The Oxford English<br>Dictionary contains over 500,000 words, and modern LLMs model multiple languages<br>simultaneously.) The most straightforward way to represent a word in this vocabulary is with a<br>one-hot vector , an array of 10000 bits, with each index in the array<br>corresponding to a word in the vocabulary.
One-hot vectors are easy to understand and perfectly machine-readable, but they have two major<br>drawbacks: they are incredibly memory inefficient, wasting most of their space storing useless 0s,<br>and they fail to encode any useful relationships between words. Indices are assigned to words<br>arbitrarily, and words that are very similar (e.g. “happy” and “glad”) may<br>end up far away from each other, whereas words with adjacent indices may have no meaningful<br>semantic relationship.
Word2vec
The primary insight of embeddings is that it is possible to generate an alternative representation<br>of tokens which is both more dense than one-hot encoding and represents semantic relationships<br>between words. HPLM goes into detail on Word2vec, a family of algorithms used to generate word<br>embeddings. Word2vec comes in several flavors, but they all share the same core conceptual idea.<br>Imagine a snippet of text containing a target word (in this case “cat”) and several<br>surrounding context words:
If you were to train a neural network such that, when given the target word as input, it produces<br>the context words with high likelihood, then if you were to pass in a similar word, one that is<br>likely to appear in similar contexts (e.g. “kitten”), you should expect the network to<br>produce similar output. The structure of the network looks like this: two layers, an input layer<br>that scales from the vocabulary size down to the embedding dimension size and an output layer that<br>does the reverse. The output of the first layer is a vector of floats, also called the<br>embedding vector , which can be used as a dense representation of the<br>input word when training or using a language model.
The part that blew my mind
So far so good. It makes intuitive sense that the vectors for “cat” and<br>“kitten” should be similar i.e. close to each other in vector-space. But then, the<br>author casually drops this bomb – these embedding vectors support simple mathematical<br>operations. For example, if you take the embedding vector for “king”, subtract the<br>vector for “man”, and add the vector for “woman”, you get a vector very<br>similar to the one for “queen”.
What?
How?
That sounds completely made up.
I had a little chat with Claude, trying to understand why this might be true. Claude made the<br>argument to me that, within the vectors for “king” and “queen”, there is<br>some subcomponent representing the concept of “royalty”. Honestly, that seems like<br>nonsense to me.
It gets a little clearer when I think of the operations as if they were happening in the space of<br>training examples, or at least I can squint at it and pretend that it makes sense. Imagine all of<br>the context examples for the word “king”. Some of them will include context about<br>gender, many will not. When you subtract the vector for “man”, you’re not<br>removing only the examples that include both “king” and “man”. You are<br>removing all of the context examples for the word “man”, as if the concept of<br>“man” is being removed from your language entirely. That doesn’t land you on some<br>genderless concept of “monarch”; it moves you somewhere really weird.
Now, add in all of the example texts for “woman”. Not only are we adding back in some<br>examples for “queen”, we’re also...