Tokenisation: Who decides what a token is anyway? — idlemachines

essays<br>essays

How does a neural network understand text? It's a question so basic, you can be forgiven for forgetting about it.<br>Words and letters are human-readable, but are discrete (even if infinite in combination) and not compatible with the continuous representations that neural networks use.<br>To bridge the gap, we need to take a sequence of human readable information and quantise it into a fixed set of pre-defined objects.<br>This process is called tokenisation.

A tokeniser works on integer sequences. e.g. "the cat sat" → [1169, 3797, 3332] so the text can be mapped into a fixed vocabulary, the actual embeddings come after.<br>The tokeniser itself is a separate model, usually trained separately from the core language model, and is fixed before pretraining. It is then inherited by the model as it learns, which makes the choice of how we tokenise the text one of a very few decisions you can't update run to run.

Technically of course you can update it run to run, but as we'll show this is a very expensive operation, and the model can't learn to do it itself. You can transplant an updated vocabulary by re-initialising the embedding rows and continuing training, it's expensive and brittle, but some people do it.

No matter how long the training takes, the model can’t update the initial tokenised representation just from the training loss, just the embedding of each individual token.

The slightly undervalued or forgotten place of the tokeniser is exactly because it doesn't feature directly in the loss curve, it’s not a parameter you typically think about when it comes to tuning your model. But it defines what's easy or hard to learn and that alone is reason to understand where the artificial boundaries come from.

We've all seen the classic example: "how many times does the letter r appear in strawberry?" For a long time models were replying “two”. The usual explanation is that the model sees tokens str/aw/berry (as in cl100k), and not the individual letters, so it can't count. This is true but a bit shallow (just see the current models).

This was ultimately fixed by using a reasoning model's chain of thought, which spells out the word and has an easier time counting the occurrences. But the fact the model couldn't do this without a reasoning harness is a direct consequence of the tokeniser.

Why not characters or words

When we think about how to split up the text into chunks there are two natural ways to divide the text; splitting each word, or splitting each character.

If we start with the character level split, it feels obvious because we know there are only 26 letters in the English language, we can double that if we want to include capital letters, 10 numbers, a bunch of punctuation, and then even if we want to expand to Romance languages the full set of characters needed including all accented letters is measured in the low hundreds. (Deliberately avoiding non-Romance languages for now.)

The beauty of this simplicity is that it gives us a tiny vocab, and we can be confident that we never encounter anything unknown.<br>The cost we pay for this simplicity is in the length of the sequence, a passage containing a few hundred characters requires the sequence to be a few-hundred tokens as well. If we are modelling this with attention, we need to remember it scales with sequence length squared. This granularity is expensive.

The other end of this spectrum is to divide the sequence into individual words. As people we have intuition about the definition of words, so this feels like another natural option.<br>This gives us much shorter sequences, where the meaning associated with each token must be much denser. This seems like it could be a good property, but it requires a vocabulary that grows essentially unbounded. And more than that follows a characteristic long tail distribution where many of the words are almost never used (and therefore much harder to learn good representations for.)

The other aspect is it treats related words completely separately. The set run/runs/running/ran are independent in this context, and a model needs to learn essentially the same concept multiple times.

This makes it clear why the convention is sub-word tokenisation. By defining a vocabulary set of character sequences we can balance flexibility and efficiency with the natural information compression that written languages have developed over thousands of years.

The principle is that common words can be taken whole (the = 1 token) and rare words can be broken into smaller reusable pieces (tokenisation → token + isation), that can be shared across words.<br>It also means we never actually find something inexpressible as even in the worst case you can break down new words into their component characters.<br>The question is just how to decide what the tokens are?

The same sentence tokenised at character, word and subword level

How does Byte Pair Encoding (BPE) work?

The...