Are there words LLMs won’t say? — Samer Makes
Skip to article
LLMs have every English word in their vocabulary, but some of those words may never appear in their output. LLM-generated text is measurably less lexically diverse than human writing. The researchers at GING set out to investigate one possible reason why an LLM might never produce a specific word, even if it is appropriate for a given context.
When an LLM predicts the next token, it reads the preceding context and produces a logit for every token in its vocabulary. A token can be a whole word, a piece of a word, punctuation, a space, or any other text fragment. For example, Llama 3 has a vocabulary of 128,256 possible tokens. When inferring the next token, each of these options is assigned a logit.
The logit isn’t a probability but a raw numerical score. A function called softmax turns the logits into probabilities by raising the mathematical constant e to the power of each logit, then dividing that value by the sum of all the exponentiated logits.
Pi =
elogiti<br>Σj elogitj
Softmax scales based on the raw numerical gap between two numbers, not their ratio. For example, 4:2 and 2:1 represent the same ratio, but the absolute gaps are different: 4 - 2 = 2 , while 2 - 1 = 1 .
Applying softmax to [4, 2] gives approximately [0.881, 0.119] .
Applying softmax to [2, 1] gives approximately [0.731, 0.269] .
Even though the ratios are identical, the larger absolute gap makes the higher value much more dominant after softmax is applied.
Temperature
Temperature is a variable adjusted during inference, usually treated as a creativity parameter. Increasing the temperature makes the output more varied, while decreasing it makes the model's outputs more deterministic. At T = 0 , the model enters greedy sampling and always chooses the token with the highest logit.
During inference, the temperature value scales the logits by dividing each logit by T :
scaled logiti =
logiti
Suppose the prompt is: A lot of things happened! The first day of the conference was quite . If interesting gets logit 4 and long gets logit 2, softmax heavily favors interesting . If we set T = 2 , those logits become [2, 1] . The gap narrows, and the probability curve is flattened, reducing the disparity in likelihood between the two options.
Raising the temperature makes the model less concentrated on its top choice. The most likely token becomes less dominant, giving lower-probability tokens a higher chance of selection.
Temperature changes the gap before softmax
Move the temperature slider. The logits stay the same, but the scaled gap changes, and softmax gives the two tokens closer or more distant probabilities.
interesting logit: 4.0
long logit: 2.0
temperature: 2.0
interesting
73.1%
long
26.9%
But a token can only be chosen if it survives the sampling filter first.<br>Before filtering, every token has some probability, even if that probability is tiny. Sampling filters cut that huge list down to a smaller set of allowed tokens. Then the model randomly chooses from the remaining tokens, weighted by their probabilities.
The three common sampling filters are top-k , top-p , and min-p .
Top-k
For top-k , we choose a value of k, for example 5. The model ranks tokens by probability, keeps the first 5, and discards everything below rank 5. The final token is sampled from the surviving group.
Top-k keeps a fixed number of tokens
Change k. The cutoff moves by rank, regardless of the actual probability values.
k: 5
Top-p
For top-p , also called nucleus sampling , we choose a probability threshold, for example 0.9. The model starts with the most likely token, then adds the next most likely token, then the next, until the kept tokens together account for at least 90% of the probability mass. Tokens beyond that cutoff are removed.
So if the top token has probability 40%, the kept set starts at 40%. If the second token has 20%, the kept set grows to 60%. If the third has 12%, it grows to 72%. The model keeps walking down the ranked list until the running total reaches the top-p threshold. After that, the surviving probabilities are renormalized so they add up to 100%.
Top-p keeps tokens until the running total reaches p
Green segments survive. Muted red segments are beyond the cutoff. The amber line is the p threshold.
p: 0.90
p = 90%
Min-p
For min-p , the cutoff is based on the probability of the most likely token. If min-p is 0.05 and the top token has probability 40%, then the cutoff is:
0.05 × 40% = 2%
Every token below 2% is discarded. This makes min-p dynamic. When the top token is very strong, the cutoff becomes stricter. When the top token is weaker, the cutoff relaxes.
Min-p keeps tokens above a threshold relative to the top token
The amber line is the cutoff. Green bars survive; muted red bars are removed.
min-p: 0.05
top token probability: 40%
After temperature and sampling filters are applied, the model chooses one token from the...