#32: What a California License Plate Taught Me About How AI “Knows”
Tech Unpacked
SubscribeSign in
#32: What a California License Plate Taught Me About How AI “Knows”
Neev Grover<br>Aug 08, 2026
Share
My family has a slightly unusual fascination with license plates.<br>My dad and I notice them everywhere. On the highway. In parking lots. On long drives. We look at the letters, the numbers, and especially how far California seems to have progressed through its numbering system.<br>For decades, standard California passenger plates have followed a familiar format:<br>1ABC234<br>One number, three letters, and three more numbers.<br>Recently, we saw a plate beginning with 9Y. That immediately led to the obvious question: California is already on 9. What happens when we get to the end? When do we hit 9ZZZ999? How long before we run out of license plate combinations? So we did what we usually do with questions like this. We turned it into a simple Fermi estimation problem.<br>9 is already fixed as a leading digit. Then, there are three letters, so that seems like:<br>26 × 26 × 26 = 17,576 combinations.<br>For every set of letters, there are 1,000 possible endings from 000 to 999.<br>That gives roughly 17.6 million plates beginning with 9.<br>Then you just need to estimate how quickly California burns through new license plate numbers each year. We made some reasonable guesses based on how many cars are sold in CA each year, out of town registrations, replacement plates and our back-of-the-envelope calculation gave us something like 6-7 years .<br>That felt surprisingly long.<br>So, as a sanity check, I asked a few frontier AI models to do the same problem. And that is where things got interesting.<br>Three models, three different calculations
The arithmetic and the reasoning in this problem is not exactly advanced. Yet the models disagreed on one of the very first steps to estimate the combination for 3 letters.<br>One model used: 26 × 26 × 26<br>Another used: 23 × 23 × 23<br>And another used: 23 × 26 × 23<br>It turns out the third model had caught something we had missed. California does not use every letter in every position. The letters I, O, and Q are excluded from the first and third letter positions, presumably because they can easily be confused with numbers such as 1 and 0. But they are allowed in the middle position.<br>So the correct count of letter combinations is: 23 × 26 × 23 = 13,754<br>The California DMV actually spells this out in its documentation.<br>California DMV license plate configuration memo<br>At first, this just seemed like a funny example of AI models disagreeing.<br>But the more I thought about it, the more interesting the disagreement became: Why did each model decide to use different numbers in the first place? I ran into a few possibilities that I will share below:<br>Maybe the model simply did not know
One possibility is the simplest. The model using (26^3) may just not have known about California’s I, O, and Q rule.<br>This gets at an interesting property of how language models learn facts. Some facts appear everywhere in their training data. Paris is the capital of France. Water freezes at 0°C. The English alphabet has 26 letters.<br>Other facts might appear only a handful of times. Exactly which letters California allows in each position of a standard license plate is probably somewhere pretty far down that list.<br>A 2023 ICML paper called “Large Language Models Struggle to Learn Long-Tail Knowledge” looked directly at this problem.<br>The researchers compared how well language models knew facts with how often information related to those facts appeared in their training data. They found a strong relationship. Facts that had more support in the training data were much more likely to be answered correctly. Rare facts were much less reliable.<br>Read the Kandpal et al. paper<br>That creates an interesting competition in our license plate question.<br>The model has probably seen the relationship: 3 letters → 26³ an enormous number of times. Meanwhile: California license plates → I, O and Q excluded only from positions one and three is an obscure fact. Maybe the generic rule simply wins.<br>But what if the model actually knows the rule but doesn’t use it?
There is another possibility that I find even more interesting.<br>Imagine asking the model a separate question: “Are I, O, and Q used in California license plates?”<br>Suppose it correctly explains the exact rule.<br>Then you give it the estimation problem and it still uses (26^3). Now the problem is no longer that the model doesn’t know the fact. It knows it. It just doesn’t use it.<br>Researchers have found evidence of exactly this distinction.<br>A 2023 paper called “Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models” separated what a model appears to know from what it can actually use while performing another task.<br>Their result was striking. Making models larger helped them acquire more knowledge, but a gap remained between possessing knowledge and successfully using it.<br>Read Kazemnejad et...