Empty shelves or lost keys? Recall is the bottleneck for parametric factuality
Skip to main content
Research
Search
Empty shelves or lost keys? Recall is the bottleneck for parametric factuality
August 12, 2026<br>Nitay Calderon and Gal Yona, Research Scientists, Google Research
When LLMs get facts wrong, is it because they never learned them or because they can't recall what they’ve already encoded? Our knowledge profiling framework reveals the latter: frontier LLMs encode nearly all facts, yet struggle to recall many of them.
Quick links
Paper
WikiProfile
Share
Copy link
Factuality is essential for making Large Language Models (LLMs) reliable. When a model answers a factual question incorrectly, is it because the fact was never encoded, or because the fact is encoded but not accessible? Standard accuracy metrics collapse these cases together, even though they suggest very different limitations and very different interventions. Encoding failures call for scaling model size or expanding data coverage, while recall failures might also point to post-training and inference-time methods that help LLMs better utilize what they already encode.<br>In “Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality”, we introduce knowledge profiling, a behavioral framework that measures both encoding and recall, and use it to examine the underlying bottlenecks of factuality in frontier LLMs (such as Gemini3 and GPT-5). We then show that many factual errors in frontier LLMs are better understood as lost keys (recall failures), not empty shelves (encoding failures).<br>By analogy, we use encoding to denote parametric representation of facts, recall to denote retrieving encoded facts without external cues, and recognition to denote identifying the correct fact when it is presented among alternatives. To support this analysis, we introduce WikiProfile, a benchmark of 2,150 Wikipedia-derived facts, each paired with ten questions that probe encoding, recall, and recognition.
The core idea: Knowledge profiling<br>Knowledge profiling shifts the unit of analysis from individual questions to facts. Instead of asking whether a model answered a specific question correctly, we ask a broader question: what is the state of the fact? We classify each fact into one of five knowledge profiles: (1) encoding failure, (2) recall failure, (3) direct recall, (4) recall with thinking, and (5) inference without encoding. These profiles provide a more informative diagnosis than question-level accuracy alone.<br>The classification is based on whether the fact is encoded and how accessible it is: Cannot be recalled, can be directly recalled, or can be recalled only with thinking (eliciting intermediate computations before the final answer, including chain-of-thought prompting and thinking-optimized LLMs).
Five knowledge profiles that characterize facts.
We operationalize this with three behavioral notions:<br>Encoding: A model encodes a fact if it can correctly reproduce it in a pre-training-like context. In our setup, we measure this using proposition completion and contextual questioning, which place the model in contexts similar to those in which the fact would naturally appear during pre-training (without revealing the answer), thereby priming the model to expose whether the fact is encoded.<br>Knowledge: A model knows a fact if it can correctly answer semantically equivalent questions about it across different phrasings, including both direct and reverse questions (e.g., if *A is B*, a direct question asks "What is B?", while a reverse question asks "What is A?").<br>Recall: A model recalls a fact if it knows an encoded fact. If it recalls the fact without thinking, we refer to this as direct recall. If it knows a fact that is not encoded, we refer to this as inference without encoding. This occurs only when thinking is enabled and the model relies on other encoded facts and performs multi-hop reasoning or educated guesses.
Top : We extract facts from Wikipedia, a predominant source of pre-training data. Left : We measure encoding by prompting the LLM to reproduce facts within their original context. Right : We measure knowledge by asking questions across varied phrasings and relational directions, with and without thinking.
Introducing WikiProfile<br>To operationalize knowledge profiling, we constructed WikiProfile, a benchmark designed to measure factuality on naturally occurring facts. WikiProfile is constructed using a fully automated pipeline powered by a prompted LLM, Gemini-2.5-Pro with thinking. Prompts were developed through manual optimization on a small held-out subset. We extract candidate facts from Wikipedia pages by identifying facts: a proposition involving an ordered pair of entities (subject and object), where the subject appears first in the document. Each fact is paired with 10 tasks: two for encoding, four for knowledge evaluation, and four multiple-choice variants for recognition.<br>We generate direct...