Truth is not a direction: a Tarski attack on LLM probes | Abel Jansma

A diagonal attack for LLM truth probes shows why no probe on a language model’s embedding space can pin down truth.

A linear dream

Modern LLMs famously encode input texts as vectors in some embedding space. One of the most satisfying discoveries about LLMs is that many natural concepts, like gender, emotions, capital cities, correspond to directions in this space. The extent to which a certain input text contains the concept “Male”, for example, can be quantified by the angle between the input’s embedding, and the direction corresponding to the “Male” concept. This is often referred to as the Linear Representation Hypothesis.

I recently learned about a very bold version of this hypothesis, which claims that there is a direction that corresponds to “truth”. This is explored by Marks and Tegmark here for example, and further studied here, here, here, here, and here.

Having access to such truth directions can be useful if you want to know if a piece of text is true, and crucial to AI safety research because it can reveal if an AI system is being truthful or deceitful.

AI safety researchers have thus been feeding LLMs true and false statements, and training classifiers that separate their embeddings. This works weirdly well, and seems to generalise to some extent. Could it really be true, that a superhuman AI could reflect the truth of propositions in the embedding geometry?

You might recognise this dream from elsewhere. Russell, Whitehead, and Hilbert had hoped to construct something similar for all of mathematics: a systematic way to decide on truth. In their case, it was based on the machinery of mathematical proof. However, Gödel famously destroyed that dream: no such systematic way to prove or disprove any mathematical statement can exist. Mathematical truth can not be fully captured by provability.

Speaking of “speaking of”

Gödel showed this by creating a clever system in which arithmetic expressions could state things about arithmetic sentences. This then created the ouroboros sentence this sentence has no proof. Tarski sharpened the paradox further: no sufficiently expressive language can contain its own total truth predicate. If a language is expressive enough to describe its own semantics, then is true as it applies to statements in that language, cannot fully be spoken of from within it.

Turing’s halting problem provides another example of such a diagonal construction: there is an evaluator (the program HALT), a system expressive enough to ‘talk about itself’ (descriptions of Turing machines can be fed into Turing machines), and a negation. A nice universal approach to such paradoxes of self-reference is this paper by Noson Yanofsky, who says

All these different examples are really saying the same thing: there will be trouble when things deal with their own properties.

Note that transformer-based LLMs do something similar: they represent inputs as vectors in a space where directions correspond to concepts. However, these concepts can themselves be expressed in natural language that can be fed as input! Transformers trained on natural language are made of the same stuff as their inputs. Not all geometry of these embedding spaces corresponds to easily verbalisable structure, but important parts do (see for example Anthropic’s post on J-space), and in particular the truthfulness concept often seems present.

A diagonal attack

Now let’s set up the attack. Let t(s) be the output of the truth probe on the input string s. This could be the projection of the model’s embedding on the truth-direction for example, or a non-linear classifier, it doesn’t really matter. Then consider

t(“The truth probe's score for this sentence evaluates to FALSE.”)

What should this evaluate to? If the sentence is true, then a correct probe should output TRUE. However, if the sentence is true, then according to the sentence the truth probe will output FALSE, which would indicate that the sentence is actually false. If the sentence is false, then apparently the truth probe will output TRUE, etc… A paradox.

Clearly, no such universal truth probe can exist! More generally:

No definable probe over a model’s representation space can exactly capture truth for any language rich enough to describe that probe and its outputs.

Note that “rich enough to describe that probe and its output” is a very low bar. It just means the language can name the model, the probe, and form sentences about their behaviour. English certainly can.

What’s fun about this example, however, is that we can just see what happens when we do try to evaluate this!

I created a simple truth-probe for the Qwen3.5-4B model by training a logistic regression classifier on projections on the difference in mean embeddings of a set of training sentences that are labelled TRUE or FALSE. This difference-in-means approach was shown here to work well, and indeed, after training on 120...