Why I Hate LLM Confidence Scores | Justin Flick

Hello, Reader

If someone sent you this post, you have probably tried to extrude a confidence score out of an LLM.

I’ve now had this conversation at multiple companies with multiple people, and I’ve had it enough times that it seems like there’s a broader misunderstanding at work here. So my hope is that I can just send people this post instead of relitigating it in a thread every six months.

The short version: asking an LLM to generate a score for how confident it is in its own response is, from everything I can tell, completely useless .

I want to be upfront that I’d love to be wrong about this. There are people much, much smarter than me working in this space (and I work with many of them!). If you can rebut any of the arguments below, or point me at research I haven’t read, I would genuinely enjoy that conversation. But absent that, these are my current thoughts as I understand them.

What I Keep Seeing

The pattern shows up everywhere. Chat-based outputs, structured outputs, agentic task results. Someone wants the model to hand back a JSON object with a response key and a confidence key. And more often than not, that confidence key is a continuous score from 0 to 100.

That continuous scale is what really grinds my gears, and I’ll come back to why. But the broader problem is that there’s no scientific validity behind any of it. What you have built is a psychological safety trick. It makes the output feel more trustworthy without making it more trustworthy, and I’d argue the people shipping this experience are mostly lying to themselves about what they’ve shipped.

LLMs Cannot Reliably Quantify Their Own Confidence

I recognize that Anthropic and others have published research arguing that models maintain some kind of latent internal state while they generate. Anthropic’s interpretability work on tracing the thoughts of a language model found that Claude plans several words ahead when writing a rhyming couplet, which is not the behavior you’d predict from a naive next-token story. And their more recent work on emergent introspective awareness found that models can sometimes notice concepts injected into their activations and report on them accurately.

That’s real, and it’s interesting. But read the caveat the researchers themselves put on it: the capability is highly unreliable and highly context-dependent. We do not currently have a strong enough understanding of that internal state to assert that models have any usable ability to assess their own correctness. Noticing that something was injected into your activations under laboratory conditions is a very long way from quantifying how likely your paragraph about a customer’s refund policy is to be right.

If we think about what’s happening under the hood, there is some quasi-cognition going on in the sense of symbolic relationships between aspects of language. With reasoning models, there’s some observation of that process happening too. But this is where the whole thing collapses for me, because it immediately regresses into a who-watches-the-watchmen problem.

Say we want to tap into the thinking trace and use it to evaluate the self-correctness of a response. Fine. That approach has yielded real performance benefits, and I don’t dispute those. But if we’re talking about actually quantifying confidence, the thinking step can’t rescue us, because the obvious next question is: what’s the confidence of the confidence score produced by my thinking step? And then what’s the confidence of that? It’s confidence all the way down.

Do I think reflective reasoning patterns help models catch some of their own errors? Yes. Do I think models currently have a sufficient understanding of their own state to quantify that? No. It’s also worth noting that the research on intrinsic self-correction is not especially encouraging on this front. The DeepMind paper Large Language Models Cannot Self-Correct Reasoning Yet found that when models attempt to correct their initial responses using only their inherent capabilities, without external feedback, performance often degrades.

Confident of What, Exactly?

Here’s the question I now ask immediately whenever someone requests a confidence score: what is your heuristic for confidence?

Because “confidence” gets ambiguous extremely fast. Not slightly ambiguous. Extremely.

Is the model confident in the correctness of the response? In the coherence of the response? That it has attempted to fulfill the goal of the user’s request? Those are three completely different questions with three completely different failure modes, and a single float between 0 and 100 flattens all of them into the same number. Absent some heuristic that bounds the concept, these scores are effectively useless.

Confidence Isn’t Uniform Across a Response

There’s another problem with treating this as a single number, and it goes back to internal state.

In traditional machine learning, we have real...