Lost confidence

I wrote a book! Click for details.

Subscribe

By Jason Cohen on

June 21, 2026

Reading time: 13 min

ePub (Kindle)

Printable PDF

Lost confidence

by Jason Cohen on June 21, 2026

RICE and other confidence-based frameworks are mostly noise. Here’s how to make decisions without pretending to know the unknowable.

source

Confidence games

Many prioritization frameworks include a measure of confidence1⁠—how sure we are that we can execute, at more or less the predicted effort, resulting in more or less the predicted impact. This sounds rational: If two projects generate equal value for equal effort, but we’re confident we can execute the first and unsure about the second, we should pick the first.

Or a measure of risk. The reader will decide whether or not .

This is not, however, how we use confidence scores. If it were, our process would look like this:

Score projects.

If there’s one clear winner, do it.

If there’s a tie, pick the one we are more confident in.

That’s not a bad idea, but popular frameworks like RICE include “confidence” in step one:

So does RPS:

Which means, for example, the following two scenarios are deemed equally strong:

A small incremental feature, that we’re sure we can execute.

A large feature, with large impact, that carries risk.

This equality is false. Especially when you remember that these two scenarios are nearly always exactly what we’re choosing from⁠—we’re typically confident in small projects and unconfident in larger projects, as well we should be.2 But that systematically skews the prioritization away from delivering as much value as possible, which I have often argued is the opposite of what a prioritization framework ought to do.

If you disagree, consider that the primary motivation for the Agile movement was the insight we should always have low confidence that large projects will be successful, despite our best techniques of planning, analysis, and estimation. And also consider this theory of Rocks, Pebbles, and Sand.

I don’t believe your confidence score anyway. First, because it’s ill-defined. What does “30%” mean? What it should mean, is you track your confidence scores across all projects, and then see how those projects went, and from that data, compute how accurate your “confidence” scores were with mathematical precision. But you don’t do that, do you? And if you only ship a few major features per year, you don’t have enough data to know, even in hindsight.

The second reason I don’t believe you is because we all know that projects are nearly always late, and often have less impact, less quickly, than we wanted⁠—no matter how confident we were. Indeed, we were always at least “fairly confident” or else we wouldn’t have agreed to do a nine-month, six-team project. So what weight should we place on “confidence?”

Hofstadter’s Law

It always takes longer than you expect, even when you take Hofstadter’s Law into account.

For more empirical evidence that “confidence” is useless, find any experienced Product Manager and ask: “Can you recall a feature you were certain would be well-received, but wasn’t?” Perhaps they had evidence from customer conversations, explicit requests, or purchase commitments⁠—yet after building it, almost no one used it, including those who promised they would. Their eyes will roll as they share multiple stories. This doesn’t make them a bad PM. Everyone has these stories. That’s my point. The best PMs have techniques to mitigate this problem,3 but none will claim they can eliminate it entirely.

Some techniques to improve prediction include asking customers to describe exactly how they would use a feature in their normal workflow. Often people genuinely think they would use something, but when forced to walk through it step-by-step, they realize, “Oh wait, this would require me to rewrite this code, we probably wouldn’t do that.” Or, “I’d need to export it into another system⁠⁠—actually, never mind.”

Similarly, ask content creators about their most successful work. Often it’s something they hastily produced⁠—a trivial piece they almost didn’t publish because it felt uninsightful or trite⁠—yet it generated more views and engagement than anything else that year. Conversely, pieces they spent dozens of hours crafting, work they’re genuinely proud of and consider their best, generate minimal interest:

source

Figure 1

We can summarize the relationship between our confidence and actual results in a handy two-by-two table:

Was confident<br>Was not confident

They loved it<br>Lots of examples<br>Lots of examples

Nobody cared<br>Lots of examples<br>Lots of examples

So, if “confidence” is too nebulous to define, and we shouldn’t trust ourselves with it anyway, what should we do?

What to use in place of confidence and risk

The answer lies in the realm of uncertainty, rather than of probability.

Probability presumes you know the underlying distribution, enabling mathematical predictions about future events. You can...