The AI Competence/Judgement Gap - Itamar Gilad Skip to content

Can you perform every task in your organization?<br>Unless you’re a solopreneur, the answer is likely No. You’re probably very good at some types of tasks, OK or so-so at others, and have low or zero proficiency levels at most. As a product manager at Google, I could write decent product requirement documents, produce sub-par marketing copy and designs, barely knew how to add to the very complex Gmail codebase, and had no clue how to create a balance sheet for my business unit.<br>So my proficiency graph (and I believe this is typical of most *) looks something like this.<br>* All the graphs shown in this article are speculative<br>But there’s a second type of proficiency: the ability to judge the output of a task. We’re generally able to judge more than we can do. As a PM I could evaluate and give meaningful feedback on marketing materials and designs, even if I wasn’t good at producing them.<br>Our ability to judge work we can’t do is healthy and important. A mid-level manager may not be able to perform every task people in her org handle (at least not at the same level of proficiency), but she should be able to judge and critique the work.<br>Judging is also useful across peers. When a product manager and an engineer collaborate, it’s helpful for the engineer to be able to review and critique the requirements, and for the PM to give intelligent feedback on engineering choices, especially if they impact user experience and business value. There’s a definite risk of overdoing it, though. If you ever got the answer “well, if you think it’s so easy why don’t you try doing it yourself” you probably overstepped that fine line.<br>Enter Artificial Intelligence<br>AI is drastically re-drawing the lines. LLMs can accelerate the tasks we’re good at, help us do a better job on tasks we were not good at, and even let us venture into tasks we couldn’t before. I can now confidently write marketing copy, generate designs, and produce code. I can even produce legal contracts and accounting spreadsheets that I couldn’t do before.<br>But what about quality? I don’t think AI gets you to expert level on anything, but you can perform far more tasks at medium/OK level (which is sometimes all you need), and yet more at a below-average level.<br>(Sidenote: the Doing with AI curve isn’t flat because: a) the LLMs are better at certain types of tasks due to better/more training data, fine-tuning, and reinforcement learning. b) The quality of AI output heavily depends on the context provided by the user and the ability of the user to guide the AI).<br>This causes a major change in the workplace— we can now do work that traditionally was done by our peers. In many companies, PMs, designers , and other roles now contribute production code or develop internal tools — traditionally the job of developers. On the flip side, a developer or a designer armed with ChatGPT, Claude, or Gemini can now do market segmentation, idea prioritization, and spec generation — stepping into the domain of product managers. It looks as if the roles are starting to overlap, and some speculate they’ll eventually converge.

The Danger Zone<br>The sharped-eyed among you may have noticed a challenge. With AI, your proficiency level in doing, exceeds your proficiency level in judging. In other words, AI lets you do work that you’re poorly equipped to judge. A PM may generate bad production code and a developer may choose bad ideas, and neither can tell the difference.<br>Looking at the graph we can see two areas of tasks performed by AI:<br>The Safer Zone — tasks you can judge fairly well. In this area you can tell that the outputs AI has provided are sub-par, and iterate to get them to the right level. But the risk is not zero because sometimes we fail to check — more on this below.<br>The Danger Zone — tasks that are outside your area of solid judgement. Here there’s a real risk of doing something silly or harmful, like creating security vulnerabilities in your product.<br>You may argue that smart, responsible people can recognize their limitations and seek help when they step outside their comfort zone. However, decades of psychological research suggest otherwise. Psychologists have identified a long list of cognitive biases and heuristics that make us overconfident about our abilities, and even more so in areas where we don’t know what we don’t know:<br>Better-than-average effect — Research shows people tend to overestimate their own qualities and abilities compared to others. For example, following bad investment advice is a risk for the average Joe, but not for me, who knows more about investments than the regular person.<br>Dunning-Kruger effect — research shows people with low ability in a specific area tend to systematically give overly positive assessments of these abilities. The less capable you are, the bigger the gap between your perceived abilities and your real ones.<br>(Sidenote: ChatGPT gave me a much longer list of cognitive biases,...