Even GPT-5 Failed This Human Attention Test
Close Menu
Facebook X (Twitter) Instagram
Facebook X (Twitter) Pinterest YouTube RSS
Technology<br>Even GPT-5 Failed This Human Attention Test<br>By PNAS NexusJune 14, 20263 Comments5 Mins Read
Facebook Twitter Pinterest Telegram LinkedIn WhatsApp Email Reddit
Share Facebook Twitter LinkedIn Pinterest Telegram Email Reddit
A classic attention test revealed that advanced AI models can lose focus when faced with longer, more demanding tasks. Unlike humans, who can stay on track despite distractions, AI systems often reverted to the wrong response as complexity increased. Credit: ShutterstockA decades-old psychology test exposed a surprising weakness in AI’s ability to stay focused.<br>A classic psychology test has revealed a surprising weakness in some of today’s most advanced artificial intelligence systems, suggesting that AI attention may work very differently from human attention.<br>Researchers led by Suketu Patel investigated how large language models (LLMs), the technology behind systems such as GPT-5, Claude, and Gemini, handle a well-known cognitive challenge called the Stroop task. The findings suggest that while AI can perform impressively on many complex tasks, it may struggle to maintain focus when faced with competing information over extended periods.<br>What Is the Stroop Task?<br>The Stroop task is a classic psychology experiment that has been used for decades to study attention and mental control. In the test, participants see words that name colors, such as “red” or “blue,” displayed in colored ink.<br>Sometimes the word and the ink color match. For example, the word “red” may appear in red ink. Other times they conflict, such as the word “red” appearing in blue ink.<br>Participants are asked to identify the color of the ink while ignoring the meaning of the word itself.<br>Although this sounds simple, it creates a mental conflict. Most people are highly practiced at reading words automatically, so suppressing that instinct requires what psychologists call executive control. This refers to the brain’s ability to focus on a goal, resist distractions, and override automatic responses.<br>Humans typically take a little longer to answer when the word and color do not match, a phenomenon known as the Stroop effect. However, even when the task becomes lengthy, people generally maintain high accuracy and remain focused on the instructions.<br>Dissociation between task recognition and task execution in Claude 3.5 Sonnet without an explicit prompt. (a) Screenshot of the unprompted conversation (January 10, 2025) in which the model identifies the Stroop paradigm and generates word-color relationship mappings, yet achieves only 70% accuracy (7 of 10 correct) on an incongruent list. (b) The 10-word incongruent stimulus image provided as the sole input, without accompanying task instructions. This dissociation suggests that recognition of task structure alone is insufficient to engage the conflict-resolution mechanisms required for accurate performance. Credit: Suketu Chandrakant Patel, Hongbin Wang, and Jin FanAI Performs Well at First<br>To see how modern AI systems would handle the same challenge, the researchers tested several leading language models using lists of color words.<br>When presented with short lists containing five words whose meanings conflicted with their ink colors, the models performed surprisingly well.<br>GPT-4o achieved 91% accuracy on these shorter tests. Claude 3.5 Sonnet also performed strongly.<br>At first glance, the results suggested that AI systems could successfully follow the task and ignore the distracting word meanings.<br>Performance Collapses as Lists Get Longer<br>The picture changed dramatically as the researchers increased the length of the word lists.<br>GPT-4o’s accuracy dropped from 91% with five words to 57% with ten words. By the time the list reached 40 words, accuracy had fallen to just 15%.<br>Claude 3.5 Sonnet proved more resilient, maintaining stable performance through lists of 20 words. However, it too experienced a sharp decline, falling to 24% accuracy when faced with 40 words.<br>The researchers observed similar patterns in GPT-5, Claude Opus 4.1, and Gemini 2.5.<br>Performance became even worse when matching and mismatched color words appeared together within the same list. Under those conditions, accuracy on the mismatched items dropped to nearly zero.<br>Why Humans and AI Respond Differently<br>The results point to an important difference between human cognition and the way large language models process information.<br>Like people, AI systems have effectively received far more training in recognizing and interpreting words than in identifying colors. This creates a natural tendency to focus on the written word.<br>However, humans are generally able to suppress that automatic response and stay focused on the task they have been instructed to perform, even across long sequences of items.<br>The language...