AI advice made people three times less accurate but twice as confident, researchers found

Skip to content

News

Latest

Deep tech

Sustainability

Ecosystems

Data and security

Fintech and ecommerce

Future of work

Conference media hub

More

Startups and technology

Investors and funding

Government and policy

Corporates and innovation

Podcast

Image by: Canva

TL;DR<br>Researchers found AI advice suppressed judgment suspension from 44% to 3%, accuracy from 27% to 9%, while confidence rose from 30% to 76%. People trusted wrong AI answers.

Researchers from three French and Italian universities found that access to AI advice collapsed people’s willingness to say “I don’t know” from 44% to 3%. Accuracy dropped from 27% to 9%. Confidence, meanwhile, rose from 30% to 76%. “People became much worse, the accuracy was only one third, but they were twice as confident,” said Valerio Capraro, associate professor at the University of Milano-Bicocca.

The study, authored by Capraro with Chiara Marcoccia of École Normale Supérieure and Walter Quattrociocchi of Sapienza University of Rome, deliberately used questions where AI models typically fail: visual details from films, such as the colour of a team’s uniform in Bend It Like Beckham. The researchers used Step 3.5 Flash, a model that was usually wrong on these questions, precisely so any reduction in judgment could not be explained as sensible delegation to a reliable tool. Some participants who would have answered correctly on their own asked the AI and became wrong.

Monetary incentives helped, but not much. Willingness to admit ignorance rose from 3% to 8% and accuracy from 9% to 16%, both still well below the no-AI baselines of 44% and 27%. Wharton researchers coined the term “cognitive surrender” earlier this year to describe the same phenomenon: people accepting incorrect AI answers 80% of the time while reporting higher confidence than those working without AI. The new study adds a sharper data point. It is not just that people trust wrong AI answers. It is that the mere availability of AI suppresses the cognitive habit of recognising what you do not know.

The 💜 of EU tech<br>The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

“For humans, the capacity to say ‘I don’t know’ is very important because it represents the recognition of the limits of our own knowledge,” Capraro said. He is particularly concerned about children, who are growing up with these systems before they have developed critical thinking skills. Google’s AI search overhaul replaced links with confident AI-generated summaries, and Common Sense Media this week called that design an “unacceptable risk” for students. The pattern is consistent: AI products are designed to answer, never to say “I don’t know.” The humans using them are learning to do the same.

Story by

Ana Maria Constantin

With expertise in digital marketing, product management, and branding & identity, Ana Maria Constantin develops strategies that resonate

(show all)

With expertise in digital marketing, product management, and branding & identity, Ana Maria Constantin develops strategies that resonate with our target audience in the software/SaaS industry. Collaboration and teamwork are paramount to her, as she loves empowering her colleagues to achieve outstanding results and unlock their full potential.

Get the TNW newsletter

Get the most important tech news in your inbox each week.

Story by<br>Ana Maria Constantin

Popular articles

Rob Gurzeev on Why AI Has Changed Cybersecurity’s Biggest Blind Spot

Common Sense Media says Google’s AI Search poses an “unacceptable risk” to students. Google cannot turn it off.

The Company Brain and the Future of Go-to-Market Strategy

Moonshot plans to list in Hong Kong within six months at a $30 billion valuation

China is rebuilding the smartphone around AI agents. ZTE’s NaviX sold out in hours.

We use cookies and other data for a number of reasons, such as keeping TNW sites reliable and secure, personalizing content and ads, providing social media features and to analyze how our sites are used.","linkColor":"6644ff","linkHoverColor":"7755ff","primaryBtnText":"Accept & continue","primaryBtnBgColor":"","primaryBtnBgHoverColor":"","primaryBtnColor":"","primaryBtnHoverColor":"","secondaryBtnText":"Manage cookies","secondaryBtnURL":"\/cookie-statement","secondaryBtnRedirectURL":"https:\/\/thenextweb.com\/news\/ai-advice-suppresses-critical-thinking-wrong-answers-study","secondaryBtnColor":"6644ff","secondaryBtnHoverColor":"7755ff","debugMode":false}'>