The Case Against the AI Thought Partner - by Sofia Quintero
SubscribeSign in
The Case Against the AI Thought Partner<br>And Why You're Doing the Work the Labs Aren't
Sofia Quintero<br>May 28, 2026
Share
I find myself indulging in deep conversations with Claude and ChatGPT, exploring topics I always wanted to discuss with people but never found the right person to talk to, or at least no one available to talk to me about them. These rabbit holes feel like a warm bubble bath for my brain but that’s until I remember what is actually going on and my bath turns immediately cold in disappointment.<br>As a general definition an AI Thought Partner is the use case in which you have long conversations about a particular problem, normally one unrelated to optimizing or automating workflows, conversations that complement your thinking or understanding of the issue at hand. A thought partner acts as the synthetic version of a coach, advisor, therapist, friend that pushes back on your thinking.<br>At least 100 million people globally use a major AI chatbot as a thought partner occasionally, and tens of millions regularly. This range has been inferred (assuming the lower end) from public data published by frontier labs and their reported use cases.<br>The AI Thought Partner use case has been promoted by many leaders in the field as a killer use case, as a force multiplier for better thinking and rigorous analysis. But the current evidence shows that having a net positive experience is a lot harder than you think and most likely far from what you may be doing on a daily basis.<br>Recent studies and cognitive concepts can help us understand the mechanism behind the thought partner pitfalls, and why today, the AI Thought Partner use case may not be safe for users. The models you are working with, whether free or paid, are not trained to be balanced and adversarial when needed and advanced settings and instructions can only marginally minimize the negative effects of sycophancy.<br>The responsibility for sycophancy belongs with Frontier Labs. They built the behavior into the training; they own the fix. Putting the responsibility onto the users is a flawed approach to making sure the technology truly serves individuals. What Labs are doing right now is the equivalent of providing over 100 million people a highly addictive video game and asking users to be careful with their time while also pressuring them to use it every day.<br>We may not be able to regulate AI fast enough and the speed of development may not allow us to wait for longitudinal research results on this topic until it is too late but, we at least should become curious and conscious about the trade-offs we are experiencing.
The Cognitive Failure Modes
Treating AI As A Neutral Entity
The models being flattering in conversations is an issue, for sure. But the key challenge of the Thought Partner use case is that the model is rarely arguing you into anything. It is the repetition, your own act of articulating, the one-sided pool of infinite “facts”, and your reflexive social trust that are doing the work. That is the quieter and more unsettling story behind the dangers of this use case.<br>A study by Glickman & Sharot (2025) demonstrates that human-AI feedback loops amplify perceptual, emotional, and social-judgment biases and that this amplification is greater than human-human amplification. You experience this when you say things to actual humans like, “oh I knew you would say this, you always point at XYZ when it comes to X topic” However, when you use AI in conversations you are more likely to think of it as an objective entity. The reason is subtle, people partly discount a human’s quirks but treat the AI as neutral, so you absorb its biases without resistance.<br>The study focused on social and stereotype judgments, like gender and racial stereotypes, not on politics or worldview which is where we need more research, however as a cognitive loop we should consider at least the potential transferability to our judgements on AI outputs. On the other hand, the same study found that interacting with accurate AIs can improve people’s judgments. The mechanism is symmetric and it de-biases when the AI is accurate.<br>Believing Enough Pushback Can Correct Sycophancy
In my experience, as a paid user with strong systems instructions, the challenge is that as conversations lengthen, the models are more likely to mirror your beliefs in order to continue the conversation and when you push back, you are likely to experience a temporary correction only to fall into the same trap a few interactions later. Here’s an example of repeated push back with no change in a conversation about agent orchestration.
A study published in Science, led by researchers like Myra Cheng and Dan Jurafsky. Shows that AI affirms user actions 49% more than humans do, even when those actions involve deception or relational harm. They noted that this validation could cause behaviors where users become convinced they are...