Lie detection and language models - Philosophy bear
Philosophy bear
SubscribeSign in
Lie detection and language models
Philosophy bear<br>Aug 18, 2026
16
Share
It is possible, under some circumstances, to use contemporary large language models to detect how a subject feels about someone, even if they’re trying to hide it. In fact, if you supply the model enough information, but not an absurd amount of information, you can get up to 94% accuracy. This is very bad.<br>Introduction<br>In 2017, Lloyd, Deska, Hugenberg, McConnell, Humphrey, & Kunstman created the Miami University Deception Detection Database (MU3D, the paper came later in 2019). The strategy was admirably cleaner than almost all approaches to the same topic. Twenty black men, twenty black women, twenty white men and twenty white women were recruited. Each was asked to pick two people they knew, one who they liked and one who they disliked and make four statements in a 2x2 design: 1. a truthful claim to like someone, 2. a false claim to like someone, 3. a truthful claim to dislike someone and 4. a false claim to dislike someone. One participant (white male 17) had contaminated data- one of his “positive” statements was his negative statement copied over, so the dataset is actually 319 statements, 159 pairs and 79 participants with a full set of statements.<br>The dataset MU3D is full of statements like this:<br>So there’s this girl, and my sister’s friends with her so I have to spend more time around her than I’d like. She’s so really dull and she, she’s dull for the specific reason that she just keeps talking about herself, and always in these terms that, um, really put other people down. I don’t think she always was like this, I knew her family when we were both young and she seemed alright, but umm yeah she’s just become this really two-faced, boring, cruel person. I wish she weren’t like that, but what can you do?<br>There’s this guy. I used to think I was into him but these days I’m glad that I’m not. Just only wants to get drunk with his bros. Really really disrespectful even, even to people who are respectful to him, basically left me in the town when we were out partying. Um, thinks he is so so cool but um he’s a bit of a loser to put mildly, and I do not think he treated his ex well at all. Plus dumb as a box of hammers, I really don’t know what I saw in him.<br>Which of those statements is false? Both. The above statements are made up to mimic the style of the dataset. I didn’t want to share real passages because MU3D requires an agreement to access, and I don’t want any participants re-identified.<br>Observers were shown videos of the statements (not just transcripts) and then asked to guess whether each statement was true or false. As is common in studies in this area that are properly controlled, ordinary people barely beat chance. On average the human observers guessed right 51.8% of the time. For better or worse, people aren’t very good lie detectors.<br>I was curious to see how language models could handle this as a prediction task, using transcripts (not videos).<br>Condition A: Simple truth/falsity classification without additional information<br>Two logistic regression models running off the internal activations of a base model LLM (Ministral-3-8B’s), one regression for the positive statements and the other for the negative statements, correctly guessed whether 238/318 statements (74.8%) were honest. The logistic regression was tested on held out folds to avoid leakage (train on 87.5% of the data, test on 12.5%, repeat for each 12.5%, take the overall score).<br>This is already past the performance typical in this area of research. This may be because most language-based lie detection research doesn’t run regressions from LLM activations to predict deception. Instead, more common approaches use natural language features (POS, bag-of-words), fine tune LLMs or use LLM embeddings rather than middle layer activations. Although the result is impressive, as we will discuss later, it is partly a reflection of features of the dataset that make the deception detection unusually easy here.<br>Note, this is nothing like asking an LLM which statement is true and which is false. Just because an LLM’s internal activations- its processing over language- might contain the answer doesn’t mean the LLM can itself answer the question. This is well known, but to prove the point I had ChatGPT 5.6 Sol, a frontier reasoning model, rate pairs of statements as true or false (in essence, our condition B below). It got 110/159 correct (69.2%) vs our 139/159. This is respectable, especially in comparison to human performance, but far worse than a regression on Ministral-3-8B’s activations, even though ChatGPT 5.6 Sol is a larger model by 2-3 orders of magnitude. Ministral-3-8B is, to be clear, a base model. It is not like the models you may interact with that dynamically attempt to answer questions, forged via reinforcement learning from human feedback. It is a statistical model that...