Every AI memory benchmark has an asterisk | Tenure Platform<br>Project Resume Memory Modes PR Review<br>Editor Integrations<br>VS Code VSCodium<br>Compatible Clients<br>Mobile Open WebUI OpenClaw<br>Teams<br>Teams Overview Shared Memory AI Governance EU AI Act Compliance<br>How it works<br>Belief Merging Contradiction Handling Memory Modes AI Governance<br>Resources<br>Docs Writing Benchmark Paper GitHub
Install Free
Writing › Benchmarks<br>Research Every AI memory benchmark has an asterisk
Mem0 publishes 93.4% on LongMemEval as state-of-the-art. Someone runs their product through a clean harness and gets 73.8%. The CTO of Mem0 shows up in the thread and doesn't deny the gap. Instead, he says every number in the field comes with an asterisk. He's right, and that admission is worth understanding.
Tenure research · Jun 24, 2026 · ~7 min read
TL;DR<br>Mem0 announced 93.4% on LongMemEval. A clean third-party harness produced 73.8%, a 19.6-point gap on the same memory system and the same data.<br>The gap traces to hardcoded dataset-specific equivalence rules, a judge instructed to "lean toward yes," hidden chain-of-thought reasoning invisible to anyone sampling outputs, and a one-directional score-lift mechanism in their LoCoMo judge.<br>The CTO of Mem0 responded. He didn't deny the gap. He said every memory vendor tunes their own harness, and the only real fix is a shared harness everyone runs against, with cost and latency reported alongside accuracy.<br>He's right. The status quo is everyone reporting numbers with an asterisk. The question is whether the field moves toward shared, multi-dimensional evaluation frameworks or keeps letting each vendor tune their own judge.<br>Tenure's PrecisionMemBench is built for exactly this: multi-dimensional measurement (precision, noise isolation, latency, belief mutability) and deterministic evaluation that checks evidence paths instead of vibes-scoring final answers.
The gap A 19.6-point difference on the same system, the same data
Mem0 published 93.4% on LongMemEval as their state-of-the-art overall score. When a<br>third party ran their hosted product through a clean evaluation harness (gpt-5 answerer,<br>binary judge with no lean-toward-yes instruction, 5-seed mean), the best they could get<br>was 73.8%. Same memory system. Same benchmark data. A 19.6-point gap.
That kind of gap demands an explanation. The third party dug into Mem0's public benchmark<br>harness at the commit they shipped right before their April announcement, and found<br>several things.
What was found in the public benchmark harness<br>Finding What it means Dataset-specific equivalence rules 14 rules mapping 1-to-1 to specific LongMemEval question IDs. For instance, hardcoding that "scratch grains" should count as "layer feed," skipping the reasoning step the benchmark was designed to test. Hidden chain-of-thought Dataset hints get applied inside tags invisible to anyone sampling outputs. The judge only sees the cleaned final answer. Biased judge prompt The judge is told: "You have a tendency to say 'no' too quickly. Before concluding 'no', you MUST verify the answer is truly wrong, not just differently worded. When in doubt, lean toward 'yes'." A 5-step gauntlet must be cleared before marking anything WRONG. No comparable gauntlet exists before marking something CORRECT. One-directional score lift In the LoCoMo judge, evidence can promote a WRONG prediction to CORRECT, but the same evidence cannot demote a CORRECT prediction to WRONG.
None of this is hidden. The commit message from April 3rd, eleven days before the SOTA<br>announcement, reads: "Sync prompts from evals: CONTEXT CHECK, Rule 14 (contradictions),<br>conflicting numbers, personalization scan, BIAS CHECK in judge, chain-of-thought<br>tags, 5-step FINAL CHECK." Their engineer typed the words "BIAS<br>CHECK in judge" and "5-step FINAL CHECK" into git.
The response The CTO shows up and says the quiet part out loud
Deshraj, the CTO of Mem0, responded in the thread. He didn't deny the gap. He didn't<br>claim the findings were wrong. Instead, he made a different argument: these choices were<br>responses to flaws in the benchmarks themselves. The benchmarks contain hidden assumptions<br>that make questions unsolvable even with perfect memory retrieval. The reasoning traces and<br>equivalence rules were attempts to compensate for those flaws.
But then he said something more interesting. He said this:
"Yep. Most of these are a generic harness on one side and a tuned one on the other,<br>different token budgets, latency nobody mentions, half of them agentic, all squashed<br>into a single accuracy score that hides all of it. The only real fix is a shared<br>harness everyone runs against, with cost and latency reported alongside accuracy.<br>Until then every number out there, ours too, comes with an asterisk."
That is the most honest statement in the entire thread. It's not a defense of the gap.<br>It's an admission that the entire field is playing a game where everyone tunes their own<br>judge, and the resulting numbers are not comparable. The...