LLMs Are Still Toxic, Stuck in the Past, and Bad at Math | Eyosias

LLMs Are Still Toxic, Stuck in the Past, and Bad at Math

July 13, 202625 min readRSS

The flaws never left the model. We just got good at building around them.

div>p]:my-6" style="opacity:0;transition:opacity 1000ms ease-out, filter 1000ms ease-out;filter:none">It's been nearly four years since ChatGPT was released. For most of us that first encounter was GPT-3.5, and the friction was enough to convince a lot of developers that this thing was not built to handle code. The hallucination rate was high, and the complaints came down to four issues.

AI was bad at math.

AI was stuck in the past.

AI had a short memory window.

AI was toxic, reflecting the worst parts of the internet.

Take a look at what people were saying back then.

So were all of these solved over the past few years? Is that why the technology is suddenly so useful?

Nope. All of these problems still exist, even in the newest models. Some have workarounds and others have ways to limit how often they show up, but every one of them is still sitting in the model.

Being a developer through this has felt strange. For a while the tools were hit or miss. Some days they saved me real time, other days they just made more work, and I never knew which I'd get. Then in January 2026, at the peak of Claude Code, half my feed was people realizing an agent could just write the code for them. The model underneath hadn't changed much, but the tooling around it had finally caught up.

That's worth slowing down for. Chasing each new tool means relearning something that may not matter a year from now. It's agentic swarms first, then loops, then graphs, and on and on. Meanwhile the four limits underneath don't move, and every trick worth knowing is a workaround for one of them. Learn the limits and the rest of the news cycle starts making sense on its own.

AI is bad at math

First, AI is bad at math, and the solution is just a workaround.

So if you feed a batch of math equations directly into new frontier models like Fable or Sol through their API instead of their native apps, you'll still see them miss some of the equations you sent.

I sent GPT Sol High 200 simple addition questions and it got one wrong. Put that AI in charge of your company's accounting and it's fumbling a transaction every couple hundred entries.

Here is the one it missed.

35653046+16814852<br>Model answered52467998<br>Exact answer52467898

See full run details

So why did it get it wrong?

When you ask an AI a simple equation like 1 + 1, it gets it right, because that answer is found in its training data. Give it a random, long pair like 35653046 + 16814852, though, and there's a chance that exact equation has never shown up before.

With no memorized answer to lean on, the model does the only thing it knows how to. A language model only ever predicts the next likely token, so it never really calculates. It ends up imitating the standard algorithm, the long-addition method from grade school, working through it one digit at a time. But it's still picking each digit by what looks statistically likely instead of doing the actual math. In our case, that's where it slipped and wrote a single wrong digit.

That step-by-step working out is what makes it a reasoning model. Before answering, it reduces one hard problem into a chain of simpler ones it can handle, which is how the run above got 199 of 200 right. It's a real jump, but each step is still a guess, so it never quite reaches a guarantee.

Yet drop the same problem into ChatGPT or Claude and it gets it right. Something outside the model is stepping in.

The app gets around the model's weak math by handing the problem to an outside tool. When it notices a hard math problem, it sends the numbers to a server running Python and lets a small script do the calculation instead of the model.

That wrapper around the model is called the harness. A harness is just whatever code runs around the AI. It can be all sorts of things, but here it's the part that catches the math problem and calls the Python script.

Here's how it works.

To see this in action, watch how your ChatGPT or Claude app handles a math prompt. If the answer is simple enough to already be in its training data, it just responds directly. If the problem is a little harder, it first tries to work it out itself with the standard algorithm. And once the math gets complex enough that neither of those is reliable, it pauses, reaches for the Python tool, and computes the exact answer. The model is trained to judge which of these three routes fits the problem in front of it. When it decides it needs the Python tool, the harness is there to execute the tool call.

The whole thing feels like a bit of a hack. In the end, language models only output text. So to make tool calling work, developers train the model to spit out a specific JSON structure whenever it wants a tool. The harness watches the model's output, spots that...