Gloriously Underqualified — Jeff Arnold Labs<br>Skip to content<br>Gloriously Underqualified<br>2026-05-15<br>I’m Not Qualified for This

When talking or thinking about fields outside your own expertise, have you ever asked, “Why don’t they try this?” Where “this” is your own idea?

I do all the time, especially in the area of deep learning. This is probably because deep learning feels like one of those artsy sciences. Sure, you must approach it with a strong knowledge of math. But simply trying out new architecture ideas is what led us to the AI revolution we now find ourselves in.

Just look at this small sampling of ways neural networks can be architected! Convolutional Neural Networks, Generative Adversarial Networks, and the paper that got us some huge wins: Attention Is All You Need. Many more where those came from.

Neural Networks are kind of a sandbox. They’ve got just the right amount of “art” and “science” for me to get excited about them. And yes, I want to throw all my ideas into the hat. But no, I haven’t taken the time to become an expert in the domain.

I used to have only two options. Become smarter at deep learning, or never try out my ideas. But with the powers of Generative AI today… is there a third?

AI Researcher

I’m certainly not the first person to use AI for research. It’s a high-profile use case:

A survey of AI used for research

Health, environment, and creativity breakthroughs by AI

How LLMs are transforming research

Research may be one of AI’s highest-leverage uses, because better research compounds into better tools, better science, and better AI.

Now imagine if anyone who had any old idea in any old domain could use AI to try it out. Imagine they could trust the AI to get it right . Many experiments would fail. But how many great ideas or discoveries might we uncover?

And of course, there is one phrase above that’s doing almost all the work here: trust the AI to get it right . My experiment became less about whether the calculator idea worked, and more about what kind of process would make an AI researcher trustworthy at all.

The Calculating Model

My idea to test was, simply put, embedding a calculator inside a neural network. And yeah, I see you raising your hand, there in the back, asking:

Can’t AI already use calculators?

So you just want LLMs to be better at doing math?

I answer: “Yes,” and “Not just math.”

First off, yes of course LLMs can use calculators. The way they do it now is by generating text instructions to run either a code script or a calculator interface or whatever tool the developer allows the LLM to request to use.

My idea is not the same thing.

Toy calculator-in-the-middle model<br>Train the routing path until the model asks the calculator the right question.

8 + 112 + 724 + 6

Prompt<br>8 + 1 = ?<br>The text enters the network as tokens.

→↓<br>Internal route<br>4 + 22<br>The route is still a noisy guess.

→↓<br>Calculator<br>26<br>The calculator is exact, but only for the query it receives.

Answer decoder<br>26<br>Target is 9; update the route and try again.

→↓<br>Protocol strength12%

Answer confidence18%

Loss92%

Step 0/Backprop is nudging the internal query toward the original operands.<br>Train stepReset

Instead of generating text during an LLM run that triggers a calculator tool call, my idea would have the LLM’s own “brain” (in quotes because it’s not technically a brain) run a calculator mid-inference (or mid-”thought” if you will). This is the difference between a human typing their desired math equation into a calculator, and the human having a chip implanted in their brain that would make all math “obvious” to them.

My hypothesis (ever since I wanted to try this back in 2020) has always been that adding strict, accurate mathematical logic inside the “brain” of the LLM might give it new superpowers in how it thinks. The AI internally wouldn’t just think “if someone asks me a math question I use the calculator,” but instead would think “if there’s benefit for my reasoning in measuring, adding, multiplying, dividing, or performing any other kind of mathematical operation, I’ll use the calculator.”

My theory is that exact symbolic operations could not only improve an LLM’s ability to do math on the fly (something it’s woefully inadequate at right now), but also improve its planning, spatial reasoning, music comprehension, and other forms of structured thought. I don’t know what else it might improve, but it seems like the potential is high. That uncertainty is why I wanted to test the smallest version first.

Researching Above My Weight Class

The calculator idea was only half the experiment. The other half was whether I could use AI agents to investigate it seriously. I did not want an agent to write a toy demo and declare victory. I wanted it to run experiments, notice failures, update its plan, and preserve what it learned.

We ran with a small GPT-2-style (a good ol’ fashioned GenAI model) architecture we could train rapidly, and put a calculator interface inside of it....