A calculator, compiled into a transformer — Out of Distribution

Posts · Torchwright<br>A calculator, compiled into a transformer<br>Rob Porter · August 9, 2026 · Code ↗ · Share ↗

LLMs’ weakness at math was once a running complaint. Tool calling, scale, and<br>hidden reasoning scratchpads have made the problem much less conspicuous, but<br>the weakness remains. To test that directly, I turned off reasoning and gave six<br>frontier models the same 500 random five-digit multiplication problems. Each<br>model had to answer directly, with no scratchpad work along the way. Most<br>struggled: five of the six scored below 30%, and none reached 90% accuracy.

What models learn and what transformers can express are different questions.<br>Given a fixed maximum operand length, there ought to be a transformer with the<br>right weights that can multiply every pair of operands exactly. Most research on<br>this gap has approached it as a problem of learning.1 I<br>wondered about a different route: could I<br>build a transformer directly from the same basic algorithms taught in grade<br>school?

That question led to more than one calculator. I built four in all, each<br>implementing the same calculator in a different way, and compiled each one<br>directly into transformer weights. Nothing is trained. All four predict exactly<br>the same answers; what differs is how they get there. The grade-school<br>implementation is just one of those routes.

Building the grade-school calculator

I built all four calculators with Torchwright, a compiler I wrote that turns<br>fixed computation graphs into transformer weights. For each calculator, I use<br>Python to assemble one such graph from a restricted set of primitives that<br>Torchwright knows how to realize in a transformer. Introducing<br>Torchwright explains how the compiler works. The<br>compiler machinery is shared across all four calculators, but this section<br>focuses on the grade-school version.

All four calculators use the same input format and digit representation. They<br>accept expressions of the form A op B\n, where op is +, -, or *. Each<br>calculator is built to handle operands up to a fixed number of digits (set at<br>compilation time). It treats shorter operands as left-padded with zeros, giving<br>the arithmetic a fixed number of digit positions to work with. At each digit<br>position, the calculator represents the value there with a one-hot vector: the<br>coordinate assigned to that value is set to 1 and the rest are 0. Concatenating<br>the vectors from two positions produces one of 100 distinct lookup-table keys,<br>one for each possible pair of values. The Python loops below are part of the<br>construction process: they assemble tables and operations that the compiler<br>turns into weights. The loops themselves never run at inference time.

The grade-school calculator is the first of the four implementations<br>(source).<br>Its three-digit<br>checkpoint<br>accepts operands from 0 through 999. I have exhaustively verified that it<br>generates the correct answer for all 3,000,000 valid expressions that it<br>supports.

The intermediate values inside the model are not necessarily exact, but they do<br>not need to be. Approximation error can shift the final logits and narrow the<br>gap between the correct next token and the alternatives without changing which<br>logit is largest. As long as the correct token remains on top at every<br>generation step, greedy decoding produces the exact answer.

As far as I can tell, this is the first dedicated digit-level multiplier<br>compiled into runnable transformer weights. Compiled transformers themselves<br>aren’t new—Tracr compiles RASP programs into<br>weights—and neither is compiled arithmetic: addition and subtraction have been<br>compiled into a language model, and addition<br>and parity have been constructed with explicit layer<br>counts.<br>Multiplication, though, has appeared only as approximate gadgets inside an<br>existence proof, never as an instantiated<br>model.

Arithmetic

In the grade-school construction, addition and subtraction work much the same<br>way. Both move from right to left, one column at a time. For addition, a lookup<br>maps (a, b, carry) to an output digit and the carry for the next column;<br>subtraction follows the same pattern with a borrow. Multiplication eventually<br>works column by column too, but it needs some setup first: the graph must produce<br>and arrange all the one-digit products.

Each one-digit product comes from the familiar 0 through 9 times table. For<br>every pair of operand positions, the graph looks up the two digits and receives<br>their product split into tens and ones: 7 * 8 becomes (5, 6). The following<br>Python dictionary defines the 100-row mapping used to construct each lookup:

for a in range(10):<br>for b in range(10):<br>key = torch.cat(<br>[embedding.get_embedding(str(a)), embedding.get_embedding(str(b))]<br>product_table[key] = torch.tensor(<br>[float(a * b // 10), float(a * b % 10)]<br>The graph then puts the two parts of each product into their place-value<br>columns:

for i in range(n):<br>for j in range(n):<br>product = onehot_lookup(<br>concat([seq1[i],...