I Tried to Break Google's New Tabular Foundation Model. Then I Fixed It. - Yash Raj Pandey

TL;DR

Google Research released TabFM on 2026-06-30, a zero-shot foundation model for tabular data: no training, no tuning, you hand it your table and it predicts. The claim is that a single pre-trained model can match or beat tuned gradient-boosted trees out of the box. That is a big claim, so I ran an independent reproduction across three machines.

What I found, in one breath:

On small-to-mid tables, TabFM beat a properly Optuna-tuned XGBoost on all 10 fold-matched datasets , zero-shot. On the anchor dataset the gap actually widened under tuning (0.877 vs 0.821 accuracy).

But I held myself to the same bar I’d hold the authors to. A multi-seed check forced me to demote two “wins” to ties because the margins were smaller than measurement noise.

The advertised “22.75 GB GPU footprint” turned out to be mostly an XLA allocator artifact ; the real footprint is ~16.95 GB, and disabling preallocation roughly doubled the usable context (from ~10k rows to ~20k).

Along the way I found a bug that crashes predict on any machine with two or more GPUs . I root-caused it, wrote a fix and a regression test, and the PR was merged into google-research/tabfm by one of TabFM’s own authors .

Everything below is seeded, pinned, and reproducible: github.com/devYRPauli/tabfm-evaluation

How this started

I read the TabFM launch post the day it went up. The pitch is genuinely interesting: tabular data is the one domain where deep learning has repeatedly lost to gradient-boosted trees, and here was a foundation-model approach claiming to close that gap with zero training on your data. It learns in-context, the way a large language model does, except the “context” is your training rows.

My first instinct with a claim like that is not to believe it and not to dismiss it, but to reproduce it. Benchmarks in papers and blog posts are run by people who want their method to look good; that is not dishonesty, it is gravity. An independent reproduction is the only way to know which parts survive contact with someone who has no stake in the outcome.

So I set three rules for myself before writing a line of code:

The source code is ground truth, not the blog post. Where they disagree, the code wins.

Every result is seeded and reproducible. No cherry-picked runs.

Negative results are results. If it loses, or ties, or fails, that gets reported as plainly as a win.

That third rule turned out to matter the most.

What TabFM actually is (read from the code, not the marketing)

Before benchmarking anything, I wanted to know exactly what I was testing. A few facts I verified directly from the source:

It is in-context learning, not training. Calling fit() does not update the network. Your training rows become context for a single forward pass at predict time. This is why it is “zero-shot.”

The default forward pass is a 32-member ensemble. That detail matters enormously for the memory story later.

There is a hard cap of max_classes = 10. Good to know before you throw a 20-class problem at it.

Two weight repos exist. The JAX Orbax checkpoint is canonical; the PyTorch one is a converted state_dict that I validated matches within 1e-4. I used the JAX backend.

Two licenses. The code is Apache-2.0; the pretrained weights are non-commercial. They govern different artifacts, which is easy to miss.

Pinned stack for the whole study: commit b6ea70b, JAX 0.10.2, Flax 0.12.7, Python 3.12.

The setup: three machines, and one that I set on fire

I ran this across three machines, each with one job:

Machine<br>Spec<br>Role

MacBook Pro<br>M1 Pro, 16 GB<br>Orchestration only

Mac Studio<br>M4 Max, 64 GB<br>CPU reference

Workstation<br>Threadripper PRO 5955WX, 128 GB, 2x RTX 4090<br>GPU reference

The MacBook is “orchestration only” for a reason I learned the hard way. Early on I ran the full model on the 16 GB laptop. The 32-member ensemble needs a ~17 GB working set. macOS tried to cover the gap with swap, wrote about 39 GB of it, and the machine restarted itself.

That incident produced the most useful piece of infrastructure in the whole project: a safe_run.sh watchdog that preflights free memory and disk, then kills any job before its resident set, system swap, or free disk crosses a limit. A killed job is an acceptable outcome; a machine restart is not. Every heavy run after that went through it. The lesson generalizes: when you are probing the limits of a model’s resource use, build the kill-switch before you need it.

Phases 0 to 2: does it even reproduce?

Before the headline benchmark, three checks:

Phase 1 - conformance and determinism. The scikit-learn contract holds, and predictions are bit-exact deterministic under a fixed seed, on both the Studio CPU and the 4090 GPU. CPU and GPU agree numerically. Good - a model whose outputs drift run-to-run is not one you can benchmark honestly.

Phase 2 - known-answer sanity. Before trusting it on real data, I checked it can learn...