TabBench-LLMJules [ʒyl]

SearchSearch

Dark modeLight mode!a.isFolder&&!b.isFolder||a.isFolder&&b.isFolder?a.displayName.localeCompare(b.displayName,void 0,{numeric:!0,sensitivity:\"base\"}):!a.isFolder&&b.isFolder?1:-1","filterFn":"node=>node.slugSegment!==\"tags\"","mapFn":"node=>{node.displayName=node.displayName.charAt(0).toUpperCase()+node.displayName.slice(1)}"}">Content

TabBench-LLM<br>Jul 29, 20265 min read<br>research<br>ml

TabBench-LLM evaluates large language models as few-shot, in-context tabular classifiers , head-to-head with two baselines. It adapts TabArena, a general benchmark for tabular ML, to the LLM setting.

The primary suite is 19 deterministic synthetic tasks whose feature and class names carry no real-world meaning, so the decision rule can only be inferred from the rows in the prompt, never recalled. A secondary suite of 38 public TabArena-v0.1 datasets checks whether that carries over to real tables. Untuned Random Forest and TabPFN v2 fit the identical splits.

/not-a-feature/TabBench-LLM

Download all figures

Elo is a pairwise rating over shared datasets, calibrated so Random Forest = 1000. It<br>compares models only on targets they both attempted, but cannot correct selection bias when<br>the missing targets are systematically harder — so the headline is released only once every<br>model has completed the same scheduled synthetic block.

Methodology

Per dataset we draw a stratified training table of 10, 20, 50 or 100 rows and a fixed held-out test set, shared by every model. Each LLM request carries the labelled table and exactly one unlabelled test row, and asks for a single class token. Real datasets additionally compare visible class names with opaque tokens; synthetic targets are meaningless already, so only their opaque arm is run.

linear

a hyperplane over 8 columns<br>xor

XOR of two of 4 columns<br>rings

radial rule in 2 of 6 columns

class A

class B

Conceptual sketch of the synthetic suite. Three of the 19 recipes,<br>drawn in their two informative dimensions. The real tasks live in 6–64 columns<br>named x1..xd with opaque class tokens. Dashed lines mark the true decision<br>rule; points on the wrong side are the recipe's label noise.

Discussion

LLMs are rarely used for tabular prediction tasks, nor are they often evaluated on them. The two reinforce each other and neither is based on much published evidence.<br>There are several things standing in the way.

Firstly, consider what a caller actually gets back. The output is a token, not a probability .<br>Threshold tuning, ranking metrics and cost-sensitive decisions require a calibrated score. Verbalised confidences are poorly calibrated, logprobs are not always exposed and answers must be parsed before scoring. Predictions are also unstable: shuffling demonstrations, renaming classes or resampling can change them. There are no feature importances or reproducible artefacts, deployment sends rows to a third-party API, and hosted models can change or disappear.

Then the input side. Numbers arrive as text : a tokeniser splits 0.4931 into fragments and must reconstruct scale, ordering and distance, while a tree uses the value directly. CSV, JSON, Markdown, feature order and precision can all affect the score. A baseline sees the matrix; the LLM sees one arbitrary rendering, fixed here across models but not across the literature.

The regime also does not scale : in-context learning is limited by prompt length, inference needs one request per test row and one headline model takes ~9,500 requests, while Random Forest and TabPFN v2 fit the same splits in seconds on a laptop. The 100-example setting is therefore the practical comparison. At this cost, an LLM must be clearly better, not merely comparable, to earn a place in a pipeline.

Underneath it all lies a measurement problem. Standard tabular datasets have been online for years , including targets, so recall is indistinguishable from inference. The stronger results on TabArena datasets are therefore likely helped by memorisation. The synthetic suite and opaque class tokens reduce that risk, although TabPFN has a slightly unfair advantage there: the data is generated from priors similar to those used in its training.

None of this required anyone to decide that LLMs should be excluded. Weak expected results, high cost and an ambiguous protocol were enough on their own. Running the benchmark replaces that inference with a number: how far off they are and on which kinds of structure.

Reading the tables:

Elo - pairwise rating, calibrated so Random Forest = 1000.

Score - min–max normalized macro-F1 per dataset (best model = 1, worst = 0).

Improvability % - mean relative gap to the best model per dataset; lower is better.

Rankings can use any recorded metric via the selector.

🎁 Donate tokens

The code and the experiment design are ready; hosted frontier models still need API<br>quota. Credits, higher rate limits, or GPU hours let us add models without shrinking<br>the common evaluation block.

What your...