Creating a niche AI Benchmark with token anxiety — Thijs Brits

Creating a niche AI Benchmark with token anxiety

4 Aug 2026

30 Seconds Bench on GitHub

Why another AI benchmark?

The AI benchmarks that we look at most focus on how well models can code, how good they are at math/science etc.

But niche benchmarks are fun. Take skatebench (guessing the skateboard trick by textual explanation of how the trick works) and vendingbench (AI running a vending machine company simulation).

While the main benchmarks have the goal of measuring how AI is improving. More niche, and at first glance somewhat ridiculous, benchmarks can be a creative exploration of what models can and cannot do. I think we need more of them. We might stumble into the more unexpected or just have some fun creating.

Let's build one.

Ideas come from constraints

I have one real-life constraint you probably also feel right now, especially if you are the kind of person reading about some random AI benchmark.

I spend too much on AI stuff.

Subscription max at one place, some smaller ones I keep forgetting to cancel, still have some unused credits on that platform etc.

This AI stuff is so exciting, tiring, and expensive all at the same time. But I can't stop playing around with it.

What if I try to make a benchmark that is cheap to run. Or even better, has incentive to not output a lot of tokens?

Designing for token anxiety

Simplest form are the exam like benchmarks: 1 prompt problem and evaluate the answer to true or false.<br>Great simple setup to build a niche bench on. You can pour all your creativity in the questions itself but I wanted something that messed with this structure a little bit.

Tokens come in 1 by 1, with the output speed of LLMs you would almost forget that. It is only visible on the API level if you turn on streaming. So what if we focus on a per token granularity of evaluation. I haven't seen many benchmarks have been done with an individual token by token dynamic. Let's do that! Let's anti-tokenmax, tokenminimize, tokenmin... you know what I mean.

The game: explain, stream, guess

The idea: A secret word guessing game.

We have 2 models working together as a pair, an explainer, and a guesser.

The explainer model gets a secret word to explain, streams its explanation of the word token by token, and the guesser model can, after every added token, guess the word it's trying to describe. The guesser model can either guess a word, or stay silent and wait for more tokens.

The explainer model is obviously not allowed to output the secret word. However, as we will see that is not always obvious to the models.

Keeping the prompts small

I tried to keep the system prompts as minimal as possible of course, token stingy as we try to be.

System prompt for the explainer model:

You are playing 30 seconds the word game.<br>You are explaining the word to the user.<br>Start your explanation right away.<br>Do not mention the word, part of the word or a translation of the word in your explanation. If you do you lose.<br>Your goal is to help them guess the word in as little time as possible.

The "30 seconds" in the system prompt is a reference to a Dutch game that is a similar secret word guessing game which helps me not having to explain the game as much to the models: We are tokenminning right?

Also at first I had an even more minimal system prompt by leaving out the emphasis on not mentioning the secret word. However, first version there were a lot of models leaking the word itself by prefixing their explanation start with Bombastic: {continued actual explaination of the secret word...}.

System prompt for the guesser model:

You are playing 30 seconds. Guess the word from the explanation so far.<br>You must call the react tool every turn: action=pass if you need more explanation,<br>or action=guess with one word if you have a guess.<br>Be fast, but every guess distracts your teammate, so pass when unsure.<br>Don't guess the same word multiple times; your teammate only confirms a correct guess.<br>So if you already guessed a word, that is not the correct word.

In the guesser prompt it's visible that I used a tool-call for the guess, I tried without it before, just giving it instructions to output 1 word or ... for passing. But this had its downside as the models often responded with things like:

My guess is: {word} or tried to pass with something more than just ... for a pass which meant I would have to hack around and use string matching to see if it had the intention to actually guess or tried to pass.

But tool-calling was also not perfect as Mistral tried to guess the tool name: submit itself as a guess... And overall it increased the token use of the benchmark.

On tokenizers

Now I realize 1 token of 1 model is not equal to another model token. And we will partly rank the influence of tokenizers. But for this simple bench I think this is fine to do.

Picking the models, picking the words

I took the top open source models from Artificial...