The Deadline Dividend

Analysis · Test-time compute · AI execution<br>The Deadline Dividend<br>What happens when a faster model has to keep working until the answer is due?<br>2,783 words13 min read28 linked sources<br>Read the source notes ↓<br>In this essay<br>In 1946, ENIAC weighed 30 tons and occupied a gymnasium-sized room. Fifty years later, students at the University of Pennsylvania reproduced its architecture and capacity on a chip small enough to balance on a fingertip. That became computing’s familiar story: fit the same work into a smaller machine. Penn’s ENIAC retrospective

Greater density opened a different path. Engineers could keep the room and ask: how much computation can we fit inside it? In 1996, ASCI Red became the first massively parallel computer to exceed one teraFLOP. Internet companies later built a different kind of room-sized computer: a warehouse packed with coordinated servers. In 2009, Google engineers called the data center itself a massive “warehouse-scale computer.” As computation took less room, builders could spend the space they saved on still more computation. DOE on ASCI Red Google on the warehouse-scale computer

Give two endpoints ten seconds to answer the same hard question. One uses most of the interval; the other replies almost at once. The interface feels “blazingly fast,” but both endpoints still face the same deadline.

Suppose one second of the ten goes to overhead and the answer requires 500 tokens. At 159.3 tokens per second, the first endpoint can generate about 934 reasoning tokens before answering. At 1,841.5 tokens per second, the second can generate 16,074. Calculation: under constant-rate extrapolation, the synthetic reasoning budget grows 17.2 times. Artificial Analysis measured the rates. The scenario adds a one-second overhead and a fixed deadline, then assumes extra tokens do useful work. Source rates

Calling the endpoint “blazingly fast” conflates speed with latency. Speed counts output tokens per second; latency counts time from request to useful result. Fast inference offers the same choice within a deadline. A faster decoder can return fixed work sooner or spend the time it saves on more computation before the deadline. Call the useful extra work that fits before the same deadline the deadline dividend .

What the speed test proves

Artificial Analysis lists 17 gpt-oss-120b “high” endpoints under a 10,000-token prompt and rolling 72-hour medians. Fact: Cerebras produced 1,841.5 output tokens per second. SambaNova reached 705.7 and Groq 477.6; the median of the 15 endpoints reporting speed was 159.3. Calculation: Cerebras ran 2.61 times as fast as the next listed endpoint and 11.56 times as fast as the median. The comparison also recorded Cerebras’s first chunk at 0.52 seconds, first answer token at 1.60 seconds, and modeled 500-token response at 1.87 seconds. Provider comparison

Those measures describe different clocks. Time to first token includes request handling, queueing, prefill, and initial generation. Inter-token latency measures the gap between decoded tokens; its inverse gives the displayed tokens per second. End-to-end latency ends when the useful result finishes. Throughput counts work across the fleet, often with batching, while single-user speed follows one request. Price, energy, reliability, model support, and answer quality require separate measurements.

Artificial Analysis times one request at a time, starting after the first chunk. That produces a single-stream decode rate. When each token waits on the previous token, this rate determines how much generation fits before the deadline. Performance methodology

Spending the dividend

A model can spend compute after a prompt arrives to pursue a better answer. Researchers call that post-prompt work test-time compute , or TTC. Intrinsic TTC extends one serial reasoning path with more tokens, revisions, or recurrent passes; every step depends on the one before it. Extrinsic TTC surrounds the model with parallel samples, agents, tools, critics, verifiers, and a selection step. Two hundred and fifty candidates may run together, but still consume 250 trajectories and require a reliable selector.

The immediate opportunity starts with deployed models. Give a deployed model spare seconds for a longer path, more candidates, or a check. If the extra work catches an error, the system handles that task better without retraining.

Scaling changed AI because it offered a simple recipe: more parameters, data, and training compute drove loss down predictably across tested regimes. Each larger training run consumes more scarce inputs. Fact: DeepMind’s compute-optimal experiments found model size and training tokens should grow together. Epoch AI judges larger runs feasible through 2030, while naming power, chip production, data, and training latency as constraints. OpenAI’s scaling-law paper Chinchilla Epoch AI

The longer-run bet is that models can be trained to use larger inference budgets well. Pretraining...