Reducing Doom Loops with Final Token Preference Optimization — Blog — Liquid AI

Connect

h2]:clear-both [&>h3]:clear-both">A doom loop is a common failure mode during inference: the model emits a span (often something like “Wait, let me reconsider…”), then repeats the same span again and again, until the context window is exhausted. Small reasoning models are more prone to this behavior, especially on long thinking traces and hard problems [1].<br>The commonly applied inference-time fix is to apply repetition_penalty to reweight the output distribution. However, this is a band-aid solution and can degrade performance. Reinforcement learning can target repetitive looping, but it typically requires carefully calibrated rewards and costly online rollouts.<br>Our method takes a more targeted approach. We identify the exact token that begins a loop, train the model to prefer coherent alternatives at that single position, and leave the rest of the distribution largely untouched. The method adapts Antislop [2], training on chosen/rejected pairs that represent a single completion token, using Final Token Preference Optimization (FTPO). We call our approach “Antidoom”.<br>On an early checkpoint of LFM2.5-2.6B, 10.2% of completions on hard math and coding prompts produced repetitive loops. After Antidoom training, that rate fell to 1.4%, with eval scores improving across the board as a direct result of reduced looping.<br>Anatomy of a doom loop<br>Doom loops can arise in inference from three mechanisms working together:<br>Mechanism 1: Overtrained tokens + Uncertainty<br>Some tokens in the vocabulary are more likely to be selected in general. Well-known examples in the wild include "delve" and "testament". This may occur if synthetic data is used in the model's training set, creating higher distributions of these words than would ordinarily occur in human writing. In reasoning models, high-prior continuations often include discourse markers and self-reflection tokens such as “Wait” or “Alternatively.” These tokens are not necessarily bad and can mark a useful change of strategy, a verification step, or a branch in the reasoning trace. However, when the model is uncertain or stuck, they can become attractive fallback continuations, restarting the same local reasoning pattern instead of helping the model make progress.<br>For an early checkpoint of LFM2.5-2.6B, the most common tokens to begin a doom loop were:<br>count share token<br>2277 11.39% ' the'<br>902 4.51% ' So'<br>644 3.22% 'Alternatively'<br>511 2.56% 'Wait'<br>493 2.46% ' But'When the model is uncertain, these overtrained tokens dominate the next-token distribution, which appears to be why looping shows up most often inside reasoning traces for hard math and coding problems. Prior work [4,5] gives a similar account of degeneration: likelihood-trained models can overassign probability to repeats and frequent words, and reasoning models can loop under low-temperature decoding when they fail to identify a useful next step and instead fall back to repetition.<br>Mechanism 2: Prior context reinforces the loop<br>Earlier sequences make those same sequences more likely later. With each repetition, the probability of every token in the looping span climbs toward 1.

Duan et al. [6] study this looping in their work on circular reasoning. They link it to a "V-shaped" attention pattern and find that semantic repetition (the model getting stuck on an idea) precedes textual repetition (the same words showing up in the output).<br>Mechanism 3: Greedy sampling<br>Reasoning models are typically run at low temperature so that traces stay stable and reproducible. At temperature 0, the most likely token is always selected, and a locally reinforced loop has no exit. Higher temperatures help in theory, but once mechanism 2 has pushed the loop token's probability close to 1, there is almost no probability assigned to the remaining vocab, so sampling can still get stuck in loops at higher temperatures (our results show significant looping even at temp=0.67). The lower the temperature, the more looping is exacerbated.<br>Locating the failure<br>To build a targeted training set, we generate completions on a prompt mix designed to elicit looping (LiquidAI/antidoom-mix-v1.0) at low temperature, then mine the failures.<br>A loop is detected in a sample if a section repeats at least four times, over at least 60 characters. In practice, these constraints help avoid false positives and false negatives. Once the looping sequence is identified, we target the first token of the first repeat.

At that position, we take the base model's top-k log-prob alternatives, filter out short or non-alphanumeric noise, and keep up to 20 plausible substitutes as chosen tokens. Each training row is comprised of a [prompt prefix, one rejected token, one or more chosen tokens] tuple. We then regularise the rejected and chosen distributions before training: a small set of culprits (Wait, So, the) would otherwise dominate, and over-suppressing them degrades...