LLMs Corrupt Your Documents When You Delegate - NotesByLex.com

Recent<br>Tags

Completing A Computer Science Degree On Coursera<br>Jul 5, 2026

Shipping a Laptop to a Refugee Camp in Uganda<br>May 23, 2026

Heavy Thinking: A Test-Time Scaling Pattern for Hard Problems<br>May 17, 2026

LLMs Corrupt Your Documents When You Delegate<br>May 9, 2026

AI-Induced Cognitive Atrophy<br>May 7, 2026

OpenGame: Open Agentic Coding for Games<br>Apr 29, 2026

Naming Things Is Easy Now<br>Apr 25, 2026

Obsidian Markdown Notebook: code execution with outputs stored in the file<br>Apr 21, 2026

Research, Plan, Implement Workflow<br>Mar 22, 2026

Self-Generated Agent Context Files Don't Help Either<br>Feb 26, 2026

See more →

MachineLearning<br>29

SoftwareEngineering<br>17

LinearAlgebra<br>16

LargeLanguageModels<br>13

GameDesign<br>12

AgenticReasoning<br>11

ComputerScience<br>11

AudioEngineering

LearningAndTeaching

Zettelkasten

DiscreteMath

AutomatedTesting

ReinforcementLearning

Roblox

DataStructures

Lua

StoryWriting

GameMath

LearningWithAI

KnowledgeManagement

See all 101 topics →

Home<br>/paper

LLMs Corrupt Your Documents When You Delegate

May 9, 2026

AgenticReasoning<br>LimitationsofLLMs

Notes on LLMs Corrupt Your Documents When You Delegate by Philippe Laban, Tobias Schnabel and Jennifer Neville.

An interesting paper from researchers at Microsoft.

They introduce a benchmark called DELEGATE-52 that tests whether LLMs can safely carry out, what they call, "long-delegated workflows" for document editing across 52 domains. Every set of instructions in the benchmark is lossless and reversible, allowing the authors to measure how much each task degrades the file's information over multiple interactions.

They found that even the strongest frontier models, including Gemini 3.1 Pro, Claude 4.6 Opus, and GPT 5.4 (the paper was released before their successors), corrupted an average of about 25% of document content after 20 interactions. Across all tested models, average degradation was about 50% (Laban et al., 2026).

Python was the main exception. It was the only domain in which most models met the paper&rsquo;s &ldquo;delegation-ready&rdquo; threshold, with 17 of 19 models scoring at least 98% after 20 interactions.

Figure 1 from (Laban et al., 2026) shows examples of document degradation across different domains. The benchmark itself is text-only; the visual renderings are illustrative.

Surprisingly, the degradation didn't happen gradually over instructions, but models would typically fail catastrophically after a certain number of steps. Stronger frontier models would fare better only by delaying the step at which the degradation occurs.

They also found that tool use did not prevent degradation. The tested models performed worse with tools, averaging an additional 6% of degradation.

Measuring Document Corruption

To measure document corruption, they introduce a domain-specific Document Similarity Measure that parses documents into components. For a recipe, that means ingredients (name, quantity, unit), steps, and tips; for Python code, it means functions, classes, and imports. This lets them compare two parsed documents based on their actual content, rather than just raw text. Typical document similarity measures might overlook seemingly small changes, such as 200g to 800g of butter, which can be really bad in a recipe, whereas a surface-level rewrite that preserves the underlying structure doesn't need to be heavily penalised.

Figure 5 from (Laban et al., 2026) - the domain-specific parsing pipeline, with a concrete recipe example showing how ingredients, steps and tips are extracted and compared

The approach of creating reversible transforms was inspired by Backtranslation, a machine translation technique in which text is translated into another language and then back, allowing the result to be compared with the original. DELEGATE-52 adapts that idea to document editing: apply a forward edit, apply the inverse edit, and compare the reconstructed document to the original. Imagine splitting a CSV into separate files by expense category, then merging them back together. Or converting all amounts in an accounting ledger to euros, then converting back.

They use a round-trip relay simulation method in which every task is assumed to be reversible, defined by a forward instruction and its inverse.

Seed Doc<br>2 cups flour<br>1 cup sugar<br>&frac12; tsp salt<br>— Mix dry<br>— Bake 350&deg;F

Original s

Forward edit<br>"&rarr; metric"🔄 Fresh context

Transformed<br>473 ml flour<br>237 ml sugar<br>2.5 ml salt<br>— Mix dry<br>— Bake 175&deg;C

Edited t

Backward edit<br>"&rarr; imperial"🔄 Fresh context

Reconstructed<br>2.01 cups flour<br>1 cup sugar<br>&frac12; tsp salt<br>— Mix dry<br>— Bake 350&deg;F

Reconstructed &scirc;

sim(s, &scirc;)

RS@2 = 97.3

Reconstruction<br>Score

Each LLM call is independent with no conversation history. Errors survive into the next round because they are baked into the document itself, not the context window.

It's worth checking out some examples in the GitHub repo, see...