The discussion about a paper on what happens when you leave document maintenance to an LLM

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

A little while ago, I wrote an article titled "The Pattern Called 'LLM Knowledge Base' Articulated by Karpathy." It described how I actually adopted the idea of passing raw materials to an LLM and having it compile them into a structured wiki, and how things started running smoothly once I did.

Right after publishing that article, someone shared with me on social media a paper from Microsoft Research titled "LLMs Corrupt Your Documents When You Delegate" (arXiv:2604.15597v1, Laban / Schnabel / Neville, April 17, 2026). As the title suggests, it seems to throw cold water on the idea of "delegating maintenance to an LLM," as Karpathy proposes.

Reading it, I found it wasn't a negation for the sake of negation, but rather a grounded observation: "just because it works well in the short term doesn't mean the degradation when left to run for a long period is negligible—and when measured, it turned out to be non-trivial." I wanted to organize the content and its implications, partly as an opportunity to pause and reconsider the idea I introduced last time.

The Paper's Argument in One Sentence

The paper's argument is simple.

Current LLMs are unreliable delegates: they introduce sparse but severe errors that silently corrupt documents, compounding over long interaction.

Specifically, across an average of 52 domains using 3 state-of-the-art models (Gemini 3.1 Pro / Claude 4.6 Opus / GPT 5.4), 25% of documents were corrupted after 20 interactions. An "interaction" refers to a single exchange in which an LLM edits a document, and the paper confirms that degradation does not stop even after extending the experiment to 100 rounds.

Looking at the actual measurements for all 3 models, the Reconstruction Score at 20 interactions is Gemini 3.1 Pro 80.9% / Claude 4.6 Opus 73.1% / GPT 5.4 71.5%.

Degradation curves for 3 models (Source: Laban et al. 2026, Table 1)

You can see that the picture looks quite different between short-term and long-term. At 2 interactions, every model sits at 94–97%, a passing grade by anyone's standard, but at 20 interactions they drop to 71–81%. The paper emphasizes this point—that "short-term tests cannot predict long-term performance"—and also calls for a reconsideration of benchmark design itself.

DELEGATE-52 and Round-Trip Evaluation

The experimental design behind these numbers is the other core of the paper.

The authors created a new benchmark called DELEGATE-52. It collects 52 specialized domains and a total of 310 working environments, ranging from code-related areas like Python, Docker, and Makefile, to crystal structures, musical notation, recipes, genealogies, and library catalogs. Each includes real documents of 2–5k tokens (actual documents from the web, not synthetic data) and 5–10 editing tasks.

For evaluation, they introduced a back-translation round-trip technique borrowed from the field of machine translation.

Forward  : t   = LLM(s; x→)
Backward : ŝ   = LLM(t; x←)
Score    : RS  = sim(s, ŝ)

The idea is to alternate between a forward edit ("split into files by category") and a backward edit ("integrate into one file in chronological order"), then measure whether the original document can be recovered at the end. An error-free model would yield sim(s, ŝ) = 1. Because no correct annotations are needed, it was possible to expand the benchmark to 52 domains.

I also found it to be a careful design that domain-specific parsers and weighted similarity metrics were implemented for each domain, in order to catch degradation that general-purpose similarity measures (like Levenshtein or embedding distance) would miss. For recipes, for example, there's a weighting of 40% for ingredients, 40% for cooking steps, and 20% for tips, with a mechanism that strictly detects numerical alterations like 200g → 800g.

Six Types of Degradation

There are six types of degradation that I felt couldn't be overlooked while reading the paper.

1. Compounding Errors

Deductions accumulate with each round and never converge, even in the long term. Even models that exceeded 95% in short-term tests decline linearly as the number of rounds increases.

2. Interaction Between Document Size and Number of Rounds

For a 1k-token document, even after 20 interactions the score stays at 91.4%. But for 4k it drops to 79.0%, and for 10k it falls to 59.9%. The paper, using GPT 5.4 as an example, shows a calculation where "the degradation per additional 1,000 tokens is 0.7% at 2 rounds and 3.6% at 20 rounds"—roughly a fivefold amplification.

Cumulative degradation progresses exponentially as document size increases (Source: Laban et al. 2026, Table 5)

3. Tool Use Does Not Improve Things—It Actually Makes Them Worse

When using an agent configuration with file read/write and code execution tools, the result was an additional 6-point deterioration at the end of the simulation, averaged across 4 models. Input tokens ballooned to 2.1–4.6 times their original size, exposing weaknesses in long-context processing. Intuitively, one might think "a Lint tool could fix this," but in the context of the paper it actually makes things worse.

4. Impact of Distractors (Irrelevant Documents)

When documents that are topically adjacent but not relevant to the task at hand are placed nearby, an initial impact of 0.4–4% grows to 2–8% by round 20. This is a property that becomes more pronounced as the wiki grows.

5. Extreme Disparity Across Domains

Of the 52 domains, even the top-performing Gemini 3.1 Pro only had 11 domains recognized as "ready (RS@20 ≥ 98%)." Python was within the passing range for nearly all models (17 out of 19 models were ready), but domains like crystal structures, musical notation, and genealogies fell short of passing for nearly all models.

6. "Alteration" Becomes Dominant Over Deletion

Weaker models tend to delete content, which is easier for humans to notice. State-of-the-art models, on the other hand, rewrite content, making it harder to spot anomalies even when reviewing a git diff. The "sparse but severe" phrasing used in the paper captures this feeling exactly.

Why Can't LLMs Themselves Notice the Degradation?

At this point, let me step back and think about the paper's numbers in terms of how LLMs fundamentally work.

LLMs are essentially models that "probabilistically predict the next token"; they are not editing documents with an understanding of their meaning. They are simply generating a statistically plausible continuation from the received context. When this is combined with document editing, it means that "plausible-looking" changes pass through, while tokens with high information content—like specific numbers or proper nouns—are more likely to fall into the valleys of the probability distribution and get quietly replaced with something else. The "sparse but severe" feeling described in the paper seems like the natural flip side of this mechanism.

What makes this particularly troublesome is that LLMs themselves have difficulty detecting this degradation. Even if you delegate Lint or self-review to an LLM, ultimately it's the same "probabilistically writing" engine doing the "probabilistic checking"—it has no internal means of comparing against a ground truth. The paper's finding that tool use did not lead to improvement also starts to seem natural, given that the fact of operating on probability distributions doesn't change.

This is precisely where the value lies in placing something like recompilable raw sources or a round-trip metric "outside the LLM."

Implications for the Karpathy Pattern

This is the main point. Does the paper's critique directly apply to Karpathy's LLM Knowledge Base pattern, or can it be structurally avoided? Let me try to work through this.

There are several points of direct impact. First, the mechanism in the Karpathy pattern of filing back Query responses—that is, writing answers back into the wiki to grow it—is precisely the "multi-round editing" that the paper measures for degradation. The phrase "LLM takes on tedious maintenance tasks" also, from the paper's perspective, reads as a somewhat unsettling flip side: "sparse modifications are being introduced during periods when humans are not monitoring." One might hope for a tool-based solution like Lint, but the paper doubles down with "6-point deterioration with tool use."

On the other hand, there are points worth qualifying. In Karpathy's 3-layer structure, raw / schema / wiki are separated, and the raw sources layer is kept immutable as a domain curated by humans. Because the design allows you to discard a corrupted wiki and recompile it from scratch, it's structured in a way that makes catastrophic loss less likely. For wikis whose subject matter leans toward domains the paper rates as "ready"—such as Python, infrastructure, and AI (17 out of 19 models passing)—it may be reasonable to be more optimistic than the 52-domain average. Also, the authors themselves acknowledge in their Limitations section that each step was run as an independent single-turn experiment, so it may not directly apply to a real-world workflow where you check git diffs while editing in multi-turn dialogue.

I don't think it's fair to use the paper as a tool to negate Karpathy. At the same time, having read the paper, I can't leave unchanged the somewhat vague sense that "it's fine to just delegate maintenance to an LLM."

Operational Ideas for Suppressing Degradation

Taking the paper's numbers at face value, it seems hard to continue with a "delegate everything to an LLM" approach. At the same time, combining safety mechanisms with the Karpathy 3-layer structure might cover things to a reasonable degree. Without going into implementation details, let me raise three possible directions.

Full Rebuild at Regular Round Intervals

An approach of discarding the entire wiki and recompiling from raw every N rounds. If 25% is corrupted at 20 rounds, at half that—10 rounds—you'd still expect somewhere around 10–15% degradation depending on the numbers. I think something like resetting to a clean slate every 5–10 rounds would be a safe margin. It should naturally fit as an extension of the 3-layer structure where raw sources are kept immutable.

Round Limit Guard for Filing Back

A mechanism that counts how many times responses have been written back into the wiki and triggers a forced rebuild when a threshold is exceeded. If you leave all the writing back entirely to the LLM, rounds can accumulate without you noticing, and before you know it the threshold has been crossed. The idea is to have the operational tooling keep watch.

Round-Trip Style Regression Check

Existing Lint tools focus on detecting broken links and contradictions, and don't go so far as to check whether "if recompiled from the same raw, the current wiki would be semantically the same." Bringing the paper's round-trip method directly into operations would be overkill, but something like "randomly sample the last 10 Queries and verify that the same answers come from both the current wiki and a newly generated wiki" seems practically feasible.

None of these can be confirmed to work without actually implementing them. Still, having the paper's numbers is helpful for knowing "what to measure" and "how frequently to rebuild."

Conclusion

After finishing the paper, my conclusion was not "LLM Knowledge Base is broken so let's stop," but rather: let's use it with regenerability and monitoring built in, on the premise that it will degrade. Karpathy himself wrote "intentionally abstract" in his gist, taking a stance of not imposing a definitive operational flow, so there is still plenty of room to build additional safety mechanisms on top of that.

The figure of 25% at 20 interactions is indeed heavy to bring into knowledge work. On the other hand, the fact that a benchmark has emerged that makes that loss visible is itself a sign that this paradigm is starting to be taken seriously as a subject of research. If you're going to delegate, grow the safety mechanisms to match. It's a perspective that seems obvious but is easy to forget—and this paper gave me a fresh reminder.

Going back and forth between the paper and the Karpathy gist, thinking about how to develop ways of thinking like LLM Knowledge Base. I'd like to keep thinking and keep building, going forward.

References

• Laban, P., Schnabel, T., Neville, J. (2026). LLMs Corrupt Your Documents When You Delegate. arXiv:2604.15597v1. https://arxiv.org/abs/2604.15597
• Karpathy, A. (2026). LLM Knowledge Bases (gist). https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f
• Previous article: The Pattern Called "LLM Knowledge Base" Articulated by Karpathy