Context Rot: Why Stuffing More Into Your Prompt Can Make Things Worse

It's tempting to treat a large context window as a dumping ground, paste in the whole document, the whole chat history, the whole knowledge base, and let the model sort it out. Research from Chroma (July 2025) shows why that instinct backfires. Testing 18 leading LLMs, including GPT-4.1, Claude 4, and Gemini 2.5, across controlled tasks that isolated input length as the only variable, they found accuracy degrades steadily as context grows. Most of the time well before a model hits its advertised limit.

A few findings worth internalizing if you build with LLMs:

Position matters: Relevant information placed in the middle of a long prompt is retrieved less reliably than information near the start or end. This is the effect "lost in the middle" first documented by Liu et al. (2024) and confirmed again here.

Distractors hurt more than you'd expect. Even a single irrelevant passage mixed into an otherwise clean prompt measurably reduces accuracy, and the effect compounds as more get added.

Structure can backfire. Counterintuitively, prompts with a clear logical flow sometimes performed worse than the same content shuffled.

The practical takeaway: treat your context window's capacity as a ceiling, not a target. Curate aggressively: keep only what's directly relevant, front-load or end-load the most important instructions and facts, and summarize or drop the rest. This is the core idea behind "context engineering," and it's quickly becoming as important a skill as prompt writing itself.

References

Chroma. (2025, July 14). Context rot: How increasing input tokens impacts LLM performance. Chroma Research. https://www.trychroma.com/research/context-rot

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638