Advanced · Degradation
Context Rot: Why Long Context Degrades
Context rot is the accuracy drop as prompts grow — models miss or misweight information buried in long contexts — which is why selective memory retrieval beats stuffing everything into the window.
Long context
Accuracy ↓ as tokens ↑
Selective memory
Top-k relevant facts only
Definition
What is context rot?
Degraded recall and reasoning as context length increases — the “lost in the middle” and needle-in-haystack failures.
Even with million-token windows, models struggle to find facts buried thousands of tokens back. LOCOMO benchmarks this directly — testing recall from distant context in long conversations. Full-context baselines score 72.9 J but use ~26,000 tokens per query (Chhikara et al., 2025).
Mechanism
Why context rot happens
- Attention limits — not all tokens receive equal weight in long sequences
- Lost in the middle — information in the center of long contexts is retrieved less reliably
- Interference — newer messages compete with older facts for attention
- Training distribution — models see shorter contexts more often during training
Mitigation
How memory mitigates context rot
Retrieve only relevant slices; summarize history; page with Letta — don’t send everything.
- Selective retrieval — Engram/Mem0 top-k facts (~1,800 tokens vs ~26,000 full-context, Chhikara et al., 2025)
- Summarization — compress old history, keep recent messages verbatim
- Letta paging — core memory in-context, archival paged on demand (MemGPT DMR 93.4%, Packer et al., 2023)
- Context engineering — rank and budget what enters the window each turn
FAQ
Frequently asked questions
Is context rot a real phenomenon?
Yes — accuracy degrades as context grows. LOCOMO benchmarks long-conversation recall; 'lost in the middle' research documents the effect across models.
Do 1M token windows fix context rot?
No — larger windows increase cost and latency without solving attention dilution. Mem0 uses ~1,800 vs ~26,000 tokens with better efficiency (Chhikara et al., 2025).
What LOCOMO evidence exists for context rot?
LOCOMO tests recall from distant context in long chats. Full-context baseline 72.9 J but 17.1 s p95 latency vs Mem0 1.44 s p95 (Chhikara et al., 2025).
Is summarization enough to prevent context rot?
Helps but can lose fine-grained facts. Best pattern: summarize + selective memory retrieval. Validate recall@k after compression.
Does Mem0 help with context rot?
Yes — Engram and Mem0 both retrieve only relevant facts (~1,800 tokens) instead of full history (~26,000), reducing context rot. Choose Engram for Weaviate-native stacks.
Does Letta paging help with context rot?
Yes — MemGPT pattern pages archival memory in/out. DMR 93.4% (Packer et al., 2023). See Letta alternatives.