Fundamentals · Disambiguation
AI Memory vs Context Window: What’s the Difference?
The context window is temporary working memory with a hard token limit; AI memory is persistent storage outside the window that agents read and write across sessions.
Context window
Volatile · token-limited · current session only
AI memory
Persistent · external store · cross-session
Definition
What is the context window?
The context window is the maximum number of tokens a model can attend to in one inference call — system prompt, history, tools and user message combined.
It is volatile: when the session ends, nothing persists unless you resend it. GPT-4 supports 128K tokens; Claude 3.7 Sonnet 200K; Gemini models exceed 1M tokens (Hurst et al., 2024; Anthropic, 2025; Team et al., 2024, as cited in Chhikara et al., 2025).
Definition
What is AI memory (in this comparison)?
AI memory is an external store plus a retrieval pipeline that persists beyond a single inference — writable, updatable and searchable across sessions.
It is not limited to one model call’s attention span. See the full definition on the AI memory homepage and the four-step loop in how AI memory works.
Side by side
Context window vs AI memory
| Dimension | Context window | AI memory | Combined |
|---|---|---|---|
| Persistence | Session only | Cross-session | Both |
| Capacity | Model limit (128K–1M+ tokens) | Effectively unbounded (external store) | Window + store |
| Cost model | Pay per token in window | Pay per embed + retrieve | Selective retrieval cuts cost |
| Update mechanism | Append to prompt | Extract → store → retrieve | Memory injects into window |
| Cross-session | No (unless resent) | Yes | Yes |
| Best for | Current turn reasoning | Durable facts, preferences | Production agents |
Why memory still matters
Why bigger context windows don’t replace memory
Cost scales with every token; context rot degrades distant tokens; there is no structured forget/update; sessions still end.
On LOCOMO, a 26,000-token full-context approach scores 72.9 LLM-as-a-Judge but costs 17.1 s p95 latency — Mem0’s selective retrieval scores 66.9 at 1.44 s p95, a 91% latency cut with >90% token savings (Chhikara et al., 2025). Bigger windows delay the problem; they do not solve persistence or personalization.
Integration
How context window and memory work together
Memory retrieves relevant facts → injected into the context window → model reasons on the combined prompt.
Letta (MemGPT) adds tiered paging between window and deep store for very long conversations. Typical production stack: external memory store + top-k retrieval + context window as working memory.
Taxonomy
Short-term vs long-term: where each lives
Short-term/working memory lives primarily in the context window; long-term memory lives in external stores.
Consolidation promotes facts from short-term to long-term between sessions.
Example
Worked example: support chatbot
Turns 1–3 live in the context window; customer history loads from memory each turn.
A customer with 10 past tickets (~50,000 tokens of history) cannot fit in a 128K window alongside system prompt and tools. Memory retrieves the 8 relevant facts (~800 tokens) — “prefers email contact,” “premium tier,” “open billing dispute from March” — instead of resending the full transcript.
FAQ
Frequently asked questions
Is the context window the same as memory?
No. The context window is volatile working memory for the current inference — it clears when the session ends. AI memory persists in external stores across sessions. See comparison table above.
What is working memory in AI agents?
Working memory is the information actively in the context window for the current task — recent turns, tool outputs and retrieved memories. It maps to short-term memory in the agent taxonomy. See short-term memory.
What does context memory mean?
Often confused with AI memory. 'Context memory' usually means information in the context window (working memory), not persistent cross-session storage. For durable memory, use an external store. See how AI memory works.
What is role context memory?
In agent frameworks, role context is the system prompt and persona instructions in the context window — not persistent user memory. Durable facts about users belong in external memory stores.
Does a 1M-token context window replace memory?
No. Even 1M-token windows are session-bound, costly per token and suffer context rot. Memory provides cross-session persistence, selective retrieval and structured update/forget. See long context vs memory.
How do I persist conversation beyond the context window?
Extract salient facts after each turn to an external memory store; retrieve top-k memories before each response and inject into the window. See persist conversation memory.
What is the difference between context engineering and memory?
Context engineering curates what goes into the window each turn. Memory is the external system that stores and retrieves facts across sessions. They work together. See context engineering.
Is Claude's memory the same as the context window?
Claude's product memory is a managed feature for that app. The API context window is still the working-memory limit per call. Custom agents need their own memory layer. See add memory guide.
How does Letta relate to the context window?
Letta (MemGPT) treats the context window like RAM and pages memories to a deep store — effectively unbounded history without stuffing every token into the window. See virtual context and MemGPT.
What is the best production pattern?
External memory store + selective retrieval into the context window. Validate with LOCOMO/LongMemEval on your domain. See add memory to an agent and evaluation hub.