Architecture · Hierarchy
Tiered Memory Hierarchy in AI Agents
Agent memory hierarchy tiers information across the context window (hot), fast stores (warm) and deep external memory (cold) — paging between tiers like an operating system manages RAM and disk.
OS analogy
Model
The three-tier model
L1 context window (hot) → L2 fast buffer (warm) → L3 vector/graph archival (cold).
| Tier | Store | Capacity | Latency | Cost |
|---|---|---|---|---|
| L1 Hot | Context window | 128K–1M tokens | Zero (in prompt) | Per-token inference |
| L2 Warm | Redis, session cache | MB–GB | Milliseconds | Low |
| L3 Cold | Vector DB, graph | Unbounded | 100ms+ search | Storage + embed |
Paging
Paging between tiers
Promotion on salience, demotion on eviction, recall on retrieval — the MemGPT OS analogy.
- Promote — durable facts move from L3 → L2 → L1 when repeatedly needed
- Demote — overflow evicts oldest L1 turns to L2/L3 archival
- Recall — semantic search pulls L3 segments into L1 before generation
Frameworks
Hierarchy in production frameworks
| Framework | Hot tier | Cold tier |
|---|---|---|
| Letta (MemGPT) | Core memory (in-context) | Archival memory (paged store) |
| Engram | Current turn in prompt | Weaviate memory collections |
| Mem0 | Context window | Vector LTM API |
| Zep | Retrieved graph facts | Graphiti temporal graph |
Design
Designing your agent’s hierarchy
A single L3 vector tier is enough for most personalization POCs; paging (Letta) wins when conversation history is the bottleneck.
- Single tier — Engram or Mem0 vector store + context window (simplest)
- Two tiers — Redis session cache + vector LTM (common production)
- Full paging — Letta core/archival when unbounded multi-session chat
FAQ
Frequently asked questions
How many memory tiers does an agent need?
Most agents need two: context window (hot) + external vector/graph store (cold). Add L2 Redis cache or Letta paging only when latency or unbounded history demands it.
Is tiered memory the same as human memory hierarchy?
Inspired by cognitive science and OS design, but implemented differently — software tiers (context, cache, vector DB), not biological structures. See AI memory vs human memory.
What are Letta's tier names?
Core memory (in-context hot tier) and archival memory (external cold tier). Agent tools page between them. See virtual context and MemGPT.
Which tier is Redis in agent memory?
L2 warm — fast session cache between context window and vector archival. Not a replacement for semantic LTM search. See Redis for agent memory.
Flat vector store vs tiered hierarchy?
Flat vector + context window suffices for most personalization. Hierarchy adds paging (Letta) or caching (Redis) when cost or history volume grows.
How do you benchmark tiered memory?
LOCOMO and LongMemEval on long-session recall. Measure tokens in L1 vs retrieval hits from L3. Mem0 reports ~1,800 vs ~26,000 tokens per query (Chhikara et al., 2025).
Engram memory hierarchy?
Current turn in context (L1) + Weaviate memory collections (L3). No MemGPT-style paging — simpler two-tier model. See Engram explained.
When to add a warm cache tier?
When repeated retrieval from cold storage adds latency — cache last N turns or top memories in Redis between sessions.