Architecture · Retrieval
Memory Retrieval in AI Agents
Memory retrieval is how agents find the right stored information at inference time — using semantic search, recency, graph traversal and relevance scoring to inject only what matters into the context window.
Retrieve path
Pipeline
The retrieval pipeline
Query formation → candidate generation → ranking → injection into prompt → LLM generation.
Production agents run retrieval before every response and write after. The retrieve path is the cost and quality bottleneck: bad retrieval wastes tokens; good retrieval beats resending full history.
Vectors
Semantic vector retrieval
Embed the query → similarity search top-k → filter by user ID and metadata.
Default for Engram, Mem0 and DIY vector stores. Strength: paraphrase tolerance. Limit: misses exact IDs and can surface stale but similar memories. Mem0 median search latency 0.148 s on LOCOMO eval (Chhikara et al., 2025).
Ranking
Recency and importance signals
Combine vector similarity with time decay and explicit importance scores — not similarity alone.
Park et al. (2023, Generative Agents): score = weighted sum of recency + importance + relevance, with recency decaying exponentially at 0.995 per hour since last access. Worked example: fresh core preference (0.9 + 0.9 + 0.8 = 2.6) outranks week-old small talk (0.3 + 0.2 + 0.6 = 1.1).
Hybrid
Hybrid and graph retrieval
Keyword + vector hybrid catches exact IDs; graph traversal follows entity relationships and temporal validity.
Hybrid search helps when memories contain order numbers, SKUs or names vectors miss. Zep Graphiti combines graph traversal with semantic search for temporal fact queries (Rasmussen et al., 2025). LongMemEval accuracy 71.2% on gpt-4o with temporal updating.
Disambiguation
Retrieval vs RAG retrieval
Same vector technology, different data — memory retrieval is personal and dynamic; RAG retrieval is static document corpus.
Pattern: docs = rag.search(query) + memories = memory.search(query, user_id) → separate prompt sections. Memory answers “what did this user tell me?”; RAG answers “what does the manual say?”
Injection
Injection and context engineering
Format retrieved memories for the prompt under a token budget — typically top-k bullets under ## User memories.
Tune top_k (usually 5–10). Order by score descending. Truncate low-value memories first when budget is tight. Mem0 uses ~1,800 tokens per LOCOMO query vs ~26,000 full-context — over 90% fewer (Chhikara et al., 2025).
Frameworks
Framework retrieval implementations
- Engram — Weaviate semantic search on memory collections
- Mem0 —
memory.search(query, user_id)API - Zep — Graphiti graph query + hybrid search
- Letta — archival recall functions (MemGPT paging)
- LangMem —
store.search()with LangGraph checkpointer
FAQ
Frequently asked questions
How many memories should agents retrieve per turn?
Typically top-k = 5–10 memories, tuned on your eval set. Too few misses context; too many dilutes attention and raises token cost. See injection section above.
How do you tune top-k for memory retrieval?
Run LOCOMO or LongMemEval on your domain with k = 3, 5, 10, 20. Pick the smallest k that maintains recall. Mem0's published eval uses selective retrieval vs 26,000-token full context (Chhikara et al., 2025).
What is typical memory retrieval latency?
Mem0 reports median search latency 0.148 s and p95 total latency 1.44 s on LOCOMO vs 17.1 s full-context baseline (Chhikara et al., 2025). Latency depends on index size and embedding model.
What if the wrong memory is retrieved?
Improve scoring (recency + importance + relevance), add metadata filters (user_id, memory_type), use hybrid search for exact matches, or invalidate stale facts. See memory scoring.
When should you use hybrid search for memory?
When memories contain exact IDs, SKUs, order numbers or names that pure vectors miss. Combine BM25/keyword with embedding similarity. See hybrid search.
When is graph retrieval needed?
When facts involve relationships, temporal validity or conflict resolution — CRM timelines, policy versioning. Zep Graphiti is built for graph retrieval. See knowledge graphs.
How do you benchmark memory retrieval?
LOCOMO and LongMemEval measure conversation recall. Track hit rate, latency and tokens injected per query. See evaluation hub.
How does Engram retrieve memories?
Semantic search on Weaviate memory collections — same platform as RAG vectors, separate namespaces. See Engram explained.
Retrieval before or after writing memory?
Retrieve before generate (agent needs context). Write after generate (extract new facts). Standard sequence: retrieve → generate → write.
Memory retrieval vs RAG retrieval?
Same vector tech, different stores — memory is per-user dynamic facts; RAG is static org documents. Run both pipelines in production agents. See memory vs RAG.