Evaluation hub · benchmarks
How to Evaluate AI Agent Memory
Evaluate AI agent memory on recall accuracy, retrieval latency and cost — using benchmarks like LOCOMO and LongMemEval plus production metrics — so framework choices are evidence-based.
What to measure
Why
Why evaluation matters
Neutral tool comparisons require benchmarks — not vendor marketing numbers. LOCOMO and LongMemEval are the most widely cited public benchmarks in agent memory.
→ Best AI memory tools (methodology)
Metrics
Key memory metrics
Accuracy / recall
Does the agent retrieve the right memory for a query?
Latency
Retrieval round-trip time per turn.
Cost
Prompt size and API cost with memory injected.
Your data
Running your own evaluation
Subset your queries
Pick representative user questions from production logs
Conflict cases
Test contradictory and time-changing facts
Compare frameworks
Same queries across Engram, Mem0, Zep, Letta, etc.
Before migration
Benchmark before switching memory backends
→ Add memory guide (evaluation step)
FAQ
Frequently asked questions
LOCOMO vs LongMemEval — what's the difference?
LOCOMO tests recall within very long single conversations. LongMemEval tests recall across separate sessions. Use both — they measure different production failure modes. See LOCOMO and LongMemEval.
Should I trust vendor benchmark scores?
Treat vendor numbers skeptically — datasets and tasks vary. We cite LOCOMO and LongMemEval because they're public and widely referenced. Always run a POC on your own data.
What are Mem0's benchmark scores?
See our best AI memory tools comparison table — scores cite published benchmarks with N/A where no public data exists.
How do I compare frameworks using public benchmarks?
Run the same query set on each framework and score against LOCOMO (long-conversation) and LongMemEval (cross-session) subsets. See our comparison table for published scores where available.
How do I run a DIY memory evaluation?
Create a query set from production logs, write ground-truth expected recalls, run retrieve → check hit rate. Add conflict and temporal cases. See add memory guide.
Can I integrate memory evals into CI?
Yes — run a fixed query suite on each deploy, track recall@k and latency regression. Start with a small golden set before scaling.
Latency vs accuracy — how do I trade off?
Tighter retrieval (more candidates, reranking) improves accuracy but adds latency. Profile p50/p99 and set SLAs per use case. See memory metrics.
How do I evaluate memory for customer support?
Test ticket history recall, user preference persistence and conflict resolution when account details change. Combine LOCOMO-style long threads with cross-session LongMemEval patterns.