Evaluation · Metrics
AI Memory Metrics: Accuracy, Latency and Cost
The metrics that matter for agent memory are recall accuracy, retrieval latency and token/storage cost — track all three because optimizing one in isolation misleads.
Three pillars
Accuracy
Recall and accuracy metrics
Recall@k, probe accuracy and benchmark alignment — LOCOMO and LongMemEval as standard evals.
- Recall@k — fraction of relevant memories in top-k retrieval results
- Probe accuracy — can the agent answer questions requiring stored facts?
- LOCOMO J score — Mem0 66.9, Zep 66.0, LangMem 58.1 (Chhikara et al., 2025 eval)
- LongMemEval — Zep +18.5% vs baseline with gpt-4o at 71.2% (Rasmussen et al., 2025)
Production: maintain golden Q/A sets per domain; run eval in CI on each deploy.
Latency
Latency metrics
p50/p99 retrieval latency, end-to-end turn latency and embedding time.
Mem0 on LOCOMO: median search 0.148 s, p95 total 1.44 s vs 17.1 s full-context (Chhikara et al., 2025). Support bots often target sub-200 ms retrieval p99.
Cost
Cost metrics
Tokens per turn, embedding API cost, storage GB and managed API pricing.
- Tokens per query — Mem0 ~1,800 vs ~26,000 full-context (Chhikara et al., 2025)
- Embedding cost — per-write API calls at scale
- Storage GB — vector dimensions × memory count
- Managed API — per-call pricing vs self-host infra
Observability
Observability stack
Log every write and retrieval; alert on recall@k drops; trace IDs per conversation.
- Log injected memories per turn (context engineering audit)
- Dashboard: recall@k, latency p50/p99, tokens per query, store size
- Alert when recall@k drops below threshold vs CI baseline
- Gate deploys on LOCOMO regression in staging
FAQ
Frequently asked questions
What is the most important agent memory metric?
Recall accuracy (can the agent retrieve the right facts?) — but track latency and cost together. Optimizing tokens alone can tank recall@k.
Benchmark metrics vs production metrics?
LOCOMO/LongMemEval for CI gates. Production golden Q/A sets for domain-specific recall. Run both — benchmarks don't cover your CRM schema.
What latency SLA for agent memory retrieval?
Support bots often target sub-200 ms p99 retrieval. Mem0 median search 0.148 s on LOCOMO (Chhikara et al., 2025).
How do you benchmark memory cost?
Track tokens per query before/after memory adoption. Mem0 ~1,800 vs ~26,000 tokens per LOCOMO query (Chhikara et al., 2025).
What metrics does Mem0 report?
LOCOMO J 66.9, median search 0.148 s, p95 total 1.44 s, ~1,800 vs ~26,000 tokens (Chhikara et al., 2025).
What metrics does Engram report?
N/A for LOCOMO/LongMemEval publicly as of July 2026. Track recall@k, retrieval latency and token injection on your Weaviate deployment. See Engram explained.