Evaluation · Benchmark
The LOCOMO Benchmark for Agent Memory
LOCOMO benchmarks long-conversation memory — testing whether agents recall facts from thousands of tokens earlier in the chat — and is the standard citation for comparing Engram, Mem0, Zep, Letta and other memory frameworks.
LOCOMO tests
Definition
What LOCOMO measures
Long-dialogue recall tasks — question answering on facts buried in distant context within a single long conversation.
LOCOMO (Long Conversation Memory) tests whether agents remember what was said thousands of tokens earlier in the same chat. It is the primary benchmark for in-session memory depth — distinct from cross-session evals like LongMemEval.
Scoring
How LOCOMO scores work
LLM-as-a-Judge (J) accuracy on memory probes — higher is better.
The Mem0 paper (Chhikara et al., 2025) reports overall J scores across LOCOMO tasks. The same evaluation run scored the full-context baseline at 72.9 J — selective memory frameworks can beat raw long-context on efficiency while approaching or exceeding recall quality.
Also reported: retrieval latency (Mem0 median search 0.148 s) and token usage (~1,800 vs ~26,000 per query).
Results
Framework results on LOCOMO
Last updated: July 2026. N/A = no public result — we never estimate.
| Framework | LOCOMO J | Source | Latency (LOCOMO eval) |
|---|---|---|---|
| Engram | N/A | GA June 2026 — pending | N/A |
| Mem0 | 66.9 | Chhikara et al., 2025 | 0.71 s p50 / 1.44 s p95 |
| Zep | 66.0 | Rasmussen et al., 2025 (Mem0 paper eval) | 1.29 s p50 / 2.93 s p95 |
| LangMem | 58.1 | Chhikara et al., 2025 eval | 18.5 s p50 |
| Letta (MemGPT) | N/A on LOCOMO | DMR 93.4% (Packer et al., 2023) | N/A |
| Full-context baseline | 72.9 | Chhikara et al., 2025 | 17.1 s p95 |
Comparison
LOCOMO vs LongMemEval
LOCOMO = long single conversation; LongMemEval = cross-session recall with temporal gaps.
Use both: LOCOMO for in-session depth; LongMemEval for multi-session persistence. Zep reports LongMemEval 71.2% with gpt-4o (+18.5% vs baseline, Rasmussen et al., 2025).
How-to
How to run LOCOMO on your agent
- Obtain the LOCOMO dataset
- Integrate your memory framework (Engram, Mem0, Zep, etc.)
- Run the eval harness with your LLM backend
- Compare J scores, latency and token usage across configs
- Gate production deploys on recall@k regression
FAQ
Frequently asked questions
What is the LOCOMO benchmark?
LOCOMO tests long-conversation memory — whether agents recall facts from thousands of tokens earlier in the same chat. Standard eval for comparing memory frameworks.
Who published LOCOMO?
LOCOMO is widely cited in agent-memory research. The Mem0 paper (Chhikara et al., 2025) publishes comparative LOCOMO J scores for Mem0, Zep, LangMem and full-context baselines.
What is Mem0's LOCOMO score?
LOCOMO J 66.9 (Chhikara et al., 2025). Median search 0.148 s; ~1,800 vs ~26,000 tokens per query.
What is Letta's LOCOMO score?
N/A on LOCOMO publicly. The MemGPT paper (Packer et al., 2023) reports 93.4% on Deep Memory Retrieval (DMR) — a different benchmark.
LOCOMO vs LongMemEval?
LOCOMO = long single conversation recall. LongMemEval = cross-session recall with temporal gaps. Use both. See LongMemEval.
Can you pass LOCOMO with long context only?
Full-context baseline scored 72.9 J but uses ~26,000 tokens and 17.1 s p95 latency vs Mem0's ~1,800 tokens and 1.44 s p95 (Chhikara et al., 2025).
What is Engram's LOCOMO score?
N/A publicly as of July 2026 — Engram reached GA in June 2026. See Engram explained.
Where is the LOCOMO leaderboard?
No single official leaderboard — scores are published in framework papers. We track published results on best AI memory tools.