How to Evaluate AI Agent Memory

What does evaluating agent memory actually mean?

Measuring whether the memory system changed the agent’s answers for the better, which is a different question from whether the agent performed well. An agent can score highly with no memory at all, and a memory system can be excellent while sitting behind an agent that never uses what it returns.

That separation is why general agent evaluation does not transfer. Task-completion benchmarks measure reasoning, planning and tool use, and a memory system contributes to those indirectly at best. Memory evaluation asks a narrower question: given that something was established earlier, does the system use it correctly now.

There are two ways to ask it. Direct evaluation tests the store on its own: given this query, are the right memories returned, in what order, and how fast. Indirect evaluation tests the agent’s answers with and without memory and attributes the difference. Direct measurement localises faults, and indirect measurement is the one that tells you whether any of it mattered.

Both should be applied at each stage of the memory loop, because a failure at any of the three, write, maintain and read, presents identically to a user. A fact that was never extracted, one that was stored but superseded and never retired, and one that exists but ranks tenth all produce the same complaint.

Before choosing an instrument, it is worth knowing what a complete evaluation has to cover: the competencies a memory evaluation must include.

Which agent memory benchmarks exist, and what does each test?

Four matter in practice, and they sit on a difficulty gradient from recall inside one conversation to memory inside a working agent. Vendor claims cite the first two almost exclusively.

LOCOMO tests recall within very long conversations, using dialogues averaging around 300 turns. It is the most reported number in the field and the narrowest, covering accurate retrieval and little else. The dataset, the tasks and the published scores are on the LOCOMO benchmark.

LongMemEval tests recall across separate sessions, including questions whose answer changed between them, which brings conflict resolution partly into scope. It is the harder benchmark and the closer analogue to how products are used, covered on the LongMemEval benchmark.

MemoryAgentBench was built around the four competencies and evaluates them through incremental multi-turn interaction rather than static question answering. STATE-Bench, released open source by Microsoft in 2026, goes further in the same direction by measuring what memory does for an agent completing real tasks. Both are newer and less widely reported, which is precisely why a score on either carries more information than another LOCOMO figure.

Knowing what a benchmark tests is only half of reading a claim, because the same benchmark yields different numbers in different hands: why vendors report different scores.

What do benchmark scores fail to predict?

Almost everything that breaks in production: stale facts, scope leaks between users, latency at load, cost per conversation, and how the system behaves when a memory is simply wrong. Benchmarks run on curated dialogues with clean ground truth, and production has none of those properties.

The most instructive result in this area came from Letta, who benchmarked a plain filesystem against specialised memory tools on LOCOMO and found the filesystem competitive. The honest reading is not that memory tools are unnecessary. It is that the benchmark was measuring something a filesystem also does well, which is retrieval over a bounded conversation, and not the things a memory system is bought for.

Four gaps recur. Staleness is untested because benchmark conversations end before facts have time to go out of date. Isolation is untested because benchmarks run one user at a time, so a scope leak scores perfectly. Operational cost is invisible because no benchmark reports the extraction calls the system made to reach its score. And failure behaviour is unmeasured, since a benchmark records a wrong answer identically whether the system hedged or asserted.

None of this makes benchmarks useless. It makes them a filter rather than a decision: a system scoring poorly can be excluded, and a system scoring well still has to be evaluated on your own data. The production-side numbers that fill the gap are on memory evaluation metrics.

Which raises the practical question of what to watch once the system is running: the metrics to track in production.

How do you build your own memory evaluation set?

From real traces, not from imagined questions, and tagged by which of the four competencies each case exercises. A set written from scratch tests the failures you thought of, which are by definition not the ones that surprised you.

Start by logging the parts of a turn that are usually thrown away: which memories were retrieved, what scores they had, which ones entered the prompt, and what the agent answered. Without that record a failure cannot be diagnosed, because there is no way to tell a retrieval failure from an injection failure from a reasoning failure.

Then convert failures into fixed cases. Each one is a conversation prefix, a question, an expected answer, and a tag naming the competency it tests. Fifty such cases drawn from real use are worth more than a thousand generated ones, and the tags are what stop the set drifting into fifty variations of accurate retrieval.

Include the awkward cases deliberately: a fact stated then corrected, a question needing two memories from different sessions, a question whose answer is genuinely not stored, and a memory belonging to a different user that must not appear. The last two matter most, since a system that invents rather than admitting absence, or that leaks across a scope, fails in ways no accuracy score reports.

With a fixed set in place, one question remains, and it is the one most evaluations never answer: how to know the memory caused the improvement.

What does a repeatable memory evaluation loop look like?

Collect traces, convert failures into cases, run the set on every change, and ablate. The value is in the repetition: a memory system has enough interacting parts that a fix in one reliably breaks something in another.

Run the set against every change to extraction, scoring, retrieval or injection, because those four are coupled more tightly than they look. Tightening a salience threshold improves precision and quietly removes the fact a later case depends on, and only a fixed set catches it.

Keep a human in the loop on a sample. Automated grading misses the failures that matter most in memory systems, particularly an answer that is correct but reveals something the agent should not have surfaced, and an answer that is confidently stale. Both read as passes to a judge model comparing against a reference.

Two habits make the loop pay for itself. Every production incident becomes a case, so the set grows in exactly the places the system is weak. And every vendor claim gets re-run locally before it informs a decision, since the configuration behind a published number is rarely yours. The tool comparison that follows from doing this is on the best AI memory tools, and the build side on the memory build guides.