Advanced · Research
Memory and Reinforcement Learning in Agents
Reinforcement learning can train agents when to store, retrieve and forget memories — optimizing long-horizon tasks where memory tool use is part of the policy.
RL policy
Learn when to store / retrieve / forget
Supervised
Fixed extraction pipelines (production default)
RL policy
RL for memory tool use
Instead of fixed extraction rules, an RL agent learns a policy over memory operations — store, search, update, forget — rewarded for correct recall on long-horizon tasks.
AgeMem (arXiv 2601.01885) frames memory management as a reinforcement learning problem: the agent receives reward when retrieved memories lead to correct downstream answers. This extends the memory-as-a-tool pattern — where the LLM calls search_memory and write_memory — by learning optimal timing rather than relying on heuristics.
Production stacks today use supervised extraction (Engram async pipeline, Mem0 auto-extract) because RL-trained memory policies lack broad benchmark validation. LOCOMO J 66.9 for Mem0 (Chhikara et al., 2025) reflects supervised/managed extraction, not RL-learned policies.
Long-horizon
Memory in long-horizon RL
External memory augments agent state for tasks spanning many steps — where the context window cannot hold full history.
In long-horizon RL, external memory stores serve as state augmentation: the agent writes observations to memory, retrieves relevant past states and uses them in planning. This parallels production patterns — Letta virtual-context paging (DMR 93.4%, Packer et al., 2023) manages what enters working memory across long interactions.
The research frontier asks whether RL can optimize what gets written and retrieved better than fixed pipelines. For current production agents, external memory layers (Engram, Mem0, Zep) with supervised extraction remain the proven path.
FAQ
Frequently asked questions
When should you train memory with RL?
Research setting — long-horizon tasks where fixed extraction heuristics fail. Production agents today use supervised pipelines (Engram, Mem0 auto-extract) with LOCOMO validation.
What is the AgeMem paper?
AgeMem (arXiv 2601.01885) frames memory store/retrieve/forget as an RL problem — reward for correct recall on downstream tasks.
RL memory vs supervised extraction?
Supervised: fixed pipeline extracts facts after each turn (Engram, Mem0). RL: agent learns policy over memory tool calls. Supervised has published LOCOMO benchmarks; RL is research.
Is RL-trained memory in production?
Not mainstream in 2026. Ship with Engram, Mem0 or Zep supervised extraction and LOCOMO gates.
RL memory vs Mem0 auto-extract?
Mem0 auto-extract is supervised/heuristic. RL would learn when to store/retrieve. See memory as a tool for the tool-call pattern both build on.