Fundamentals
How Does AI Memory Work?
AI memory works in a loop: agents write what matters to an external store, retrieve relevant memories on each turn, update when facts change and forget what is stale — turning stateless LLMs into persistent agents.
The memory loop
Core model
The AI memory loop in four steps
AI memory runs write → store → retrieve → forget/update on every production agent. Each step is a distinct mechanism with its own design decisions.
Step 1
Writing and storing memories
Agents store memory after a turn when extraction decides a fact, preference or event is worth keeping. Triggers include end-of-turn processing, tool calls, or explicit memory.add() in frameworks like Mem0.
An LLM or rules pipeline extracts durable facts — turning “I moved to Berlin and I’m vegetarian” into two clean memories — then deduplicates against existing entries before writing to a vector database, knowledge graph or Redis buffer. Extraction quality is measurable: the Mem0 paper (Chhikara et al., 2025) reports a LOCOMO LLM-as-a-Judge score of 66.9 for its extract-and-store pipeline (68.4 with graph memory), against 72.9 for the full-context baseline at roughly 15× the token cost.
Step 2
Retrieving the right memories
Retrieval runs before each response: embed the query, search the store, rank candidates and inject the top matches into context.
Semantic vector search is the default; production systems add recency boosts, hybrid keyword search and graph traversal. The canonical retrieval-scoring formula comes from the Generative Agents paper (Park et al., 2023): each memory is scored on recency × importance × relevance — implemented as a weighted sum of the three normalized scores (each weighted 1.0 in the paper), with recency decaying exponentially at 0.995 per hour since the memory was last accessed. Worked example: a fresh, core preference relevant to the query (recency 0.9 + importance 0.9 + relevance 0.8 = 2.6) outranks a week-old small-talk memory (0.3 + 0.2 + 0.6 = 1.1) — so “user is vegetarian” wins over “user mentioned rain last Tuesday” when the query is about dinner.
Selective retrieval is why memory beats resending history: on the LOCOMO benchmark, Mem0 retrieves with a median search latency of 0.148 s and answers with a p95 total latency of 1.44 s, versus 17.1 s for the 26,000-token full-context baseline — a 91% reduction — while consuming about 1,800 tokens per query instead of 26,000, over 90% fewer (Chhikara et al., 2025, Mem0 paper).
Step 3
Updating and consolidating memories
When new information arrives, agents update existing memories or consolidate short-term entries into long-term storage.
Consolidation merges duplicate facts, summarizes long threads and promotes session memories to durable stores — often between sessions or during background jobs. Temporal updating pays off on benchmarks: Zep’s temporal knowledge graph, which invalidates superseded facts instead of deleting them, improved LongMemEval accuracy by up to 18.5% over a full-context baseline while cutting response latency by about 90% (Rasmussen et al., 2025, Zep paper).
Step 4
Forgetting and eviction
Agents must forget — stale, irrelevant or contradictory memories degrade retrieval quality and raise token cost.
Eviction uses TTLs, decay policies and relevance pruning. Without forgetting, memory stores grow unbounded and wrong facts persist alongside correct ones.
Tiers
Memory hierarchy: where data lives
Production memory uses tiers: context window (hot), fast buffer (warm) and deep vector/graph store (cold).
The MemGPT research line (Packer et al., 2023) pages memories between tiers like an operating system — giving agents effectively unbounded history without stuffing every token into context. On the Deep Memory Retrieval benchmark the MemGPT team introduced, the paged architecture scores 93.4% accuracy (Packer et al., 2023, as reported in the Zep evaluation, 2025).
Orchestration
Who orchestrates the loop?
A memory-management layer decides when each step runs — a framework (Engram, Mem0, Zep, Letta, LangMem) or custom code over Redis/pgvector.
RAG
AI memory vs RAG in the loop
RAG retrieves static documents; AI memory reads and writes personal state. Agents combine both: RAG for org knowledge, memory for user/session facts.
Frameworks
Frameworks that implement the loop
| Class | Framework | Loop implementation |
|---|---|---|
| Vector-native | Engram | Managed extract → transform → commit pipeline on Weaviate |
| Vector API | Engram, Supermemory | Managed write/retrieve API |
| Temporal KG | Zep | Graph write + temporal invalidation |
| Virtual paging | Letta | Tiered page in/out |
| LangGraph | LangMem | Checkpointer + store |
| DIY | Redis, pgvector | You implement each step |
FAQ
Frequently asked questions
How do AI agents store memory?
After each turn, an extraction pipeline decides what is worth remembering, embeds it, and writes to an external store (vector DB, graph or KV). See writing and storing memories.
What role does a vector database play in AI memory?
Vector databases index embeddings so agents can find memories by semantic similarity — the typical backend for long-term semantic and episodic memory. See vector databases for memory.
How is memory retrieved in AI agents?
Before each response, the agent embeds the query, searches the store, ranks candidates by relevance and recency, and injects top matches into the prompt. See memory retrieval.
Do AI agents forget memories?
Yes — production agents evict stale, irrelevant or contradictory memories via TTLs, decay and relevance pruning. Without forgetting, retrieval quality degrades. See forgetting and eviction.
What is the difference between AI memory and the context window?
The context window is working memory for the current session only. AI memory persists in external stores across sessions. See memory vs context window.
Is AI memory the same as RAG?
No. RAG retrieves from a fixed document corpus. AI memory is dynamic and personal — updated across sessions. See memory vs RAG.
What is memory consolidation?
Consolidation merges short-term memories, summarizes threads and promotes them to durable long-term storage — often between sessions. See memory consolidation.
What is MemGPT memory paging?
MemGPT treats the context window like RAM and pages memories between a fast tier and deep store — effectively unbounded conversation history. Letta implements this pattern. See virtual context and MemGPT.
How do I build AI memory from scratch?
Pick a store, implement write after each turn, retrieve before each response, and add eviction policies. Or use a framework like Engram, Mem0 or LangMem. See add memory to an agent.
How do I measure AI memory quality?
Use LOCOMO (long-conversation recall) and LongMemEval (cross-session recall), plus track latency and token cost. See evaluation hub.