Guides · Cost optimization
How to Cut LLM Token Costs With Memory
Memory and summarization cut LLM token bills by retrieving only relevant facts instead of resending full history — often reducing per-turn prompt size by 80%+ in long-running agents.
Full context
~26,000 tokens / query
Selective memory
~1,800 tokens / query
Problem
Why full history is expensive
Long context pricing scales with input tokens — resending entire conversation history every turn multiplies cost linearly with session length.
Mem0 reports ~1,800 tokens per LOCOMO query with selective retrieval vs ~26,000 full-context — over 90% reduction (Chhikara et al., 2025). At $3/M input tokens, that’s $0.078 vs $0.054 per query — compounding across millions of agent turns.
Tactics
Token reduction tactics
- Selective memory retrieval — top-k relevant facts only (Engram, Mem0, Zep)
- Rolling summaries — compress old history, keep recent messages verbatim
- Letta paging — core/archival memory tiers; page in only what’s needed (MemGPT DMR 93.4%, Packer et al., 2023)
- Evict stale memories — smaller store = fewer irrelevant retrievals
- Compact injection templates — bullet facts not full transcripts
- Cache embeddings — avoid re-embedding unchanged memories
Measurement
Measuring savings
Track tokens before/after memory optimization; gate deploys on recall@k regression.
Metrics: input tokens per query, retrieval latency, LOCOMO recall@k, cost per conversation. Mem0 p95 total latency 1.44 s vs 17.1 s full-context (Chhikara et al., 2025) — memory saves both tokens and latency.
FAQ
Frequently asked questions
Does memory always reduce LLM token costs?
Usually yes for long-running agents — selective retrieval replaces full history. Short single-turn queries may see no benefit. Retrieval API calls add minor overhead vs token savings.
How often should agents summarize to save tokens?
At session end, every N turns or when context exceeds a threshold. Validate recall@k after each compression level change.
What does Mem0 cost for token savings?
Engram and Mem0 both reduce tokens by retrieving ~1,800 vs ~26,000 tokens per query. Pricing depends on the managed API tier chosen.
Does retrieval overhead negate token savings?
Rarely — Mem0 median search 0.148 s, p95 total 1.44 s vs 17.1 s full-context (Chhikara et al., 2025). Retrieval latency is small vs LLM processing time.
How much do Letta paging strategies save?
Letta keeps core memory in-context and pages archival memory on demand — MemGPT DMR 93.4% (Packer et al., 2023). See Letta alternatives.
How do you monitor token savings in production?
Log input tokens per query, track before/after memory adoption, alert on recall@k drops. See AI memory metrics.