An AI memory architecture is the set of components that let an LLM application keep information beyond one prompt: an extraction step that decides what to save, one or more stores, indexes to search them, a retrieval and ranking step, a context assembler that builds the prompt, and background jobs that consolidate and forget. Most designs differ only in how they build each piece.
What is an AI memory architecture?
An AI memory architecture is the design of how an LLM system writes, stores, retrieves and maintains information outside a single model call. The model itself is stateless. Everything it “remembers” comes from its training weights or from text the application puts into the context window, so memory architecture is mostly about deciding what text goes into that window, and when.
That makes most LLM memory external memory: a store outside the weights and outside the context window that the application writes to and searches. The survey by Zhang et al. (2024) groups the design space by memory source, memory form and memory operations. This page follows the operations, because that’s how you build one.
The CoALA paper (Sumers et al., TMLR) gives a useful vocabulary. It splits an agent’s internal actions into retrieval (read from long-term memory), reasoning (update the short-term working memory with the LLM) and learning (write to long-term memory). Every architecture below implements those three actions in some form. For the memory types themselves (working, episodic, semantic, procedural), see the pillar guide AI agent memory explained.
AI agent memory architecture: what an agent adds
“AI memory architecture” and “AI agent memory architecture” describe the same six components below, but an agent stresses three of them harder than a chatbot does:
- More write events. A chat app writes once per user message. An agent writes after every step: tool calls, their results, intermediate decisions. The extraction step decides what survives, so a noisy write path fills the store with transcript dust.
- Procedural and planning memory. Beyond facts about the user, an agent keeps how-to knowledge (rules, skills, workflows) and plans it can reuse. See procedural memory in AI agents and AI agent planning memory.
- Lessons from failure. Agents repeat mistakes unless the store keeps negative results. ReasoningBank is the clearest published design for that.
Everything else is the same machinery: stores, indexes, retrieval, context assembly, consolidation. If you are designing for an agent rather than a chat UI, budget for the extra writes and plan a consolidation job from day one.
The six components of a memory system
A full memory system has six parts. Simple ones skip some; none skip retrieval or context assembly.
| Component | Job | Typical implementation |
|---|---|---|
| Extraction (write path) | Decide what to keep and in what shape | LLM call that pulls facts, entities, dates from messages |
| Stores | Hold the memories | Vector DB, graph DB, relational tables, files |
| Indexes | Make memories findable | Embeddings, BM25, entity index, timestamps |
| Retrieval and ranking | Pick the few memories that matter now | Hybrid search, fusion, reranking, scoring |
| Context assembly | Fit memories into the prompt | Token budget, ordering, formatting |
| Consolidation and governance | Keep the store accurate and safe | Merge, invalidate, expire, delete, scope per user |
1. Extraction: the write path
Memory extraction turns raw interactions into storable units. The cheapest option stores every message verbatim. Most frameworks instead run an LLM over each exchange to pull out facts, entities and events. The Mem0 paper (Chhikara et al., 2025) describes this as dynamically extracting, consolidating and retrieving salient information, and reports more than 90% token savings versus sending full context on the LoCoMo benchmark (self-reported).
Extraction costs one LLM call per write and can drop or invent details. Keeping a pointer from each extracted memory back to its source message makes errors auditable.
2. Stores
The store determines which questions are cheap to answer. Vectors are good at fuzzy “something like this” recall. Graphs answer “how is X connected to Y.” Tables answer exact lookups. Files are human-readable and versionable. Many systems use two or three at once. The pillar guide compares these storage models side by side.
3. Indexes
One store can carry several indexes. A memory record often has a dense embedding for semantic search, a keyword index (BM25) for exact names and codes, an entity index for graph hops, and a timestamp for time filters. Pure vector search misses exact matches and has no sense of time, which is why most current systems index more than one signal.
4. Retrieval and ranking
Retrieval picks candidates; ranking orders them. Two published designs show the range:
- Scored retrieval. The Generative Agents paper (Park et al., 2023) ranks each memory by the sum of three normalized scores: recency (exponential decay, factor 0.995 per game hour since last access), importance (a 1-10 rating the LLM assigns when the memory is written) and relevance (cosine similarity to the query).
- Hybrid search with fusion. Graphiti combines semantic embeddings, BM25 and graph traversal. Hindsight runs semantic, keyword, graph and temporal retrieval in parallel, merges them with reciprocal rank fusion, then reranks with a cross-encoder.
5. Context assembly
The context assembler decides how many tokens memory gets and where they go in the prompt. A common layout is: system instructions, then a short block of retrieved memories, then recent conversation turns, then the new message. Too much memory is as harmful as too little, since irrelevant facts distract the model. The broader prompt-budget problem is covered in LLM context window optimization.
6. Consolidation and governance
A store that only grows gets noisy and contradictory. Consolidation jobs merge duplicates, summarize old episodes, and resolve conflicts. Governance covers per-user scoping, deletion requests, and screening writes for injected instructions. Hindsight is one implementation of the full loop: a background job merges related facts into observations that keep their supporting quotes and get refined, not overwritten, and mental models keep standing answers current so reading one needs no retrieval or LLM call. Details are in memory consolidation in AI agents.
Six families of LLM memory architecture
Most “LLM memory approaches” fall into one of these families. Real systems mix them.
| Family | Where memory lives | How the model reads it | Example | Main limit |
|---|---|---|---|---|
| In-weights (parametric) | Model parameters | Implicitly, during generation | Pretraining, fine-tuning | Costly to update, can’t cite or delete |
| In-context | The prompt | Attention over tokens | Long-context models, chat history | Cost and accuracy degrade as context grows |
| External retrieval | Vector / keyword index | Search, then insert into prompt | RAG, Mem0 | Recall quality depends on the query |
| OS-style tiered | Main context plus external tiers | The LLM pages data in and out with tools | MemGPT / Letta | More LLM calls, model must manage itself |
| Graph and temporal | Entities and relations with validity times | Graph search plus text search | Graphiti / Zep, Hindsight | Extraction cost and errors |
| Self-organizing | Linked notes the LLM structures | Search plus link traversal | A-MEM | Young, few production deployments |
In-weights memory
Knowledge learned in training is the model’s parametric memory. It’s broad but frozen at the training cutoff, and you can’t inspect or delete one user’s data from it. Fine-tuning can add knowledge, but it’s slow to update. See LLM parametric memory.
In-context memory
The simplest approach resends history every call. It works until the conversation outgrows the window or the cost per call gets too high. Larger windows push the limit back but don’t remove it: the LongMemEval paper (Wu et al., ICLR 2025) found a 30% accuracy drop for commercial assistants and long-context LLMs on information spread across sustained interactions.
External retrieval
The application writes memories to an index and searches it before each model call. This is the base of almost every production memory layer. It differs from document RAG mainly in that the data is the agent’s own interactions and changes on every turn.
OS-style tiered memory
MemGPT (Packer et al., 2023) treats the context window like RAM and external storage like disk. Main context holds read-only system instructions, a read/write working context, and a FIFO queue of recent messages headed by a recursive summary. External context holds recall storage (the full message history) and archival storage (arbitrary text). When the prompt reaches a warning threshold (70% in the paper’s example), the system tells the model so it can save what matters; at the flush threshold it evicts messages and updates the summary. The MemoryOS paper extends the same idea.
Graph and temporal memory
Graph memory stores entities as nodes and facts as edges. Temporal graphs add validity windows, so a changed fact gets invalidated, not deleted. Graphiti’s README calls this “explicit bi-temporal tracking with automatic fact invalidation.” This family answers “what was true in March?” well, which flat vector stores can’t.
Hindsight takes a hybrid route: an event-centric graph (memories linked through shared entities and causal links) searched in parallel with vectors, BM25 and time ranges. That combination holds up on very long histories. On the BEAM benchmark it scores 73.9 at 1M tokens and 64.1 at 10M, against 64.1 and 48.6 for Mem0’s Platform, per Mem0’s own comparison (vendor-run). The price is a bigger read: Mem0 measures 24K-44K tokens per Hindsight retrieval against under 7K for its own, so the context assembler needs a tighter max_tokens.
Self-organizing memory
Self-organizing memory lets the LLM decide the structure. In A-MEM (Xu et al., NeurIPS 2025), inspired by the Zettelkasten note method, each new memory becomes a note with a contextual description, keywords and tags. The system links it to related notes, and adding it can update the attributes of existing notes. The authors report gains over baselines across six foundation models.
Memory built into the model
A separate research line augments the model architecture itself rather than the application around it. These aren’t drop-in tools for most teams, but they explain where “LLM memory augmentation” research is heading.
- RETRO (Borgeaud et al., 2021) retrieves chunks from a 2-trillion-token database during generation and reports performance comparable to GPT-3 on the Pile with 25x fewer parameters.
- Memorizing Transformers (Wu et al., ICLR 2022) add approximate kNN lookup into a non-differentiable memory of past (key, value) pairs; performance kept improving as memory grew to 262K tokens.
- Titans (Behrouz et al., 2024) add a neural long-term memory module that learns to memorize at test time, with attention as short-term memory. The abstract claims scaling past a 2M-token context window on needle-in-haystack tasks.
What benchmarks say about memory architecture choices
More structure is not always better. On short chat histories, plain retrieval over the raw messages beat most memory systems. Structure (time, entities, consolidation) pays off on questions about change over time and on histories too long for any context window.
Each row compares two architecture choices inside one study, so the numbers in a row share a setup. Don’t compare numbers across rows.
| Choice tested | Result | Source |
|---|---|---|
| In-context (full history) vs external retrieval over raw messages | LoCoMo, GPT-4o-mini: full context 72.30, Raw-RAG 89.07 | Bao et al., Oct 2026 |
| Extraction into a memory store vs keeping raw messages | Raw-RAG beat 12 memory systems (Mem0, A-MEM, MemOS, EverMemOS and others) in 6 of 7 settings, by 0.13 to 5.53 points | Bao et al. |
| Full context vs temporal + entity memory, same 20B open model | LongMemEval temporal reasoning: 31.6 vs 79.7; multi-session: 21.1 vs 79.7 | Hindsight paper, Table 3 (vendor-run) |
| Flat extracted memory vs multi-strategy recall at 10M tokens | BEAM 10M: Mem0 48.6, Hindsight 64.1 | Mem0’s comparison page (vendor-run) |
What this means for design:
- Keep the raw messages. Bao et al. found extraction throws away evidence some questions need. Store the source next to anything you extract.
- Start with retrieval over raw history as the baseline. The paper’s tests use histories that fit in a large context window; the authors list longer histories as future work.
- Add time and entity structure for questions about change. The biggest gaps in the Hindsight paper are on temporal and multi-session questions, where a fact changed or is spread across sessions. Those are vendor-run numbers, and the paper’s own caveat applies: some baseline columns use a different judge model.
- Long histories need more than one search. At 10 million tokens nothing fits in context. Hindsight (MIT, self-hostable) runs semantic, keyword, graph and time search together and keeps quotes from the source messages in its consolidated observations, which is the “keep the raw evidence” advice built in. It wasn’t in the Bao et al. test.
For the full Bao et al. table and its limits, see RAG vs agent memory.
How to choose an architecture
Start from the questions your agent must answer, not from a database.
- List the recall questions. “What does this user prefer?” needs a profile. “What happened last Tuesday?” needs timestamps. “Who reports to whom?” needs a graph.
- Start with in-context plus summaries if sessions are short and one-off.
- Add external retrieval once history crosses sessions or exceeds a few thousand tokens.
- Add extraction when raw transcripts make retrieval noisy.
- Add temporal or graph structure when facts change or relations matter.
- Add consolidation and deletion before you have real users, not after.
- Measure on 20-50 real questions before and after each step.
Each step adds cost and failure modes, so stop when your recall questions pass. Anthropic’s agent guidance makes the same point about agent systems in general: find the simplest solution possible and add complexity only when needed.