RAG vs Agent Memory: Differences and When to Use Each

RAG vs agent memory comes down to who writes the data. Retrieval-augmented generation (RAG) searches a corpus of documents someone curated, like product docs or policies, and adds the best matches to the prompt. Agent memory searches information the agent wrote itself from its interactions, like a user’s preferences or what happened last week. RAG is mostly read-only and shared; memory is read-write, per user, and keeps changing.

They use the same building blocks (embeddings, vector search, reranking), which is why they’re often confused. This page explains each one, the differences that matter, and how to run both in one agent.

What is retrieval-augmented generation (RAG)?

Retrieval-augmented generation is a technique that answers a question by first retrieving relevant passages from an external collection of documents, then giving them to an LLM together with the question. The model grounds its answer in the retrieved text instead of relying only on what it learned in training.

The term comes from Lewis et al. (NeurIPS 2020), who described “models which combine pre-trained parametric and non-parametric memory for language generation.” The parametric memory was the model’s weights. The non-parametric memory was a dense vector index of Wikipedia, searched by a neural retriever.

A typical RAG pipeline today:

1. Split documents into chunks.
2. Embed each chunk and store it in a vector database, often with a keyword index too.
3. At query time, embed the question and retrieve the top matches.
4. Optionally rerank them.
5. Put the best chunks in the prompt and generate an answer.

RAG is good at one thing: answering from knowledge that exists before the conversation starts, and that’s the same for everyone who asks.

What is agent memory?

Agent memory is a system that saves information from an agent’s own interactions, such as facts about a user, past events and lessons from earlier tasks, and retrieves the relevant parts into the prompt in later turns and sessions. It has a write path that decides what to keep and an update path that handles change.

The model is stateless, so without memory each session starts from zero. A memory system typically:

• Extracts facts from conversations with an LLM (“user is vegetarian”, “prefers short answers”).
• Scopes them to a user, agent or project.
• Updates or invalidates facts that change.
• Consolidates duplicates and old details over time.
• Retrieves a few relevant memories per turn.

The full picture of memory types and tools is in AI agent memory explained.

RAG vs agent memory: the key differences

The Zep paper (Rasmussen et al., 2025) puts the gap this way: existing RAG frameworks “are limited to static document retrieval,” while agents need “dynamic knowledge integration from diverse sources including ongoing conversations and business data.”

Why RAG over chat history isn’t enough

The obvious shortcut is “RAG memory”: embed every past message and search it like documents. It works for simple recall, and it’s how many memory setups start. It breaks in four places.

Changed facts. “I live in NYC” from January and “I just moved to SF” from June both match “where do I live?” Plain RAG has no notion that one replaced the other. The LongMemEval benchmark (Wu et al., ICLR 2025) tests this as knowledge updates, along with multi-session reasoning, temporal reasoning, information extraction and abstention. Its authors found that commercial assistants and long-context LLMs show a 30% accuracy drop on memory across sustained interactions.

Time. “What did we decide in the last meeting?” needs to know which meeting was last. Chunk similarity doesn’t encode order.

Noise. Raw messages are full of small talk. A question about diet can retrieve ten “thanks!” messages that happen to sit near a food mention. Memory systems extract facts first, so the search runs over cleaner units. Mem0’s paper reports its extraction-based approach saved more than 90% of token cost versus full context on LoCoMo (a vendor-run evaluation).

Granularity. LongMemEval’s authors found that how you split history into stored units matters, and proposed session decomposition, fact-augmented keys and time-aware query expansion to fix it. Those are memory-specific steps a document RAG pipeline doesn’t have. Coding agents show the same thing: in the REM paper (October 2026), injecting a past bug-fixing run as a raw trajectory or a plain summary did not help, while a compact, phase-structured version did; see AI coding agent memory.

Dedicated memory systems handle these four problems at write time, not query time. Graphiti, for example, marks a replaced fact as superseded. Hindsight tags each fact with its time and entities as it’s stored, runs a time-range search next to semantic, keyword and graph search, and folds related facts into observations that get refined as new facts arrive instead of piling up. That design is why it reports 94.6 on LongMemEval, the benchmark built around these failure modes (vendor-reported).

Does plain RAG lose to memory systems on benchmarks?

Not always. An October 2026 paper found that plain RAG over raw chat history beat 12 memory systems on LoCoMo and LongMemEval. Its edge was small, and the histories in those tests are short. The case for memory systems rests on histories far longer than these tests, which the paper didn’t cover.

The paper is Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought (Bao et al., arXiv, October 22, 2026). Its baseline, Raw-RAG, does no extraction: it embeds the raw conversation, retrieves with dense search, reranks, and hands the evidence to the reader LLM. Overall accuracy (%) from its main table:

The paper also tested MemOS, MemU, Nemori, MemoryOS, A-MEM and RecMem; all scored below Raw-RAG. Across seven benchmark and model settings (adding MEME and MemoryAgentBench), Raw-RAG came first in six. Its lead over the best memory system was 0.13 to 5.53 points.

Why it won, per the authors: raw history keeps more of the evidence a question needs, and extraction throws some away. Even given full access to their own stores, the memory systems trailed raw evidence. So keep the raw messages next to whatever you extract.

How far this goes:

• Short histories. LoCoMo conversations and LongMemEval’s S set fit in a large context window. The authors list longer histories as future work, and say their evidence is “limited to offline conversation on public benchmarks.”
• Storage and privacy. They name these as costs of keeping raw history, and list privacy and deletion limits as things future work must account for.
• Scores don’t carry across papers. Mem0 scores 66.40 on LongMemEval here, but 94.4 on its own comparison page, which uses its current platform and setup. Compare systems inside one paper, not across papers.
• Hindsight wasn’t tested. It does extract facts, so the paper’s warning applies to it too, but its consolidated observations keep quotes from the original messages, and recall runs semantic, keyword, graph and time search together, not one dense search. Its lead shows on long histories: on BEAM 10M (10 million tokens, far past any context window) it reports 64.1 against Mem0’s 48.6, on Mem0’s own comparison page. Both are vendor-run numbers.

The takeaway for builders: start with RAG over raw history as your baseline, and measure. Add a memory system when the histories outgrow it, or when you need what plain RAG lacks: facts that update, per-user scope, and time-aware recall.

RAG vs long-term memory vs long context

There’s a third option people weigh: skip retrieval and put everything in a large context window.

Long context degrades as it grows. Chroma’s Context Rot report (July 2025) tested 18 models and found that “model performance varies significantly as input length changes, even on simple tasks.” On LongMemEval, all models did better with a focused prompt than with the full history. That’s the case for retrieval, whether from documents or memories.

What is a memory bank?

“Memory bank” shows up in two senses:

• Google’s Memory Bank (Vertex AI Agent Engine Memory Bank, renamed Agent Platform Memory Bank in 2026) is a managed memory service. It uses Gemini to extract facts from conversations, consolidates them with existing memories in the same scope, and returns them by scope or similarity. It’s agent memory, not RAG. Details are in the Memory Bank guide.
• MemoryBank (Zhong et al., 2023) is a research system that stores conversations, daily and global summaries, and a user profile, and forgets old memories along an Ebbinghaus-style curve.

Both differ from RAG the same way: they write and update per-user memories, while RAG reads a shared corpus.

How to use RAG and agent memory together

Most production assistants need both. RAG answers “what’s true in general?” (the refund policy). Memory answers “what’s true about this user?” (their flight was cancelled last week). The usual pattern is two stores, searched in parallel, with results placed in separate sections of the prompt.

 1import time
 2
 3import chromadb
 4
 5client = chromadb.PersistentClient(path="./store")
 6docs = client.get_or_create_collection("docs")        # RAG: shared, curated
 7memory = client.get_or_create_collection("memories")  # memory: per user, written by the agent
 8
 9docs.upsert(
10    ids=["refund-policy"],
11    documents=["Refunds for cancelled flights are issued within 7 business days."],
12)
13
14
15def remember(user_id: str, fact: str) -> None:
16    memory.add(
17        ids=[f"{user_id}-{time.time_ns()}"],
18        documents=[fact],
19        metadatas=[{"user_id": user_id, "ts": int(time.time())}],
20    )
21
22
23def build_context(user_id: str, message: str) -> str:
24    knowledge = docs.query(query_texts=[message], n_results=3)["documents"][0]
25    recalled = memory.query(query_texts=[message], n_results=5, where={"user_id": user_id})
26    facts = recalled["documents"][0]
27    return (
28        "Reference documents:\n" + "\n".join(f"- {d}" for d in knowledge)
29        + "\n\nWhat you know about this user:\n" + "\n".join(f"- {f}" for f in facts)
30    )
31
32
33remember("u7", "User's flight LH716 to Tokyo was cancelled on 3 October")
34print(build_context("u7", "When will I get my money back?"))

This is the bare structure, and it runs as-is with chromadb. Three rules carry over to real systems:

1. Keep the stores separate. Documents are shared; memories are private. Mixing them makes access control and deletion hard.
2. Always filter memory by user. The where={"user_id": ...} line is what stops one user’s facts showing up for another.
3. Label the sections in the prompt. The model should know which text is policy and which is about the user.

In production, remember() is where a memory system earns its keep: extracting facts with an LLM, merging duplicates, and handling changes. Frameworks such as Mem0, Zep, Letta, LangMem and Hindsight do that part. Most of them are memory only, so you keep your RAG store next to them: Hindsight has no Drive or Notion-style document connectors, and its writes cost an LLM call each. Supermemory is the main one that bundles document RAG and connectors with memory in one API. The merging step itself is covered in memory consolidation in AI agents. For the storage layer under both stores, see vector databases for LLM memory.

When to use RAG, memory, or both

RAG alone is enough when:

• Answers come from a knowledge base: docs, policies, manuals, code.
• Every user should get the same answer to the same question.
• Users don’t come back, or don’t expect to be recognized.

Add agent memory when:

• Users return and expect the agent to remember them.
• Preferences or context should shape answers (diet, plan tier, past tickets).
• The agent does multi-step work over days and must recall decisions.
• The agent should learn from its own mistakes. That needs more than storage: Hindsight’s reflect, for instance, reasons over stored memories to answer, and its mental models keep standing answers up to date.

Use both when the agent answers from shared knowledge and serves known users. That covers most support bots, copilots and personal assistants.

A quick test: if you’d store the information in a CMS or a wiki, it belongs in RAG. If you’d store it in a CRM or user profile, it belongs in memory.