AI Agent Long-Term Memory: How It Works, How to Build It

AI agent long-term memory is information an agent keeps across sessions: facts about users, past events, and lessons from earlier tasks. The model can’t hold it, because LLMs forget each conversation when the call ends. So the agent’s code writes memories to an external store and, on each new turn, retrieves the few that matter into the prompt.

Building it well comes down to a handful of design choices: what to store, how to index it, how to search it, and how the model reads what comes back. This page walks through each, using findings from the LongMemEval benchmark, then shows a working example. The pillar guide, AI agent memory explained, covers memory types and tools in more breadth.

What is long-term memory in AI agents?

Long-term memory in an AI agent is persistent storage outside the model that keeps information across conversations and sessions, scoped to a user, agent or organization. The agent writes selected facts, events and lessons into it and retrieves the relevant ones into its context window when they’re needed.

LangGraph’s memory docs define it by scope: long-term memory “is shared across conversational threads” and saved under custom namespaces such as a user ID. That’s the line between it and short-term memory, which ends with the thread. The comparison is in short-term vs long-term memory in agentic AI.

The CoALA framework (Sumers et al., 2023) splits long-term memory into three types: episodic (past experiences), semantic (facts) and procedural (how to act). Most chat products store semantic facts, and the hard part is keeping them right when they change. Task agents gain the most from episodic and procedural memory, such as the reusable routines of Agent Workflow Memory.

Does AI have long-term memory?

The model doesn’t. A language model’s weights hold general knowledge from training, sometimes called parametric memory, but nothing about you or yesterday’s chat. Every API call starts from a blank slate.

What users experience as an AI that remembers is a system feature. ChatGPT, Claude and Gemini all ship memory features that store user information outside the model and bring it back in later chats. Agent frameworks do the same with a memory store: Microsoft Agent Framework plugs in memory providers such as Mem0, Foundry Agent Service memory or Cosmos DB, and OpenClaw keeps Markdown memory files with swappable memory plugins. How one product does it is covered in Claude’s long-term memory.

So “when will AI have long-term memory?” has two answers. As a product feature, it already does. Inside the model itself, as weights that keep learning from each conversation, it’s still a research problem: fine-tuning per user is costly, and editing weights risks damaging what the model already knows.

How long-term memory works: index, retrieve, read

LongMemEval (Wu et al., ICLR 2025) frames every long-term memory system as three stages over a key-value store:

1. Indexing: turn each past session into one or more stored items (values), each with keys used for search.
2. Retrieval: turn the current message into a query and fetch the most relevant items.
3. Reading: the LLM reads the retrieved items and writes the answer.

The paper names four control points inside those stages, and tested each one. Its findings are the most useful public evidence on what actually matters.

The last row is easy to miss. Even when retrieval finds the right memories, the model can still misuse them. How memories are formatted in the prompt is part of the memory system.

Relevant memories are not the same as sufficient ones

Every stage above ranks memories by relevance: how close each item is to the query on its own. Two October 2026 papers measured the gap that opens up when the answer needs several memories together.

QEMem (arXiv 2610.22668, Li et al., 15 Oct 2026) argues that a top-k list of relevant records can still be short of evidence, because the pieces of an answer get split across records, buried under a summary, or separated from the link that connects them. Instead of scoring each memory alone, it assembles a query-specific evidence set: it searches several levels of granularity at once, organizes what it finds into structured cards that keep each piece’s evidential role and its source record, then checks whether the set is sufficient and goes back for what is missing. It runs on an existing memory store without changing how memories are written. On LoCoMo the authors report 91.43% LLM-judge accuracy with Claude Opus 4.5, 7.6 points above the strongest baseline they tested (MemR³) and 10.7 points above NEMORI on a shared store. Vendor-run, like nearly every number in this field. The “buried under a summary” problem is also why it pays to keep raw history searchable behind any summary; see summaries or raw history.

JointCore (arXiv 2610.23266, Subrahmanyam and Choudhary, 16 Oct 2026) attacks the same gap from the other end: what is the smallest set of memories that still produces the agent’s full-memory answer? Ranking by individual influence misses it, because records can be redundant or useful only in combination. Its JointCoreBench builds cases where the sufficient set is known exactly (6,912 exact conditions, 9 interaction families, 6 workflow domains), and deleting memories in cost-ordered groups rather than one at a time recovered a minimum-cost sufficient set in every exact condition, with subset evaluations down from 30.00 to 11.13 and 87.8% active-memory savings. The authors are explicit that this is a benchmark baseline, not a memory-compression method ready to deploy.

The practical read: if your agent has the right memories in context and still answers badly, the next fix is usually not a better ranker. It is a second pass that reasons over what came back and asks what is still missing, which is what Hindsight exposes as reflect (neither paper tested it). Formatting and note-taking, from the Reading row above, do the same job more cheaply.

How to build long-term memory for an agent

A minimal build follows these steps:

1. Choose the scope. Per user, per agent, per team? This becomes your namespace or ID, and it’s your main privacy boundary. Make a call with no ID fail rather than match; an empty key that matches is how memory leaks between users.
2. Decide what to keep. Durable preferences and facts, important events, lessons from failures. Not every message.
3. Pick write timing. During the turn (hot path, adds latency) or after it (background, adds lag).
4. Index for several kinds of search. Embeddings for fuzzy matches, keywords for names and IDs, timestamps for time questions.
5. Retrieve before each model call and cap memory at a fixed token budget.
6. Handle updates and deletes. Facts change; users ask to be forgotten.
7. Evaluate on your own data. Write 20 to 50 real questions and check what the agent recalls.

Here’s steps 3 to 5 with LangMem’s background memory manager and a LangGraph store, adapted from LangMem’s background quickstart:

 1from langchain.chat_models import init_chat_model
 2from langchain_core.runnables import RunnableConfig
 3from langgraph.func import entrypoint
 4from langgraph.store.memory import InMemoryStore
 5from langmem import create_memory_store_manager
 6
 7store = InMemoryStore(index={"dims": 1536, "embed": "openai:text-embedding-3-small"})
 8llm = init_chat_model("anthropic:claude-sonnet-4-5")
 9
10# Extracts, updates and consolidates memories in the store after each turn
11manager = create_memory_store_manager(
12    "anthropic:claude-sonnet-4-5",
13    namespace=("memories", "{user_id}"),  # filled from config at runtime
14)
15
16@entrypoint(store=store)
17def chat(message: str, *, config: RunnableConfig):
18    user_id = config["configurable"]["user_id"]
19    hits = store.search(("memories", user_id), query=message, limit=5)
20    known = "\n".join(h.value["content"]["content"] for h in hits)
21
22    messages = [
23        {"role": "system", "content": f"What you know about this user:\n{known}"},
24        {"role": "user", "content": message},
25    ]
26    response = llm.invoke(messages)
27    manager.invoke({"messages": messages[1:] + [response]})  # reads user_id from config
28    return response.content
29
30cfg = {"configurable": {"user_id": "alice"}}
31chat.invoke("I'm allergic to peanuts and I live in Lisbon.", config=cfg)
32print(chat.invoke("Any snack ideas for my flight?", config=cfg))

In production, swap InMemoryStore for AsyncPostgresStore and run the manager off the request path, for example with LangMem’s ReflectionExecutor, which can delay processing until a conversation goes quiet. The same loop works with dedicated memory services; only the search and save calls change.

Long-term memory tools

History length is the biggest split. On the BEAM benchmark, Hindsight scores 73.9 at 1M tokens and 64.1 at 10M, against 64.1 and 48.6 for Mem0’s Platform (Mem0’s own comparison page; vendor-run). On shorter tests the two are close: 94.6 vs 94.4 on LongMemEval, 92 vs 92.5 on LoCoMo. Hindsight’s tradeoff is size: by Mem0’s measurement it returns 24K-44K tokens per retrieval against under 7K, so cap max_tokens if prompt cost matters. See Mem0 vs Hindsight.

Picking between them depends on your data and stack; the decision guide is in how to pick the best AI memory framework. If it’s down to Letta or LangMem, Letta vs LangChain memory compares the two. And measure a plain baseline first: in an October 2026 paper, RAG over the raw chat history beat 12 memory systems on LoCoMo and LongMemEval (details and limits).

Is there a limit to long-term memory in AI?

Storage isn’t the limit. A database holds millions of items. The real limits are elsewhere:

• Retrieval quality. LongMemEval found commercial chat assistants and long-context LLMs showed “a 30% accuracy drop” on information across sustained interactions, and that long-context LLMs lose 30% to 60% of their accuracy on its LongMemEval_S setting compared with being handed only the relevant sessions.
• Changing facts. When a user’s situation changes, systems that can’t tell old from new facts return either one.
• Extraction errors. An LLM decides what to save. It can drop details or record things that were never said, and every bad memory gets reused.
• Token budget. Only a few hundred to a few thousand tokens of memory fit usefully in each prompt. More memories mean more ranking, not more context.
• Noise and drift. Stores that only grow get worse over time without consolidation and expiry.

Vendor accuracy numbers on these benchmarks are almost all self-reported and use different models and judges, so they aren’t directly comparable. The published scores, with who ran each test, are collected in the agent memory leaderboard. Methods for testing your own setup are in LLM memory evaluation.

Long-term memory and AI self-evolution

Some researchers see long-term memory as the route to agents that improve after deployment. The 2024 report Long Term Memory: The Foundation of AI Self-Evolution (Jiang et al.) argues that models need long-term memory “to store and manage processed interaction data” in order to evolve during use, not only through bigger training runs. The authors report their LTM-based multi-agent framework, OMNE, took first place on the GAIA benchmark at the time.

CoALA makes a related point from the agent side. Storing task-relevant language in memory is “cheaper and quicker than parameter updates,” so most agent learning today happens in memory, not in weights. That’s likely to stay true until per-user model updates become cheap and safe.