LLM Memory Evaluation: Benchmarks for Agent Memory

14 min read

How to evaluate LLM and agent memory: LoCoMo, LongMemEval, MemoryAgentBench, BEAM, FEB-30, Bemula, SciValidityBench, BehaviorMemBench, MSC and DMR compared by size, tasks and metrics, plus how to run one.

LLM memory evaluation means testing whether a model or agent can recall, update and reason over information from earlier sessions. The standard benchmarks are LoCoMo and LongMemEval, with MemoryAgentBench and BEAM as harder, newer tests. Each feeds a long chat history to the system, then asks questions only that history can answer.

The tests differ in what they stress: plain recall, time, changed facts, forgetting, or scale. Picking the wrong one tells you little about your use case. Two 2026 benchmarks, FEB-30 and Bemula, go further and score the memory store itself rather than the answer, and three late-October papers split recall from the decision made with it. This page covers what each benchmark measures, how it’s scored, and how to run one yourself.

What is LLM memory evaluation?

LLM memory evaluation is the measurement of how well a language model or memory system stores information from past interactions and uses it later. A benchmark supplies a long history (chat sessions or documents), asks questions that depend on it, and scores the answers with exact match, F1 or an LLM judge.

Zhang et al.’s survey of agent memory splits evaluation in two. Direct evaluation scores the memory itself, for example whether the right fact was retrieved. Indirect evaluation scores an end task, like answering a question or holding a conversation, that needs memory to succeed. Most benchmarks below do both: they score the final answer and, where evidence is labeled, the retrieval step.

For background on what’s being tested, see our guide to AI agent memory. For the research taxonomies behind these tests, see the LLM memory survey papers.

LLM memory benchmarks compared

BenchmarkYear, venueDataWhat it testsMain metric
MSC (Multi-Session Chat)2021Up to 5 chat sessions per episodeLong-term open-domain dialoguePerplexity
DMR (Deep Memory Retrieval)2023, MemGPT paperMSC plus a 6th session with a questionRecall of facts from past sessionsLLM-judged accuracy, ROUGE-L
LoCoMo2024, ACL10 conversations released (50 in first version), ~300 turns, ~9K tokens, up to 35 sessionsQA, event summarization, multimodal dialogueF1 (QA)
LongMemEval2024, ICLR 2025500 questions; S ≈ 115K tokens, M ≈ 500 sessions (≈1.5M tokens)Extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstentionGPT-4o-judged accuracy; Recall@k, NDCG@k
MemoryAgentBench2025, ICLR 2026Long-context datasets turned into incremental multi-turn inputAccurate retrieval, test-time learning, long-range understanding, selective forgettingTask-specific (accuracy, LLM-judged)
BEAM2025, ICLR 2026100 conversations, 2,000 questions, up to 10M tokensTen abilities, incl. contradiction resolution, event ordering, preference followingLLM-judged nugget scores; Kendall tau-b for event ordering
FEB-302026, arXiv30 generated fact-evolution trajectories, 6 diagnostic categories, 3 difficulty levelsThe memory store itself: whether it holds the right facts and updates them when they changeFact accuracy, update detection recall, false update rate
Bemula2026, arXiv50 multi-session dialogues, 135 forget requests, 1,742 questionsDeleting a memory on request, and whether unrelated memories survivePass rate across five test suites
SciValidityBench2026, arXiv64 task specs across 6 scientific domainsWhether a remembered result is valid to use in a decisionValid-decision rate
BehaviorMemBench2026, arXivReal e-commerce logs, 1,194 users over 3+ yearsRemembering user behavior vs inferring preferences, scored apartRemember and infer scores against future-behavior labels

The sections below describe each one from its paper. FEB-30 and Bemula are the newest, and they test two things the older benchmarks never scored: the contents of the store, and deletion.

LoCoMo: very long-term conversational memory

LoCoMo (Maharana et al., Snap Research) is the benchmark most vendors cite. Its conversations average about 300 turns and 9K tokens over up to 35 sessions, generated by LLM agents with personas and event timelines, then checked by humans.

The QA task has five categories: single-hop, multi-hop, temporal, open-domain knowledge and adversarial (questions with no answer in the conversation). The paper scores QA with F1. In its results, humans scored 87.9 F1 overall while the best model tested, GPT-3.5-turbo-16K, scored 37.8.

Two details trip people up:

  • The released set is smaller than the paper’s. The LoCoMo repo ships locomo10.json with ten conversations, “a subset of the conversations released previously” (the first arXiv version had 50). Vendor scores labeled LoCoMo or LoCoMo10 use this set.
  • Vendors often change the metric. Many memory papers, such as Mem0’s (arXiv 2504.19413), report LLM-as-judge accuracy instead of the original F1, so their numbers aren’t on the same scale as the LoCoMo paper.

LongMemEval: five long-term memory abilities

LongMemEval (Wu et al., ICLR 2025) targets chat assistants. It has 500 hand-curated questions, each with its own chat history in which the evidence is hidden among unrelated sessions.

It tests five abilities:

  1. Information extraction: recall a detail the user or assistant said.
  2. Multi-session reasoning: combine facts spread across sessions.
  3. Temporal reasoning: answer questions about when things happened.
  4. Knowledge updates: use the newest value when a fact changed.
  5. Abstention: say “I don’t know” when the history has no answer.

Question types in the data are single-session-user, single-session-assistant, single-session-preference, multi-session, knowledge-update and temporal-reasoning. Abstention questions have a question_id ending in _abs.

There are three history sizes. Oracle holds only the evidence sessions. LongMemEval_S is about 115K tokens per question. LongMemEval_M has about 500 sessions, roughly 1.5M tokens. In the paper, GPT-4o dropped from 0.870 accuracy on oracle histories to 0.606 on the full S history, about a 30% decline. Answers are judged by GPT-4o, which the authors report agrees with human experts more than 97% of the time.

The paper also tested fixes. Adding extracted user facts to the retrieval index raised recall by about 9.4% and accuracy by about 5.4% on average. Time-aware query expansion improved temporal-reasoning recall by 6.8% to 11.3%. For more on the time problem, see temporal reasoning in AI memory.

MemoryAgentBench: incremental input and forgetting

MemoryAgentBench (Hu, Wang and McAuley, UC San Diego; ICLR 2026) argues that older benchmarks either assume short contexts or test static long documents. Real memory agents receive information piece by piece, so it feeds long-context datasets to the agent in incremental chunks and asks questions afterwards.

It defines four competencies:

  • Accurate Retrieval: single- and multi-hop document QA, LongMemEval, and a new EventQA set built from novels.
  • Test-Time Learning: learning classification labels or movie preferences from examples in the history.
  • Long-Range Understanding: summarizing novels (∞Bench-Sum) and Detective QA.
  • Selective Forgetting: new FactConsolidation sets, where later facts overwrite earlier ones. Early versions called this “conflict resolution.”

The results are humbling. RAG agents beat the plain long-context baseline on retrieval, but long-context models did best on test-time learning and long-range understanding. All methods struggled with selective forgetting: multi-hop accuracy reached at most about 7%. Code and data are on GitHub and Hugging Face (ai-hyz/MemoryAgentBench).

BEAM: memory beyond a million tokens

BEAM (“Beyond a Million Tokens”, Tavakoli et al., ICLR 2026) tests what happens when history outgrows any context window. It has 100 conversations and 2,000 validated questions, with dialogues up to 10M tokens. It tests ten memory abilities: abstention, contradiction resolution, event ordering, information extraction, instruction following, knowledge update, multi-hop reasoning, preference following, summarization and temporal reasoning. Each reference answer is broken into atomic “nuggets” that an LLM judge marks as unsatisfied, partly or fully satisfied (0, 0.5 or 1). Results are reported per length tier: 100K, 500K, 1M and 10M tokens.

The authors found that models with 1M-token windows, with or without retrieval, degrade as conversations get longer. They also propose LIGHT, a memory framework with long-term episodic memory, short-term working memory and a scratchpad, which improved scores by 3.5% to 12.69% over the strongest baselines depending on the model.

BEAM is newer and fewer vendors report it, but it spreads systems out more than LoCoMo and LongMemEval, where top claimed scores now cluster in the high 80s to mid 90s. At 1M tokens, Mem0’s own comparison page lists Hindsight at 73.9 and Mem0 Platform at 64.1; at 10M, 64.1 and 48.6. Both systems sit within a point of each other on LongMemEval and LoCoMo, so the gap only shows on the longer test. These are vendor-run numbers.

FEB-30: scoring the memory store, not the answer

Every benchmark above scores an answer. Beyond Downstream QA (Toma et al., Amazon Web Services and the Technical University of Cluj-Napoca, October 2026) argues that this hides the thing you wanted to measure. A downstream QA score mixes memory quality with retrieval and generation quality, so a strong answer model can cover for a store full of stale facts, and a retrieval miss can get blamed on memory that stored the fact correctly.

The paper splits the two:

  • Extrinsic evaluation asks a question and judges the reply. That’s LoCoMo, LongMemEval, BEAM.
  • Intrinsic evaluation reads the store after each ingestion step and checks what’s in it.

Its benchmark, FEB-30, is 30 fact-evolution trajectories across 6 diagnostic categories (LLM compensation, retrieval bottleneck, classification, synthesis quality, temporal correctness, multi-author) at 3 difficulty levels. Documents are generated from a known fact trajectory, so the ground truth is deterministic and needs no human annotation, and the store can be inspected at every state. The intrinsic metrics are claim coverage (how much of the expected claim set the store entails), fact accuracy (claim coverage averaged over a trajectory), update detection recall (share of required updates or supersessions the system actually made) and false update rate (how often it changed a fact that should have stayed put).

Two results matter if you pick systems by published scores:

  • On the same system, intrinsic and extrinsic scores diverge by an average of 16 percentage points.
  • The system classified its own memory operations (create, update, reinforce, supersede) correctly only 20% of the time, a failure downstream QA never surfaced.

The paper deliberately doesn’t name the system it tested, describing it as a representative extract, embed, classify, synthesize design, and names Hindsight, Mem0 and AriadneMem only as examples of that pattern. None of them was scored, so there’s no ranking to read here. The abstract points to github.com/amazon-science/feb-30-benchmark for the benchmark, pipeline and scoring code; that URL still returned 404 on 18 October 2026, so expect to wait for the release.

What you can do today is the cheap version of the same idea: after ingesting a history, dump what your memory system stored and read it. Systems that keep evidence make this easier. Hindsight’s observations, for example, carry the supporting quotes and a proof count, so you can see which stored belief a wrong answer came from instead of guessing between memory and retrieval.

Bemula: can the system forget on request

The other gap is deletion. Bemula (Xie et al., October 2026) tests whether an agent can drop a fact when the user asks: 50 multi-session dialogues, 135 forget requests in six types, 1,742 questions, each dialogue paired with a control version without the forget request. Five suites check that the fact was stored, that it’s gone, that unrelated memories survived, that a similar fact can be learned again afterwards, and that the deleted fact can’t be pulled back out by an extraction attack.

No system tested passed more than 1.5% of forget requests end to end, and most passed none. Some left the fact extractable; others deleted far more than asked. Details, including which systems were tested, are on our AI memory decay page.

New domain benchmarks: decisions, shopping logs, returning to a chat

Three arXiv papers from the last week of October 2026 make the same point from different sides: finding the right memory is only the first step. Each one splits memory into abilities that older benchmarks scored as one number.

BenchmarkPaperDataWhat it splits apartMain finding
SciValidityBenchPan and Chen, 26 Oct 202664 task specs, 5 validity modules, 6 scientific domainsRecalling a result vs knowing whether it’s valid to act onGiving the model compiled “state” raised valid decisions by 9.2-18.8 points over a generic event graph; retrieval alone didn’t
BehaviorMemBenchWang et al., 26 Oct 2026Real e-commerce logs: 1,194 users, 17,462 behavior batches, 3+ yearsRemembering what a user did vs inferring what they wantMethod rankings differ between the two abilities; simple statistical baselines are competitive
Social re-entryWan et al., 29 Oct 2026Shared-snapshot tests across six LLM setupsFinding an old chat, choosing to return to it, rebuilding its contextPersistent memory raised replies to the right chat, but neither memory nor resurfacing reliably improved rebuilding the context

SciValidityBench asks whether a remembered fact should be used at all. Its example: the same test result supports a firm conclusion if the stopping rule was fixed in advance, but only exploration if it was the best of twenty results someone looked at. The memory has to keep that context, not just the number.

BehaviorMemBench builds memory from clicks and purchases instead of chat. It scores remember and infer separately, and the authors found a system can rank well on one and poorly on the other.

Social re-entry tests agents that sit in long-running group chats. Remembering a past conversation doesn’t mean the agent knows when to go back to it, or can pick up where it left off.

None of these papers tested Mem0, Zep or Hindsight, so they give no product ranking. What they suggest for your own tests: score retrieval, the decision made from it, and the answer as separate numbers, so you can see which step failed.

MSC and DMR: the older dialogue tests

Multi-Session Chat (MSC) (Xu, Szlam and Weston, Meta, 2021) is a human-human dataset where crowdworkers chat across sessions held “hours or days later.” Training episodes have 3 or 4 sessions; validation and test extend to 5. It was built to train and test dialogue models on long-term chat, scored mainly with perplexity.

Deep Memory Retrieval (DMR) comes from the MemGPT paper. It adds a sixth session to MSC with a question answerable only from the first five, and scores the answer with GPT-4-judged accuracy and ROUGE-L. MemGPT reported raising GPT-4 accuracy from 32.1% to 92.5%. Zep later reported 94.8% on DMR. With histories this short, DMR is now close to saturated.

How to read published memory benchmark scores

Nearly every memory vendor publishes LoCoMo or LongMemEval numbers. Treat them as self-reported. The LLM memory comparison collects the published scores for Mem0, Zep, Hindsight, Supermemory, MemOS, Cognee and others, with who ran each one.

Before comparing two numbers, check:

  • Answer model and judge model. A stronger answer model raises every score. LongMemEval’s script uses GPT-4o as judge; some vendors swap it.
  • Metric. F1, LLM-judged accuracy and Recall@k are not interchangeable. Recall@15 measures retrieval, not answers.
  • Dataset variant. LoCoMo10 vs the original 50; LongMemEval S vs M vs oracle; the cleaned LongMemEval release from September 2025.
  • Context budget. A system that returns 20K tokens of context isn’t comparable to one returning 1K. Mem0, for example, measured Hindsight returning 24K-44K tokens per retrieval against its own under 7K; Hindsight lets you cap this with max_tokens, so check what budget a score was run at.
  • Who ran it. Independent reproduction counts for more than a vendor’s own run.
  • What it actually measured. Every published score on this page is extrinsic: a judged answer, not the contents of the store. FEB-30 measured a 16-point average gap between the two on one system, so two systems a point apart on LongMemEval can be far apart on whether they update a changed fact.

How to run a memory benchmark yourself

The fastest way to test your own system is LongMemEval, because the data and judge script are public.

  1. Clone the repo: git clone https://github.com/xiaowu0162/LongMemEval.
  2. Download the data from Hugging Face into data/, for example longmemeval_oracle.json and longmemeval_s_cleaned.json from xiaowu0162/longmemeval-cleaned.
  3. Ingest each question’s haystack_sessions into your memory system, in date order (haystack_dates).
  4. Ask the question with question_date as the current date, and save the answer.
  5. Write a JSONL file with one {"question_id", "hypothesis"} object per line.
  6. Score it from src/evaluation: python3 evaluate_qa.py gpt-4o your_hypothesis_file ../../data/longmemeval_oracle.json (needs OPENAI_API_KEY).
  7. Print per-type results: python3 print_qa_metrics.py gpt-4o your_hypothesis_file.log ../../data/longmemeval_oracle.json.

Here’s a full-context baseline for steps 3 to 5. It pastes the whole history into the prompt, which is the number any memory system should beat on cost or accuracy:

 1import json
 2from openai import OpenAI
 3
 4client = OpenAI()
 5data = json.load(open("data/longmemeval_oracle.json"))
 6
 7def render_history(item):
 8    parts = []
 9    for date, session in zip(item["haystack_dates"], item["haystack_sessions"]):
10        turns = "\n".join(f'{t["role"]}: {t["content"]}' for t in session)
11        parts.append(f"### Session on {date}\n{turns}")
12    return "\n\n".join(parts)
13
14with open("hypotheses.jsonl", "w") as out:
15    for item in data:
16        prompt = (
17            f"{render_history(item)}\n\n"
18            f"Current date: {item['question_date']}\n"
19            f"Question: {item['question']}\n"
20            "Answer from the chat history. If it has no answer, say so."
21        )
22        resp = client.chat.completions.create(
23            model="gpt-4o-mini",
24            messages=[{"role": "user", "content": prompt}],
25        )
26        out.write(json.dumps({
27            "question_id": item["question_id"],
28            "hypothesis": resp.choices[0].message.content,
29        }) + "\n")

To test a memory system, replace render_history with your system’s ingest and search calls, and log how many tokens of context it returns per question. Run the same answer model and judge for every system you compare. For a list of systems to try, see open-source memory systems compared.