# LLM Memory Evaluation: Benchmarks for Agent Memory

> How to evaluate LLM and agent memory: LoCoMo, LongMemEval, MemoryAgentBench, BEAM, FEB-30, Bemula, SciValidityBench, BehaviorMemBench, MSC and DMR compared by size, tasks and metrics, plus how to run one.

Source: https://aiagentmemory.org/articles/llm-memory-evaluation/ · Updated 2026-10-30 · aiagentmemory.org


**LLM memory evaluation** means testing whether a model or agent can recall, update and reason over information from earlier sessions. The standard benchmarks are **LoCoMo** and **LongMemEval**, with **MemoryAgentBench** and **BEAM** as harder, newer tests. Each feeds a long chat history to the system, then asks questions only that history can answer.

The tests differ in what they stress: plain recall, time, changed facts, forgetting, or scale. Picking the wrong one tells you little about your use case. Two 2026 benchmarks, **FEB-30** and **Bemula**, go further and score the memory store itself rather than the answer, and three late-October papers split recall from the decision made with it. This page covers what each benchmark measures, how it's scored, and how to run one yourself.

## What is LLM memory evaluation?

**LLM memory evaluation is the measurement of how well a language model or memory system stores information from past interactions and uses it later. A benchmark supplies a long history (chat sessions or documents), asks questions that depend on it, and scores the answers with exact match, F1 or an LLM judge.**

Zhang et al.'s [survey of agent memory](https://arxiv.org/abs/2404.13501) splits evaluation in two. **Direct evaluation** scores the memory itself, for example whether the right fact was retrieved. **Indirect evaluation** scores an end task, like answering a question or holding a conversation, that needs memory to succeed. Most benchmarks below do both: they score the final answer and, where evidence is labeled, the retrieval step.

For background on what's being tested, see our guide to [AI agent memory](/articles/ai-agent-memory-explained/). For the research taxonomies behind these tests, see the [LLM memory survey papers](/articles/llm-memory-survey/).

## LLM memory benchmarks compared

| Benchmark | Year, venue | Data | What it tests | Main metric |
|---|---|---|---|---|
| **MSC** (Multi-Session Chat) | 2021 | Up to 5 chat sessions per episode | Long-term open-domain dialogue | Perplexity |
| **DMR** (Deep Memory Retrieval) | 2023, MemGPT paper | MSC plus a 6th session with a question | Recall of facts from past sessions | LLM-judged accuracy, ROUGE-L |
| **LoCoMo** | 2024, ACL | 10 conversations released (50 in first version), ~300 turns, ~9K tokens, up to 35 sessions | QA, event summarization, multimodal dialogue | F1 (QA) |
| **LongMemEval** | 2024, ICLR 2025 | 500 questions; S ≈ 115K tokens, M ≈ 500 sessions (≈1.5M tokens) | Extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention | GPT-4o-judged accuracy; Recall@k, NDCG@k |
| **MemoryAgentBench** | 2025, ICLR 2026 | Long-context datasets turned into incremental multi-turn input | Accurate retrieval, test-time learning, long-range understanding, selective forgetting | Task-specific (accuracy, LLM-judged) |
| **BEAM** | 2025, ICLR 2026 | 100 conversations, 2,000 questions, up to 10M tokens | Ten abilities, incl. contradiction resolution, event ordering, preference following | LLM-judged nugget scores; Kendall tau-b for event ordering |
| **FEB-30** | 2026, arXiv | 30 generated fact-evolution trajectories, 6 diagnostic categories, 3 difficulty levels | The memory store itself: whether it holds the right facts and updates them when they change | Fact accuracy, update detection recall, false update rate |
| **Bemula** | 2026, arXiv | 50 multi-session dialogues, 135 forget requests, 1,742 questions | Deleting a memory on request, and whether unrelated memories survive | Pass rate across five test suites |
| **SciValidityBench** | 2026, arXiv | 64 task specs across 6 scientific domains | Whether a remembered result is valid to use in a decision | Valid-decision rate |
| **BehaviorMemBench** | 2026, arXiv | Real e-commerce logs, 1,194 users over 3+ years | Remembering user behavior vs inferring preferences, scored apart | Remember and infer scores against future-behavior labels |

The sections below describe each one from its paper. FEB-30 and Bemula are the newest, and they test
two things the older benchmarks never scored: the contents of the store, and deletion.

## LoCoMo: very long-term conversational memory

**[LoCoMo](https://arxiv.org/abs/2402.17753)** (Maharana et al., Snap Research) is the benchmark most vendors cite. Its conversations average about **300 turns and 9K tokens over up to 35 sessions**, generated by LLM agents with personas and event timelines, then checked by humans.

The QA task has five categories: **single-hop, multi-hop, temporal, open-domain knowledge and adversarial** (questions with no answer in the conversation). The paper scores QA with F1. In its results, humans scored 87.9 F1 overall while the best model tested, GPT-3.5-turbo-16K, scored 37.8.

Two details trip people up:

- **The released set is smaller than the paper's.** The [LoCoMo repo](https://github.com/snap-research/locomo) ships `locomo10.json` with **ten** conversations, "a subset of the conversations released previously" (the first arXiv version had 50). Vendor scores labeled LoCoMo or LoCoMo10 use this set.
- **Vendors often change the metric.** Many memory papers, such as Mem0's ([arXiv 2504.19413](https://arxiv.org/abs/2504.19413)), report LLM-as-judge accuracy instead of the original F1, so their numbers aren't on the same scale as the LoCoMo paper.

## LongMemEval: five long-term memory abilities

**[LongMemEval](https://arxiv.org/abs/2410.10813)** (Wu et al., ICLR 2025) targets chat assistants. It has **500 hand-curated questions**, each with its own chat history in which the evidence is hidden among unrelated sessions.

It tests five abilities:

1. **Information extraction:** recall a detail the user or assistant said.
2. **Multi-session reasoning:** combine facts spread across sessions.
3. **Temporal reasoning:** answer questions about when things happened.
4. **Knowledge updates:** use the newest value when a fact changed.
5. **Abstention:** say "I don't know" when the history has no answer.

Question types in the data are `single-session-user`, `single-session-assistant`, `single-session-preference`, `multi-session`, `knowledge-update` and `temporal-reasoning`. Abstention questions have a `question_id` ending in `_abs`.

There are three history sizes. **Oracle** holds only the evidence sessions. **LongMemEval_S** is about 115K tokens per question. **LongMemEval_M** has about 500 sessions, roughly 1.5M tokens. In the paper, GPT-4o dropped from 0.870 accuracy on oracle histories to 0.606 on the full S history, about a 30% decline. Answers are judged by GPT-4o, which the authors report agrees with human experts more than 97% of the time.

The paper also tested fixes. Adding extracted user facts to the retrieval index raised recall by about 9.4% and accuracy by about 5.4% on average. Time-aware query expansion improved temporal-reasoning recall by 6.8% to 11.3%. For more on the time problem, see [temporal reasoning in AI memory](/articles/temporal-reasoning-ai-memory/).

## MemoryAgentBench: incremental input and forgetting

**[MemoryAgentBench](https://arxiv.org/abs/2507.05257)** (Hu, Wang and McAuley, UC San Diego; ICLR 2026) argues that older benchmarks either assume short contexts or test static long documents. Real memory agents receive information piece by piece, so it feeds long-context datasets to the agent in **incremental chunks** and asks questions afterwards.

It defines four competencies:

- **Accurate Retrieval:** single- and multi-hop document QA, LongMemEval, and a new EventQA set built from novels.
- **Test-Time Learning:** learning classification labels or movie preferences from examples in the history.
- **Long-Range Understanding:** summarizing novels (∞Bench-Sum) and Detective QA.
- **Selective Forgetting:** new FactConsolidation sets, where later facts overwrite earlier ones. Early versions called this "conflict resolution."

The results are humbling. RAG agents beat the plain long-context baseline on retrieval, but long-context models did best on test-time learning and long-range understanding. **All methods struggled with selective forgetting**: multi-hop accuracy reached at most about 7%. Code and data are on [GitHub](https://github.com/HUST-AI-HYZ/MemoryAgentBench) and Hugging Face (`ai-hyz/MemoryAgentBench`).

## BEAM: memory beyond a million tokens

**[BEAM](https://arxiv.org/abs/2510.27246)** ("Beyond a Million Tokens", Tavakoli et al., ICLR 2026) tests what happens when history outgrows any context window. It has **100 conversations and 2,000 validated questions**, with dialogues up to **10M tokens**. It tests **ten memory abilities**: abstention, contradiction resolution, event ordering, information extraction, instruction following, knowledge update, multi-hop reasoning, preference following, summarization and temporal reasoning. Each reference answer is broken into atomic "nuggets" that an LLM judge marks as unsatisfied, partly or fully satisfied (0, 0.5 or 1). Results are reported per length tier: 100K, 500K, 1M and 10M tokens.

The authors found that models with 1M-token windows, with or without retrieval, degrade as conversations get longer. They also propose **LIGHT**, a memory framework with long-term episodic memory, short-term working memory and a scratchpad, which improved scores by 3.5% to 12.69% over the strongest baselines depending on the model.

BEAM is newer and fewer vendors report it, but it spreads systems out more than LoCoMo and LongMemEval, where top claimed scores now cluster in the high 80s to mid 90s. At 1M tokens, Mem0's own [comparison page](https://mem0.ai/compare/mem0-vs-hindsight) lists Hindsight at 73.9 and Mem0 Platform at 64.1; at 10M, 64.1 and 48.6. Both systems sit within a point of each other on LongMemEval and LoCoMo, so the gap only shows on the longer test. These are vendor-run numbers.

## FEB-30: scoring the memory store, not the answer

Every benchmark above scores an **answer**. [Beyond Downstream QA](https://arxiv.org/abs/2610.22209)
(Toma et al., Amazon Web Services and the Technical University of Cluj-Napoca, October 2026) argues that
this hides the thing you wanted to measure. A downstream QA score mixes memory quality with retrieval and
generation quality, so a strong answer model can cover for a store full of stale facts, and a retrieval
miss can get blamed on memory that stored the fact correctly.

The paper splits the two:

- **Extrinsic evaluation** asks a question and judges the reply. That's LoCoMo, LongMemEval, BEAM.
- **Intrinsic evaluation** reads the store after each ingestion step and checks what's in it.

Its benchmark, **FEB-30**, is **30 fact-evolution trajectories** across **6 diagnostic categories** (LLM
compensation, retrieval bottleneck, classification, synthesis quality, temporal correctness, multi-author)
at **3 difficulty levels**. Documents are generated from a known fact trajectory, so the ground truth is
deterministic and needs no human annotation, and the store can be inspected at every state. The intrinsic
metrics are **claim coverage** (how much of the expected claim set the store entails), **fact accuracy**
(claim coverage averaged over a trajectory), **update detection recall** (share of required updates or
supersessions the system actually made) and **false update rate** (how often it changed a fact that should
have stayed put).

Two results matter if you pick systems by published scores:

- On the same system, intrinsic and extrinsic scores **diverge by an average of 16 percentage points**.
- The system classified its own memory operations (create, update, reinforce, supersede) correctly only
  **20%** of the time, a failure downstream QA never surfaced.

The paper deliberately doesn't name the system it tested, describing it as a representative
extract, embed, classify, synthesize design, and names Hindsight, Mem0 and AriadneMem only as examples of
that pattern. None of them was scored, so there's no ranking to read here. The abstract points to
`github.com/amazon-science/feb-30-benchmark` for the benchmark, pipeline and scoring code; that URL still
returned 404 on 18 October 2026, so expect to wait for the release.

What you can do today is the cheap version of the same idea: after ingesting a history, **dump what your
memory system stored and read it**. Systems that keep evidence make this easier. Hindsight's observations,
for example, carry the supporting quotes and a proof count, so you can see which stored belief a wrong
answer came from instead of guessing between memory and retrieval.

## Bemula: can the system forget on request

The other gap is deletion. [Bemula](https://arxiv.org/abs/2610.18052) (Xie et al., October 2026) tests
whether an agent can drop a fact when the user asks: **50 multi-session dialogues, 135 forget requests in
six types, 1,742 questions**, each dialogue paired with a control version without the forget request. Five
suites check that the fact was stored, that it's gone, that unrelated memories survived, that a similar
fact can be learned again afterwards, and that the deleted fact can't be pulled back out by an extraction
attack.

No system tested passed more than **1.5%** of forget requests end to end, and most passed none. Some left
the fact extractable; others deleted far more than asked. Details, including which systems were tested, are
on our [AI memory decay](/articles/ai-memory-decay/) page.

## New domain benchmarks: decisions, shopping logs, returning to a chat

Three arXiv papers from the last week of October 2026 make the same point from different sides: finding
the right memory is only the first step. Each one splits memory into abilities that older benchmarks
scored as one number.

| Benchmark | Paper | Data | What it splits apart | Main finding |
|---|---|---|---|---|
| **SciValidityBench** | [Pan and Chen, 26 Oct 2026](https://arxiv.org/abs/2610.34767) | 64 task specs, 5 validity modules, 6 scientific domains | Recalling a result vs knowing whether it's valid to act on | Giving the model compiled "state" raised valid decisions by 9.2-18.8 points over a generic event graph; retrieval alone didn't |
| **BehaviorMemBench** | [Wang et al., 26 Oct 2026](https://arxiv.org/abs/2610.34790) | Real e-commerce logs: 1,194 users, 17,462 behavior batches, 3+ years | Remembering what a user did vs inferring what they want | Method rankings differ between the two abilities; simple statistical baselines are competitive |
| **Social re-entry** | [Wan et al., 29 Oct 2026](https://arxiv.org/abs/2610.39781) | Shared-snapshot tests across six LLM setups | Finding an old chat, choosing to return to it, rebuilding its context | Persistent memory raised replies to the right chat, but neither memory nor resurfacing reliably improved rebuilding the context |

**SciValidityBench** asks whether a remembered fact should be used at all. Its example: the same test
result supports a firm conclusion if the stopping rule was fixed in advance, but only exploration if it was
the best of twenty results someone looked at. The memory has to keep that context, not just the number.

**BehaviorMemBench** builds memory from clicks and purchases instead of chat. It scores *remember* and
*infer* separately, and the authors found a system can rank well on one and poorly on the other.

**Social re-entry** tests agents that sit in long-running group chats. Remembering a past conversation
doesn't mean the agent knows when to go back to it, or can pick up where it left off.

None of these papers tested Mem0, Zep or Hindsight, so they give no product ranking. What they suggest for
your own tests: score retrieval, the decision made from it, and the answer as separate numbers, so you can
see which step failed.

## MSC and DMR: the older dialogue tests

**[Multi-Session Chat (MSC)](https://arxiv.org/abs/2107.07567)** (Xu, Szlam and Weston, Meta, 2021) is a human-human dataset where crowdworkers chat across sessions held "hours or days later." Training episodes have 3 or 4 sessions; validation and test extend to 5. It was built to train and test dialogue models on long-term chat, scored mainly with perplexity.

**Deep Memory Retrieval (DMR)** comes from the [MemGPT paper](https://arxiv.org/abs/2310.08560). It adds a sixth session to MSC with a question answerable only from the first five, and scores the answer with GPT-4-judged accuracy and ROUGE-L. MemGPT reported raising GPT-4 accuracy from 32.1% to 92.5%. Zep later reported 94.8% on DMR. With histories this short, DMR is now close to saturated.

## How to read published memory benchmark scores

Nearly every memory vendor publishes LoCoMo or LongMemEval numbers. **Treat them as self-reported.** The [LLM memory comparison](/articles/llm-memory-comparison/) collects the published scores for Mem0, Zep, Hindsight, Supermemory, MemOS, Cognee and others, with who ran each one.

Before comparing two numbers, check:

- **Answer model and judge model.** A stronger answer model raises every score. LongMemEval's script uses GPT-4o as judge; some vendors swap it.
- **Metric.** F1, LLM-judged accuracy and Recall@k are not interchangeable. Recall@15 measures retrieval, not answers.
- **Dataset variant.** LoCoMo10 vs the original 50; LongMemEval S vs M vs oracle; the cleaned LongMemEval release from September 2025.
- **Context budget.** A system that returns 20K tokens of context isn't comparable to one returning 1K. Mem0, for example, measured Hindsight returning 24K-44K tokens per retrieval against its own under 7K; Hindsight lets you cap this with `max_tokens`, so check what budget a score was run at.
- **Who ran it.** Independent reproduction counts for more than a vendor's own run.
- **What it actually measured.** Every published score on this page is extrinsic: a judged answer, not the
  contents of the store. FEB-30 measured a 16-point average gap between the two on one system, so two
  systems a point apart on LongMemEval can be far apart on whether they update a changed fact.

## How to run a memory benchmark yourself

The fastest way to test your own system is LongMemEval, because the data and judge script are public.

1. **Clone the repo:** `git clone https://github.com/xiaowu0162/LongMemEval`.
2. **Download the data** from Hugging Face into `data/`, for example `longmemeval_oracle.json` and `longmemeval_s_cleaned.json` from `xiaowu0162/longmemeval-cleaned`.
3. **Ingest each question's `haystack_sessions`** into your memory system, in date order (`haystack_dates`).
4. **Ask the `question`** with `question_date` as the current date, and save the answer.
5. **Write a JSONL file** with one `{"question_id", "hypothesis"}` object per line.
6. **Score it** from `src/evaluation`: `python3 evaluate_qa.py gpt-4o your_hypothesis_file ../../data/longmemeval_oracle.json` (needs `OPENAI_API_KEY`).
7. **Print per-type results:** `python3 print_qa_metrics.py gpt-4o your_hypothesis_file.log ../../data/longmemeval_oracle.json`.

Here's a full-context baseline for steps 3 to 5. It pastes the whole history into the prompt, which is the number any memory system should beat on cost or accuracy:

```python
import json
from openai import OpenAI

client = OpenAI()
data = json.load(open("data/longmemeval_oracle.json"))

def render_history(item):
    parts = []
    for date, session in zip(item["haystack_dates"], item["haystack_sessions"]):
        turns = "\n".join(f'{t["role"]}: {t["content"]}' for t in session)
        parts.append(f"### Session on {date}\n{turns}")
    return "\n\n".join(parts)

with open("hypotheses.jsonl", "w") as out:
    for item in data:
        prompt = (
            f"{render_history(item)}\n\n"
            f"Current date: {item['question_date']}\n"
            f"Question: {item['question']}\n"
            "Answer from the chat history. If it has no answer, say so."
        )
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
        )
        out.write(json.dumps({
            "question_id": item["question_id"],
            "hypothesis": resp.choices[0].message.content,
        }) + "\n")
```

To test a memory system, replace `render_history` with your system's ingest and search calls, and log how many tokens of context it returns per question. Run the same answer model and judge for every system you compare. For a list of systems to try, see [open-source memory systems compared](/articles/open-source-memory-systems-compared/).


## FAQ

### What is the best benchmark for LLM memory?

LongMemEval (ICLR 2025) and LoCoMo (ACL 2024) are the most reported. LongMemEval has 500 questions over five memory abilities; LoCoMo tests very long multi-session chats. For harder tests, MemoryAgentBench adds test-time learning and selective forgetting, and BEAM scales conversations up to 10M tokens.

### How is LongMemEval scored?

Each answer is judged correct or incorrect by GPT-4o using the evaluate_qa.py script in the LongMemEval repo. The authors report that the GPT-4o judge agrees with human experts more than 97% of the time. Retrieval can also be scored with Recall@k and NDCG@k against labeled evidence sessions.

### What is intrinsic vs extrinsic memory evaluation?

Extrinsic evaluation asks the agent a question and scores the answer, which is what LoCoMo, LongMemEval and BEAM do. Intrinsic evaluation inspects the memory store itself: are the right facts in there, were they updated when they changed. The FEB-30 paper (October 2026) found the two scores diverge by an average of 16 percentage points on the same system, so a good QA score can hide a broken store.

### What are the newest agent memory benchmarks?

In late October 2026, three arXiv papers added domain tests: SciValidityBench (whether a remembered scientific result is valid to act on), BehaviorMemBench (memory built from 3+ years of real e-commerce logs, scoring remembering and inferring apart) and a social re-entry test (finding, returning to and rebuilding an old conversation). Earlier in October, FEB-30 scored the memory store itself and Bemula tested deletion.

### Can I trust memory benchmark scores published by vendors?

Treat them as self-reported claims. Each vendor picks its own answer model, judge model, prompts and retrieval budget, so scores from different vendors aren't directly comparable. Rerun the benchmark yourself with fixed settings before choosing a system.

