Agent Memory Leaderboard: Benchmarks and Scores

There is no official agent memory leaderboard. The benchmark authors (LongMemEval, LoCoMo, BEAM) publish datasets, not rankings. The leaderboards you can find are run by memory vendors: agentmemorybenchmark.ai by Vectorize, the company that makes Hindsight; OmniMemEval by MemTensor, the company that makes MemOS; and score pages from Mem0, Zep, Honcho and past.dev.

So every ranking has an owner, and every score depends on the answer model, judge and token budget that owner chose. This page lists where the leaderboards are, what each benchmark measures, the published scores with who reported each one, and how to read them. Everything was checked on the source pages on 12 October 2026.

What is an agent memory leaderboard?

An agent memory leaderboard is a table that ranks memory systems (Mem0, Zep, Hindsight, MemOS and others) by their scores on long-conversation benchmarks such as LongMemEval, LoCoMo and BEAM. It either runs every system through one test harness with fixed settings, or collects scores that each vendor or paper published.

The difference matters. A same-harness leaderboard runs every system with the same answer model, judge and prompts, so rows can be compared, but the harness owner chose how to configure each competitor. A collected leaderboard copies published numbers, so each row was run under different settings and rows can’t be compared directly. Most pages mix both. For how the benchmarks work and how to run one yourself, see the LLM memory evaluation guide.

Where to find agent memory leaderboards

Some facts about these leaderboards that change how you read them:

• AMB is not independent. Vectorize built it, and its launch post says the team built it “because we wanted to be honest about how Hindsight performs.” Its harness supports other providers, including Mem0, Mastra and Cognee, but the results it publishes from its own runs cover Hindsight, a plain hybrid-search baseline and Cognee on parts of LoCoMo and PersonaMem. Competitor scores on AMB come from the external section.
• OmniMemEval is not independent either. It’s run by the maker of MemOS, and MemOS leads its LoCoMo and LongMemEval tables. Its results page says competitor adapters “are not a claim that every adapter has reached a globally optimal product-specific configuration.”
• MemoryBench (repo), from Supermemory, is a harness for running Supermemory, Mem0 and Zep on LoCoMo, LongMemEval and ConvoMem yourself. It doesn’t publish a leaderboard.
• The LongMemEval repo has no leaderboard. Its latest news (May 2026) is LongMemEval-V2 for agentic contexts.

What the main memory benchmarks measure

LongMemEval and LoCoMo are the most reported, and top self-reported scores on both now sit between 89% and 95%. That leaves little room to tell systems apart. BEAM spreads systems out more, because at 1M and 10M tokens no context window holds the whole history. PersonaMem and LifeBench are harder still: the PersonaMem paper found frontier models near 50% accuracy, and the LifeBench abstract says top memory systems “reach just 55.2% accuracy.” More detail on each is in the memory evaluation guide, and on the time questions in temporal reasoning in AI memory.

Agent memory benchmark results: who reported what

These are the scores each vendor publishes for itself, with the source and the models where the source names them.

Every row is vendor-run. Hindsight’s README adds that research collaborators at Virginia Tech and The Washington Post reproduced its results; that is a reproduction by collaborators, not an independent audit. Hindsight’s site also reports PersonaMem 32K at 86.6 and LifeBench EN at 71.5. BEAM scores above are nugget scores shown out of 100.

Two things stand out. On LongMemEval and LoCoMo, the top four systems are within about five points of each other, which is less than the effect of changing the answer model. On BEAM 10M, the gap is larger: Hindsight reports 64.1 against 48.6 for Mem0 and 40.9 for Honcho, and Mem0’s comparison page shows the same Hindsight and Mem0 figures. past.dev reports far higher BEAM scores than anyone else, from a run where one model both answers and judges; no other source has reproduced them.

Same-harness results: OmniMemEval

OmniMemEval is the largest published run where every system used the same answer model (GPT-4.1-mini) and judge (GPT-4o-mini). Remember it’s run by the maker of MemOS.

Source: OmniMemEval results snapshot, checked 12 October 2026. Selected rows; the page lists 14 backends plus PersonaMem v2 and HaluMem.

What this table shows:

• Scores drop a lot under someone else’s harness. Mem0 reports 94.4 on LongMemEval; here Mem0 cloud scores 56.0. Zep reports 94.7 on LoCoMo; here it scores 63.8. Hindsight reports 92 on LoCoMo; here it scores 82.0. Only MemOS matches its own README, because its README numbers come from this harness.
• Hindsight is highest on BEAM 10M (59.75) and within a point of the top at 100K, under a competitor’s settings. That matches the pattern in the vendor numbers: its lead shows on the longest histories.
• Hindsight is middle of the pack on LongMemEval here (72.20). Most of the drop is one category: 14.3% on single-session-assistant questions (things the assistant said). It was run self-hosted, not on Hindsight Cloud.

Why leaderboard numbers aren’t directly comparable

Two scores for the same benchmark can differ by 30 points without either being false. Before you compare, check:

1. Answer model. A stronger model reading the same retrieved memory scores higher. Supermemory’s own report shows 85.2 with Gemini 3 Pro and 84.6 with GPT-5 for the same memory.
2. Judge model and prompt. LongMemEval’s script uses GPT-4o. Zep uses GPT-5.4, OmniMemEval uses GPT-4o-mini, and AMB uses Gemini models.
3. Metric. LoCoMo’s paper uses F1; vendors use LLM-judged accuracy. Supermemory’s 95% Recall@15 on LongMemEval measures retrieval, not answers.
4. Dataset variant. LoCoMo10 with or without adversarial questions; LongMemEval S, M or oracle, original or the September 2025 cleaned release; BEAM tier.
5. Retrieval mode. One search call per question, or an agent that searches several times. AMB publishes Hindsight results in more than one mode, and the scores differ.
6. Token budget. A system that hands the answer model 40K tokens of context has an easier job than one that hands it 2K.

Tokens per retrieval: the cost axis

Accuracy without token counts hides the cost. Every retrieved token is paid again on every LLM call.

Hindsight returns about 4x more context per retrieval than Mem0 by default. The budget is a setting (max_tokens on recall), so you can trade some accuracy for fewer tokens, but the published scores were run at the larger budget. Compare accuracy and tokens together, not one alone.

Where Hindsight fits on these leaderboards

Hindsight is an MIT-licensed agent memory server from Vectorize. On the published data it’s a strong choice when:

• Histories are long. It reports 73.9 on BEAM 1M and 64.1 on BEAM 10M, 9.8 and 15.5 points over Mem0 by Mem0’s own page, and it has the top BEAM 10M score in MemTensor’s OmniMemEval run, where a competitor set up every system.
• You want every feature open source. Graph and temporal recall, consolidation into observations, mental models and reflect are all in the MIT release. Mem0 keeps graph memory and temporal reasoning on its paid Pro plan.
• The agent should reason over memory. reflect answers questions that need thinking across many memories, not only lookup.

Its limits on the same data: it returns more tokens per retrieval than Mem0 or Zep, writes cost more because retain and consolidation both call an LLM, and its README says it “may be overkill” for simple workflows. It also has no published compliance certifications, where Mem0 lists SOC 2 Type 1 and HIPAA. On LongMemEval and LoCoMo its self-reported lead over Mem0 and Zep is within a point or two, or behind. More in Mem0 vs Hindsight and the Hindsight explainer; for every system’s license and storage model, see the LLM memory comparison.

How to use a memory leaderboard to choose

1. Pick the benchmark that looks like your data. Chat assistants: LongMemEval. Very long histories: BEAM 1M or 10M. Personalization: PersonaMem.
2. Find out who ran each score. Vendor-run, competitor-run and paper-run numbers carry different weight.
3. Match the settings. Compare only rows with the same answer model, judge and dataset variant, or use a same-harness table.
4. Read tokens per retrieval next to accuracy. A small accuracy gain at 5x the context may cost more than it’s worth.
5. Shortlist two or three systems, then run them on your own data with one fixed answer model and judge. AMB, OmniMemEval and MemoryBench are all open harnesses you can fork for this.