Vector Databases for LLM Memory: How to Choose One

14 min read

How vector databases store long-term memory for LLM agents, what memory needs beyond RAG, and how pgvector, Qdrant, Chroma, Milvus, Pinecone and Redis compare.

A vector database for LLM memory stores past messages, facts and documents as embeddings, then finds the ones closest in meaning to the current query so they can be added to the prompt. It’s the storage layer under most long-term memory systems. For most teams the best choice is the database they already run (pgvector in Postgres, or Redis), with a dedicated vector database like Qdrant, Milvus or Pinecone once scale or multi-tenancy demands it.

This page explains how vector memory works, what agent memory needs that plain RAG doesn’t, and how the main options compare as of October 2026.

For GitHub stars and monthly downloads of these databases, see vector database statistics.

What is a vector database for LLM memory?

A vector database for LLM memory is a store that keeps text as numerical embeddings alongside metadata (user, time, type) and answers nearest-neighbor queries fast. An agent uses it to save what it learns and, on each turn, retrieve the few memories most similar to the current request.

The model itself is stateless: it remembers nothing between calls. A vector store gives it a searchable external memory. The MemGPT paper (Packer et al., 2023), for example, kept its “archival storage” in PostgreSQL and searched it through function calls. Today’s memory frameworks do the same with a choice of backends.

For the bigger picture of where storage fits next to extraction and retrieval, see AI agent memory explained.

How vector memory works

The loop has two halves.

Write path:

  1. Take a message, or a fact extracted from it by an LLM.
  2. Turn it into an embedding with an embedding model.
  3. Store the vector with metadata: user_id, timestamp, memory type, source message ID.

Read path:

  1. Embed the current query.
  2. Run an approximate nearest-neighbor (ANN) search, filtered to the right user and time range.
  3. Optionally combine with keyword search and rerank.
  4. Put the top few results into the prompt.

ANN indexes

Searching every vector is exact but slow at scale, so databases build an index. Two types dominate:

  • HNSW (hierarchical navigable small world) builds a layered graph of neighbors. It’s fast and accurate but uses more memory. pgvector’s defaults are m = 16 and ef_construction = 64, with hnsw.ef_search = 40 at query time (pgvector README).
  • IVF (inverted file) clusters vectors into lists and searches only the closest lists. It builds faster and uses less memory but needs tuning (lists, probes).

Redis’s vector search docs give a useful rule of thumb: use a FLAT (exact) index for small datasets “(< 1M vectors)” or when perfect accuracy matters, and HNSW for larger ones.

What agent memory needs beyond RAG

A RAG index over documents is written once and read many times. Agent memory is different, and those differences should drive the choice of database.

NeedWhy memory needs itWhat to look for
Per-user isolationUsers must never see each other’s memoriesFast metadata filters, tenant partitioning
Frequent small writesEvery conversation turn can add memoriesGood upsert performance, no full reindex
Updates and deletesFacts change; users ask to be forgottenDelete by ID or filter, quick visibility
Time filters“What did I say last week?”Range filters on timestamp fields
Exact matchesNames, order IDs, product codesHybrid search (BM25 or sparse vectors plus dense)
Filtered recallFilters shouldn’t silently drop resultsFilter-aware ANN (pgvector iterative scans, Qdrant filtering)

Two of these trip people up most. Multi-tenancy: Qdrant’s multitenancy guide says “Creating a separate collection for each tenant is rarely the most efficient approach” and recommends one collection with a tenant field indexed with is_tenant=true. Pinecone’s docs recommend “one namespace per tenant.” Filtered search: with plain HNSW, a strict filter can leave fewer results than requested. pgvector 0.8.0 added iterative index scans (hnsw.iterative_scan) to keep scanning until enough rows pass the filter.

Vector databases compared for LLM memory

Facts from each project’s docs and GitHub, checked 8 October 2026.

DatabaseTypeLicenseGood fit for memory when
pgvectorPostgres extensionPostgreSQL LicenseYou already run Postgres and want memory next to app data, with SQL joins and transactions
QdrantDedicated vector DBApache-2.0You need rich payload filters, tenant partitioning and sparse+dense hybrid search
ChromaEmbedded or serverApache-2.0You want the simplest local start (in-memory or on-disk client)
MilvusDistributed vector DBApache-2.0You expect very large collections; Milvus Lite covers prototypes
WeaviateDedicated vector DBBSD-3-Clause (parts under the Weaviate License)You want built-in hybrid search and modules for embedding
PineconeManaged service onlyCommercialYou want no infrastructure, with namespaces per user
LanceDBEmbedded library on the Lance columnar formatApache-2.0You want an embedded store that also handles multimodal data
Redis / Amazon MemoryDBIn-memory database with vector searchRedis 8: AGPLv3, RSALv2 or SSPLv1; MemoryDB is managedLatency matters most and you already use Redis

A few notes:

  • pgvector supports HNSW and IVFFlat, and up to 2,000 dimensions per indexed vector column (4,000 with halfvec). Distance operators include <=> for cosine and <#> for negative inner product. It needs Postgres 13 or later today; the unreleased 0.9.0 drops Postgres 13 and 14, per its changelog.
  • Redis supports FLAT, HNSW and, since Redis 8.2, SVS-VAMANA indexes, with metadata filters on text, tag, numeric and geo fields.
  • Amazon MemoryDB adds vector search to a durable Valkey/Redis-compatible service. AWS’s docs list retrieval-augmented generation among its use cases and claim, as of June 2024, the “fastest vector search performance at the highest recall rates among popular vector databases on AWS” (a vendor benchmark).
  • Microsoft Kernel Memory, once a common .NET option, is now archived. Its README says it “is an archived research project” and the code “serves as a learning resource, not production software.”

What changed for memory workloads (July to October 2026)

Latest releases and stars from GitHub on 25 October 2026, with the changes that matter for agent memory from each project’s release notes.

DatabaseLatest releaseGitHub starsChange that matters for memory
pgvector0.8.7 (1 Oct 2026)~23,500Bug fixes only: 0.8.7 fixes a buffer overflow in IVFFlat index builds, so upgrade if you use IVFFlat. 0.9.0 (unreleased) adds async I/O for HNSW.
Qdrant1.20.0 (22 Oct 2026)~35,200BM25 scoring directly on the full-text payload index, so keyword matching on names and IDs no longer needs a separate sparse vector. Also a substring filter and a code tokenizer.
Milvus3.0.3 (14 Oct 2026); 3.0.0 on 29 Jul~46,500Add, backfill and drop columns while serving, so switching embedding models doesn’t need a full rebuild with downtime. BM25 output fields can be computed over existing data.
Weaviate1.40.2 (22 Oct 2026)~16,900Security patch only (certificate fix); no new features.
LanceDB0.40.0 (7 Oct 2026)~11,800Built on Lance 13.0.0; mostly OAuth, secrets and materialized-view APIs. Has breaking changes, so read the notes before upgrading.
Chroma1.5.9 (5 May 2026)~29,600No new GitHub release since May.

The Qdrant change is the one most memory apps will feel. Its full-text search docs say “Qdrant computes BM25 scores from a full-text index on a payload field”: create the text index with "scoring": true and query it with models.TextQuery. Two limits apply: each shard keeps its own term statistics, and after deletes or updates scores can shift slightly until the optimizer rebuilds segments. Memory stores have lots of small deletes, so expect a little drift.

The Milvus change answers a problem every long-lived memory store hits: the embedding model you started with gets replaced, and old memories need new vectors. With online backfill you add the new vector column, fill it offline, and Milvus indexes it while the old one keeps serving.

If your memory must live in RAM inside the app process, with no server, see in-memory vector databases. At the other end, for very large memory banks kept on object storage, Hindsight Memlake vs Postgres, turbopuffer and LanceDB compares the object-storage engines.

What memory frameworks use

You often don’t pick the vector database directly; the memory framework does, or offers a list.

  • Mem0’s Python library supports 22 vector stores, from Qdrant, Chroma and pgvector to Pinecone, Redis, Elasticsearch, Turbopuffer and Amazon S3 Vectors. Its docs say that with no configuration, “Qdrant will be used as the vector database.”
  • Hindsight stores memories in PostgreSQL with pgvector (or Oracle AI Database 23ai), with an embedded Postgres (pg0) for local use, so teams on Postgres don’t add a new database.
  • Honcho runs on Postgres with pgvector. Its 3.4.1 patch changed how wildcards inside filters match stored conclusions; see Honcho memory.
  • Letta keeps archival passages in its own storage behind its API.
  • Graphiti and other graph-based systems store embeddings inside the graph database (Neo4j, FalkorDB or Neptune) instead of a separate vector store. See knowledge graphs for AI memory.

Example: per-user memory with Qdrant

This stores memories with a user ID and timestamp, then retrieves only that user’s memories, optionally within a time range. It uses qdrant-client 1.19 and OpenAI embeddings.

 1import time
 2import uuid
 3
 4from openai import OpenAI
 5from qdrant_client import QdrantClient, models
 6
 7oai = OpenAI()
 8qdrant = QdrantClient(":memory:")  # QdrantClient(url="http://localhost:6333") in production
 9
10qdrant.create_collection(
11    "memories",
12    vectors_config=models.VectorParams(size=1536, distance=models.Distance.COSINE),
13)
14qdrant.create_payload_index(  # co-locates each user's vectors (server mode)
15    "memories",
16    field_name="user_id",
17    field_schema=models.KeywordIndexParams(type=models.KeywordIndexType.KEYWORD, is_tenant=True),
18)
19
20
21def embed(text: str) -> list[float]:
22    return oai.embeddings.create(model="text-embedding-3-small", input=text).data[0].embedding
23
24
25def remember(user_id: str, text: str, kind: str = "fact") -> None:
26    qdrant.upsert("memories", points=[models.PointStruct(
27        id=str(uuid.uuid4()),
28        vector=embed(text),
29        payload={"user_id": user_id, "text": text, "kind": kind, "ts": int(time.time())},
30    )])
31
32
33def recall(user_id: str, query: str, since: int | None = None, k: int = 5) -> list[str]:
34    must = [models.FieldCondition(key="user_id", match=models.MatchValue(value=user_id))]
35    if since:
36        must.append(models.FieldCondition(key="ts", range=models.Range(gte=since)))
37    hits = qdrant.query_points(
38        "memories", query=embed(query), query_filter=models.Filter(must=must), limit=k
39    ).points
40    return [h.payload["text"] for h in hits]
41
42
43remember("u42", "Prefers aisle seats on long flights")
44print(recall("u42", "Book a flight to Tokyo"))

The user_id filter is the line that matters most. Leave it out and one user’s memories can show up in another user’s prompt. Payload indexes only take effect on a Qdrant server, not in local mode. Which embedding model to use is covered in embedding models for agent memory.

What a vector database won’t do for memory

A vector store answers “what’s similar to this?” Agent memory needs more:

  • Deciding what to save. Storing every message makes search noisy. Extracting facts needs an LLM step.
  • Handling change. “I live in NYC” and “I moved to SF” are both similar to “where do I live?” The database can’t tell which is current.
  • Time reasoning. Timestamps support filters, not questions like “what changed since my last order?”
  • Relationships. Multi-hop questions (“which of my teammates use the old API?”) need links between facts.

That’s the gap memory frameworks fill on top of the database, and why many add a graph or keyword index.

Hindsight is one example of all four layered on pgvector. Its retain step uses an LLM to extract facts, time, entities and relationships instead of storing raw messages. Related facts are consolidated into observations that get refined as things change. Recall runs vector search alongside BM25 keyword, entity-graph and time-range search, then fuses and reranks the results. On long histories that pays off: it reports 64.1 on BEAM at 10M tokens, where Mem0’s comparison page lists Mem0 at 48.6 (vendor-run). The cost is LLM calls on every write and more tokens per retrieval than a plain top-k vector query, so set its max_tokens budget. The difference between plain retrieval and memory is covered in RAG vs agent memory.

Embedding models for agent memory

The embedding model turns each memory and each query into a vector. Memory differs from document RAG in ways that affect that choice:

  • Units are short. “User is vegetarian” is five words. Pick a model that does well on short-text retrieval, and test with real memory items.
  • Queries and memories look different. The query is a new user message; the memory is a terse fact. Asymmetric query/document prompts help here.
  • Everything is re-embedded on a switch. A memory store grows for years. Changing models later means re-embedding all of it, and Google’s docs state that gemini-embedding-001 and gemini-embedding-2 spaces are “incompatible.”
  • Embeddings don’t know time. “I moved to SF” and “I live in NYC” are both close to “where do I live?” Handling changed facts is the job of the memory system, not the embedding model. Hindsight, for example, treats the embedding as one of four recall strategies: it runs semantic, BM25 keyword, entity-graph and time-range search in parallel, fuses them and reranks with a cross-encoder, so a weak embedding match on an ID or a date can still be found. The cost is an LLM call on every write to extract entities and timestamps.

Models compared

Specs from each provider’s docs or model card, checked 8 October 2026. Prices change often, so check the provider’s page.

ModelProviderOpen weightsMax inputDimensionsNotes
text-embedding-3-smallOpenAINo8,192 tokens1,536 (shortenable)~62,500 pages per dollar; MTEB 62.3% (OpenAI)
text-embedding-3-largeOpenAINo8,192 tokens3,072 (shortenable)~9,615 pages per dollar; MTEB 64.6% (OpenAI)
gemini-embedding-2GoogleNo8,192 tokens128-3,072 (768, 1,536, 3,072 recommended)Text, images, video, audio, PDF in one space
gemini-embedding-001GoogleNo2,048 tokens128-3,072Text only; task_type parameter
voyage-4-large / voyage-4 / voyage-4-liteVoyage AINo32,000 tokens1,024 default; 256, 512, 2,048int8 and binary output; the three are compatible
voyage-4-nanoVoyage AIYes32,000 tokens1,024 default; 256, 512, 2,048Open-weight member of the family
embed-v4.0CohereNo128k tokens256-1,536Text and images
Qwen3-Embedding (0.6B / 4B / 8B)Alibaba QwenYes, Apache-2.032K tokensUp to 1,024 / 2,560 / 4,096; 32 and up100+ languages; 8B scored 70.58 on MTEB multilingual (June 2025)
BGE-M3BAAIYes, MIT8,192 tokens1,024Dense, sparse and multi-vector output from one model
bge-small-en-v1.5BAAIYes, MIT512 tokens38433M parameters, English only; a common local default for memory stores
EmbeddingGemmaGoogleYes, Gemma license2,048 tokens768; 512, 256, 128300M parameters, built for on-device use
EmbeddingGemma 2GoogleYes, Apache-2.08,192 tokens768; 512, 256, 128Released 6 Oct 2026; 740M parameters (270M text-only); text, code, images, video, audio

Sources: OpenAI embeddings guide, Gemini embeddings docs, Voyage docs, Cohere Embed docs, and the Hugging Face model cards for Qwen3-Embedding-8B, BGE-M3 and EmbeddingGemma. Cohere’s docs also list newer embed-v5.0-pro and embed-v5.0-fast models with 128k context and up to 2,048 dimensions.

EmbeddingGemma 2 is the newest open option, released by Google on 6 October 2026 (announcement, model card). Google’s model card reports MTEB English v2 68.46, Multilingual v2 61.36 and Code 78.68 at 768 dimensions; the first EmbeddingGemma scored 61.15 multilingual and 68.76 on code. These are Google’s own numbers. Google says the text-only weights need about 191MB of active RAM on a Pixel 11 Pro with quantization, which makes it a fit for local or on-device memory where nothing should leave the machine. Because its context grew to 8K tokens and the license moved to Apache-2.0, it removes the two main limits of the first version.

Check your memory system’s default model

Self-hosted memory systems ship with a default embedding model, and the default can be small and English-only. Hindsight is an example. Its configuration docs set the embedding provider to local and the model to BAAI/bge-small-en-v1.5 by default. Its multilingual guide says that model and the default reranker (cross-encoder/ms-marco-MiniLM-L-6-v2) are English-only, and recommends BAAI/bge-m3 and BAAI/bge-reranker-v2-m3 for other languages:

SettingHindsight defaultFor non-English memories
HINDSIGHT_API_EMBEDDINGS_PROVIDERlocal (also openai, cohere, google, tei, litellm and others)any provider with a multilingual model
HINDSIGHT_API_EMBEDDINGS_LOCAL_MODELBAAI/bge-small-en-v1.5 (384 dimensions, 512 tokens)BAAI/bge-m3
HINDSIGHT_API_RERANKER_LOCAL_MODELcross-encoder/ms-marco-MiniLM-L-6-v2BAAI/bge-reranker-v2-m3

The small local default means embeddings need no API key. If your users write in other languages, change it before you store anything, because a later switch means re-embedding the whole bank. Hindsight applies the query and document prompts of models that ship them, such as Qwen3-Embedding, on its own. Whatever system you use, find this setting first: the best model in the table above only helps if the memory layer actually uses it.

How to choose

  1. Start with what you run. Postgres means pgvector; Redis, MongoDB or Elasticsearch all have vector search.
  2. Count tenants. Thousands of users need filter-friendly tenant partitioning, not a collection per user.
  3. Estimate size. Under a few million vectors, almost any option works. At hundreds of millions, look at Milvus, Qdrant, Weaviate or Pinecone.
  4. Check hybrid search. If users mention IDs, names or codes, you’ll want keyword matching too.
  5. Test deletes. Make sure you can remove one user’s data completely and quickly. Age-based expiry is not the same: AgentCore Memory’s record TTL, for example, can take a few days to remove an expired record, and it stays searchable until then.
  6. Pick the framework first if you use one. Check which backends it supports before choosing a database.