How to Give a Local LLM Memory (Ollama, SQLite, Python)

11 min read

Local LLMs don't remember between chats. How to add persistent memory offline: chat apps, MCP, and tested Python with Ollama, SQLite, Mem0 and the Agno agent framework.

A local LLM has no memory of its own. Ollama, LM Studio and llama.cpp process each request and keep nothing. To give a local LLM memory, use a chat app with a memory feature (Open WebUI, AnythingLLM), connect a local MCP memory server, or build it: save facts to SQLite or a local vector store, embed them with a local model, and add the relevant ones to each prompt. For a lightweight open-source agent in Python with SQLite memory on Ollama, use Agno (Apache-2.0) or the short script below. Every tool here is free to run locally, with no usage limits beyond your hardware (licenses). To run the Claude Code coding agent on a local Ollama model, see Claude Code with Ollama.

Does a local LLM have memory?

Local LLM memory is information your app stores on your machine and puts back into the model’s prompt, since the model itself retains nothing between calls. There are two kinds: the current chat history, which the app resends each turn, and long-term memory, which survives a new chat or a restart.

You can see the statelessness in Ollama’s chat API: the messages field is the “chat history as an array of message objects,” and there’s no session ID. The client sends the whole conversation every time.

Even within one chat, the history is cut at the context length. Per Ollama’s context length docs, the default depends on VRAM: 4k tokens under 24 GiB, 32k for 24-48 GiB, and 256k at 48 GiB or more. On a typical laptop, that means a long chat silently drops its early turns. You can raise it with OLLAMA_CONTEXT_LENGTH=64000 ollama serve or the app’s slider, at the cost of more memory. The trade-offs of bigger windows on local models are in largest context window open-source LLMs, and context window limitations covers why accuracy drops as the window fills, and when plain retrieval or full context beats a memory system.

A bigger window still empties when the chat ends. Long-term memory needs storage outside the model.

Local AI apps with long-term memory

If you just want a local chat app that remembers you, these are the options we could confirm in each project’s docs (October 2026).

AppBuilt-in memoryHow it worksWhere it’s stored
Open WebUIYesYou add memories in Settings > Personalization > Memory; with native function calling, the model manages them with tools like add_memory and search_memoriesYour Open WebUI database, per user
AnythingLLMYes, off by defaultWorkspace memories (20 max) and global memories (5 max); optional automatic extraction from recent conversationsYour AnythingLLM instance
JanNot yetHomepage lists Memory as “Coming Soon”; assistants give fixed instructions onlyn/a; can connect MCP servers
LM StudioNo built-in feature foundSearch across chats (0.3.27); MCP host since 0.3.17; community memory plugins on the LM Studio HubDepends on the plugin or MCP server

Two caveats from the docs themselves. Open WebUI warns that memory quality “depends heavily on the model” and that “only very small models struggle with appropriate memory selection.” AnythingLLM’s automatic extraction needs a chat model that supports tool calling. A 1-3B model may store junk or nothing.

AnythingLLM’s memory, in detail. Memories are off by default; turn on Enable Personalization to start. Workspaces can hold up to 20 memories each, and global memories are capped at 5. Automatic extraction runs every 3 hours by default: an Observer proposes up to 3 candidate facts per run, and a Reflector checks them against existing memories. Memories go into the system prompt under a ## Things I Remember About You heading. The docs warn that “Memory content is sent to your LLM provider as part of the system prompt,” and say not to store passwords or API keys. If privacy is why you run locally, pair AnythingLLM with a local model. Source: AnythingLLM memories docs, read 2026-11-16.

MCP memory servers work with any app that can host MCP, including Jan and LM Studio. The reference Knowledge Graph Memory Server stores entities, relations and observations in a local JSONL file you can point anywhere with MEMORY_FILE_PATH. Since version 2.0 (October 2026) it needs Node.js 24 or later; what changed in 2.0 has the upgrade notes. More options are in AI memory MCP servers.

How to give a local LLM memory in your own app

  1. Run a chat model and an embedding model. For example ollama pull llama3.2 and ollama pull nomic-embed-text.
  2. Pick a local store. SQLite is enough for one user; Chroma, Qdrant (local mode) or Postgres with pgvector scale further.
  3. Save after each turn. Store what the user said, or extract short facts with the local model.
  4. Embed and search before each reply. Find the few memories closest to the new message.
  5. Inject them into the system prompt. Keep the block small; small models get confused by long prompts.
  6. Keep recent turns too. Send the last few messages for the flow of the current chat.
  7. Add deletion. A local store is still personal data.

Example 1: local memory from scratch with Ollama and SQLite

No memory library, no cloud. This stores every user message with a local embedding, then pulls the three closest ones into the system prompt. Tested with the ollama Python package 0.6 against Ollama 0.34:

 1import json, math, sqlite3
 2import ollama
 3
 4CHAT_MODEL = "llama3.2"
 5EMBED_MODEL = "nomic-embed-text"
 6
 7db = sqlite3.connect("memory.db")
 8db.execute("CREATE TABLE IF NOT EXISTS memories (user_id TEXT, text TEXT, vec TEXT)")
 9
10def embed(text: str) -> list[float]:
11    return ollama.embed(model=EMBED_MODEL, input=text).embeddings[0]
12
13def cosine(a: list[float], b: list[float]) -> float:
14    dot = sum(x * y for x, y in zip(a, b))
15    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))
16
17def remember(user_id: str, text: str) -> None:
18    db.execute("INSERT INTO memories VALUES (?, ?, ?)", (user_id, text, json.dumps(embed(text))))
19    db.commit()
20
21def recall(user_id: str, query: str, k: int = 3) -> list[str]:
22    q = embed(query)
23    rows = db.execute("SELECT text, vec FROM memories WHERE user_id = ?", (user_id,))
24    scored = sorted(((cosine(q, json.loads(v)), t) for t, v in rows), reverse=True)
25    return [t for _, t in scored[:k]]
26
27def chat(user_id: str, message: str, history: list[dict]) -> str:
28    memories = "\n".join(f"- {m}" for m in recall(user_id, message))
29    system = {"role": "system", "content": f"Things you remember about this user:\n{memories or '- nothing yet'}"}
30    reply = ollama.chat(model=CHAT_MODEL, messages=[system, *history[-10:], {"role": "user", "content": message}])
31    text = reply.message.content
32    history += [{"role": "user", "content": message}, {"role": "assistant", "content": text}]
33    remember(user_id, message)  # store what the user said for later sessions
34    return text
35
36print(chat("ana", "I'm allergic to peanuts and I live in Lisbon.", []))
37# New session: empty history, but the fact comes back from SQLite.
38print(chat("ana", "Suggest a snack for my flight home.", []))

In our run, even a 1.5B model (qwen2.5:1.5b-instruct) answered the second, fresh chat with peanut-free snack ideas. The file memory.db is the whole memory; copy it, inspect it with sqlite3, or delete a user’s rows.

One gotcha: embeddings need an embedding model. Asking a chat model for embeddings on current Ollama returned “This server does not support embeddings” in our test, so pull nomic-embed-text or another embedding model. For other small local options, such as EmbeddingGemma 2 (Apache-2.0, released October 2026), see embedding models for RAG and agent memory.

This stores raw messages, so it grows forever and never updates a changed fact. That’s fine to start, and keeping raw messages is a good habit: on short histories, plain search over raw chat beat most fact-extracting memory systems in a 2026 study (benchmarks by architecture). For cleaner memories and facts that update, use a library that extracts facts; how each system handles a changed fact compares their update rules.

Example 2: Mem0 fully local with Ollama

Mem0 can use Ollama for both fact extraction and embeddings, with Qdrant running embedded on disk. Nothing leaves your machine. This config follows Mem0’s own self-hosted companion cookbook, switched to a local Qdrant path. Tested with mem0ai 2.2.1:

 1import os
 2os.environ["MEM0_TELEMETRY"] = "False"  # mem0 sends anonymous usage data unless disabled
 3from mem0 import Memory
 4
 5config = {
 6    "llm": {"provider": "ollama", "config": {"model": "llama3.1:8b", "temperature": 0,
 7                                             "ollama_base_url": "http://localhost:11434"}},
 8    "embedder": {"provider": "ollama", "config": {"model": "nomic-embed-text",
 9                                                  "ollama_base_url": "http://localhost:11434"}},
10    "vector_store": {"provider": "qdrant", "config": {"collection_name": "memories", "path": "./qdrant_data",
11                                                      "on_disk": True, "embedding_model_dims": 768}},
12    "history_db_path": "./mem0_history.db",
13}
14memory = Memory.from_config(config)
15
16memory.add([{"role": "user", "content": "I'm vegetarian and I run Debian on my laptop."}], user_id="ana")
17hits = memory.search("What should I cook for dinner?", filters={"user_id": "ana"}, top_k=3)
18for h in hits["results"]:
19    print(round(h["score"], 2), h["memory"])

Our run (with qwen2.5:1.5b-instruct as the LLM) printed 0.43 User is vegetarian and runs Debian on their laptop: Mem0 had the local model rewrite the message as a fact before storing it. Notes from testing:

  • embedding_model_dims must match the embedder (768 for nomic-embed-text).
  • Without on_disk: True, local Qdrant keeps vectors in memory only.
  • The base install logs that BM25 keyword search is disabled; pip install "mem0ai[extras]" adds it, and mem0ai[nlp] adds spaCy.

Lightweight open-source agents with SQLite memory

If you want an agent framework rather than a memory library, two open-source options keep memory in a local SQLite file and run on Ollama. (OpenClaw does too for its search index: its memory is Markdown notes, embedded with Ollama when you set memory.search.provider to ollama; see OpenClaw memory.)

  • Agno (Python): pass db=SqliteDb(...) and update_memory_on_run=True, and the agent saves sessions plus extracted user memories in one SQLite file.
  • Hermes Agent (a full agent, not a library): stores every session in ~/.hermes/state.db (SQLite with full-text search) next to small MEMORY.md and USER.md files. Its provider docs point it at Ollama with provider: custom and base_url: http://localhost:11434/v1, and say it needs at least 64,000 tokens of context, so raise Ollama’s default.

The Agno version of the peanut test, tested with agno 3.1.8 and ollama 0.6.3:

 1# pip install agno ollama openai "sqlalchemy[asyncio]"
 2from agno.agent import Agent
 3from agno.db.sqlite import SqliteDb
 4from agno.models.ollama import Ollama
 5
 6def make_agent() -> Agent:
 7    return Agent(model=Ollama(id="gemma4:e2b"), db=SqliteDb(db_file="agent.db"), update_memory_on_run=True)
 8
 9make_agent().run("I'm allergic to peanuts and I live in Lisbon.", user_id="ana")
10# A new agent object, same file: the memories come back from SQLite.
11print(make_agent().run("Suggest a snack for my flight home.", user_id="ana").content)

In our run, gemma4:e2b stored two facts (“User is allergic to peanuts.”, “User lives in Lisbon.”) and the second agent answered with nut-free snacks. Two things we hit:

  • Extra packages. agno.db.sqlite fails to import without sqlalchemy[asyncio] (it needs greenlet), and agno.models.ollama imports the openai package.
  • Model size matters. With qwen2.5:1.5b-instruct and phi4-mini the run worked but no memory was saved: extraction is a tool call, and those models didn’t make it. Check agent.get_user_memories(user_id=...) before trusting a small model.

Other memory systems that run locally

Several agent memory servers accept a local model as their LLM, so you can self-host the whole stack:

  • Hindsight lists ollama, lmstudio and llamacpp as fully local LLM providers. It runs under Docker, pip or embedded in Python, and can use an embedded Postgres (pg0), so you don’t need a database server. Clients are Python, TypeScript and Go, and its MCP server works with MCP hosts like LM Studio and Jan. It’s MIT-licensed with every feature in the open-source release: graph and time-aware search, consolidation of facts into observations, and reflect. The local catch: each write and each consolidation pass calls your LLM, so a small model on a laptop does a lot more work than with a plain SQLite store. It also returns more text per lookup than Mem0 by default (about 4x, by Mem0’s own measurements), which matters with Ollama’s 4k default window: set max_tokens on recall lower.
  • Mem0’s self-hosted server wraps the library above in a FastAPI service with Postgres and pgvector.
  • Letta (see Letta vs LangChain memory) and Cognee can also be self-hosted; check each project’s docs for current local-model support before you commit.

Licenses and cost of local memory tools

Run locally, none of these tools has a free tier or a usage cap: you pay only for your hardware. Licenses as GitHub reports them (October 2026):

ToolWhat it isLicenseGitHub stars
OllamaRuns the modelMIT~184K
SQLiteStores the memory filePublic domainn/a
AgnoPython agent framework with SQLite memoryApache-2.0~43K
Hermes AgentFull agent, SQLite session storeMIT~256K
Mem0Memory libraryApache-2.0~68K
HindsightMemory serverMIT, full server in the open-source release~53K
AnythingLLMChat app with memoriesMIT~67K
Open WebUIChat app with memoryOpen WebUI License (BSD-3 style plus a rule: keep the “Open WebUI” branding once a deployment has more than 50 users in 30 days)~156K

Mem0 and Hindsight also sell hosted cloud versions with their own pricing; the self-hosted versions above don’t need them.

How they differ is covered in open-source memory systems compared, and the general build (with hosted models) is in how to give AI agents memory.

Choosing a model for local memory

Memory puts extra demands on a small model:

  • Extraction needs instruction-following. Turning chat into clean facts, or deciding when to call add_memory, is where tiny models fail first. Both Open WebUI and AnythingLLM say so in their docs.
  • The prompt gets longer. Memories plus recent history plus the system prompt can exceed a 4k default window. Raise the context length if your hardware allows.
  • Split the jobs. A small model for chat and a slightly larger one for background extraction is a sensible setup, since extraction isn’t latency-sensitive.

For the concepts behind all of this, see AI agent memory explained.