Episodic Memory in AI Agents: How Agents Recall Events

Episodic memory in AI agents is a store of specific past events, each tied to when it happened and in what context. It lets an agent answer “what happened last time?” and reuse what worked. LLMs don’t have it built in. The agent’s code saves episodes outside the model and retrieves the relevant ones into the prompt.

The term comes from psychology. Endel Tulving coined “episodic memory” in 1972 to separate remembering an event from knowing a fact (Wikipedia). Agent designers kept the split. This page covers what goes into an episode, how agents retrieve and learn from episodes, and a working FAISS example. For how episodic memory fits next to the other types, see the AI agent memory explained guide.

What is episodic memory in AI agents?

Episodic memory in an AI agent is a collection of records of past experiences, such as conversations, task attempts and tool results, each stored with its time, its context and often its outcome. The agent retrieves relevant episodes into its working context to guide what it does next.

The CoALA framework (Sumers et al., 2023) defines it as memory that “stores experience from earlier decision cycles.” Its examples include input-output pairs, event logs and game trajectories from earlier episodes. During planning, the agent pulls episodes back into working memory to reason over them. After acting, it can write new experiences back as a form of learning.

LangGraph’s memory docs give the practical angle: for agents, episodic memory “is often used to help an agent remember how to accomplish a task,” and it’s often implemented as few-shot examples picked from past runs.

What is episodic memory in an LLM?

A plain LLM has no episodic memory. Its weights store general patterns from training, not a record of your conversation from last week. Each API call starts fresh. What looks like recall within a chat is just the app resending earlier messages.

The 2025 position paper Episodic Memory is the Missing Piece for Long-Term LLM Agents (Pink et al.) sets out five properties a real episodic memory needs, and checks which current techniques have them:

The paper’s conclusion is that no single technique covers all five. Long context forgets when the window ends. Retrieval over chunks loses the binding between an event and its context. Fine-tuning can’t learn from one exposure. That’s why practical episodic memory is a design problem, not a library call.

What an episode record should contain

A useful episode stores more than raw text. The fields that matter most:

• When: a timestamp, so the agent can order events and answer time questions.
• Who and where: user ID, session, task or environment.
• What happened: the observation, the action taken, the result.
• Why it mattered: an outcome or score (success, failure, user feedback).
• Importance: a rating used at retrieval time.
• A link to the source: the original messages, so a bad summary can be checked.

LangMem’s episodic memory guide uses an Episode schema with four fields: observation (the context), thoughts (the reasoning that led to the action), action (what was done) and result (the outcome and what to do better next time). An LLM fills the schema after a successful interaction. The point is to record how a success happened, not just that it happened.

How agents retrieve episodes

Similarity alone is a weak signal for episodes. A memory from five minutes ago usually matters more than an equally similar one from last year. Generative Agents (Park et al., 2023), the best-known episodic design, scores every memory on three signals:

• Recency: exponential decay over game hours since the memory was last retrieved, with a decay factor of 0.995.
• Importance: the LLM rates each memory from 1 (mundane) to 10 (poignant) when it’s created. Cleaning a room scores about 2; asking a crush on a date scores about 8.
• Relevance: cosine similarity between the memory’s embedding and the query’s embedding.

Each score is min-max normalized to [0, 1], and the three are summed with equal weights. The top memories that fit in the context window go into the prompt. Generative Agents also triggers reflection when the summed importance of recent events passes 150, which in practice happened two or three times per simulated day. Reflections are higher-level conclusions written back into the same memory stream.

Current memory servers add an explicit time signal. Hindsight (MIT) stores a timestamp with each extracted fact and runs a time-range search next to semantic, keyword and entity-graph search, then fuses and reranks the results. That’s what lets “what did we discuss two weeks ago?” hit the right episodes instead of the most similar ones. The price is an LLM call on every write to pull out the time and entities.

How agents learn from episodes

Storing episodes only pays off if they change future behavior. Three patterns show up in research and frameworks.

Few-shot examples from past runs

The agent retrieves similar past tasks and includes them as examples in the prompt. LangGraph notes that “sometimes it’s easier to ‘show’ than ’tell’,” and that the hard part is choosing the most relevant examples for each input. This works well for tool calling, formatting and classification tasks.

Reflection on failures

Reflexion (Shinn et al., 2023) has the agent write a verbal reflection after a failed attempt and keep it in an “episodic memory buffer” for the next try. With this loop it reported 91% pass@1 on HumanEval, against 80% for GPT-4 at the time. CoALA classifies those reflections as semantic knowledge, since they’re lessons rather than event records. The line between the two types is blurry by design.

Consolidation into facts

Over time, many episodes should collapse into a few stable facts or beliefs. Generative Agents’ reflections do this, and so do background jobs in current memory systems. Hindsight, for example, keeps experiences (the agent’s own episodes) separate from world facts, then consolidates both into observations. More on this in memory consolidation in AI agents.

Building episodic memory with FAISS

FAISS is a vector index, not a memory system, so you keep metadata yourself. This sketch stores episodes, finds candidates by similarity, then reranks with recency and importance in the style of Generative Agents:

 1import time
 2import faiss
 3import numpy as np
 4from openai import OpenAI
 5
 6client = OpenAI()
 7DIM = 1536  # text-embedding-3-small
 8
 9def embed(texts: list[str]) -> np.ndarray:
10    resp = client.embeddings.create(model="text-embedding-3-small", input=texts)
11    vecs = np.array([d.embedding for d in resp.data], dtype="float32")
12    faiss.normalize_L2(vecs)  # unit vectors: inner product = cosine
13    return vecs
14
15class EpisodicMemory:
16    def __init__(self):
17        self.index = faiss.IndexFlatIP(DIM)
18        self.episodes: list[dict] = []  # same order as vectors in the index
19
20    def add(self, text: str, importance: int, outcome: str = "") -> None:
21        self.index.add(embed([text]))
22        self.episodes.append(
23            {"text": text, "importance": importance, "outcome": outcome, "ts": time.time()}
24        )
25
26    def recall(self, query: str, k: int = 3, candidates: int = 20, decay: float = 0.995):
27        if not self.episodes:
28            return []
29        n = min(candidates, len(self.episodes))
30        sims, ids = self.index.search(embed([query]), n)
31        now = time.time()
32        scored = []
33        for sim, i in zip(sims[0], ids[0]):
34            ep = self.episodes[i]
35            recency = decay ** ((now - ep["ts"]) / 3600)  # decay per hour
36            score = float(sim) + recency + ep["importance"] / 10
37            scored.append((score, ep))
38        scored.sort(key=lambda x: x[0], reverse=True)
39        return [ep for _, ep in scored[:k]]
40
41memory = EpisodicMemory()
42memory.add("Deploy failed: DB migration timed out after 300s on orders table.", 8, "failure")
43memory.add("User asked for a CSV export of March invoices; sent via email.", 3, "success")
44for ep in memory.recall("Why did the last deploy break?"):
45    print(ep["text"], ep["outcome"])

Two simplifications to know about. Generative Agents decays from the time a memory was last accessed, not created, and it normalizes all three scores before summing. IndexFlatIP does exact search, scanning every vector per query; for large stores FAISS also offers approximate indexes such as HNSW.

Episodic vs semantic memory

Semantic memory is often derived from episodic memory: many episodes about a user ordering vegetarian meals become one fact. The semantic memory in AI agents page covers the fact side.

Common problems with episodic memory

• Unbounded growth. Every turn adds episodes. Without importance filtering, consolidation or expiry, retrieval gets noisy and slow.
• Lost context. Chunking a conversation into fragments drops who said what and when. Store episodes at the level of a session or a task step, with metadata.
• Weak time handling. Questions like “what did we discuss two weeks ago?” need timestamps and a time-range filter, not just embeddings. See temporal reasoning in AI memory.
• Recording failures as successes. If outcomes aren’t stored, the agent may copy a past approach that didn’t work.
• Privacy. Episodes are raw personal data. Scope them per user and support deletion.