Short-Term Memory in AI Agents: Context, State, Limits

Short-term memory in AI agents is what the agent keeps for the task in front of it: recent messages, tool results, the plan and retrieved facts. It lasts for one conversation or run. LLM calls are stateless, so the agent’s code holds this state and rebuilds the prompt from it each step, within the context window.

It’s the one memory type every agent has, even agents with no memory features at all. Getting it right decides cost, latency and whether the agent loses track halfway through a long task. For the other memory types and how they fit together, see AI agent memory explained.

What is short-term memory in AI agents?

Short-term memory in an AI agent is the session-scoped state the agent keeps while working on one conversation or task, including message history, tool outputs, intermediate results and goals. The agent builds each prompt from it, and it’s bounded by the model’s context window.

Two references define it. LangGraph’s memory guide calls short-term memory “thread-scoped” memory that “tracks the ongoing conversation by maintaining message history within a session.” It’s stored as agent state and persisted by a checkpointer, so a thread can be resumed later.

The CoALA framework (Sumers et al., 2023) calls the same thing working memory: “active and readily available information” for the current decision cycle, including inputs, retrieved knowledge and active goals. CoALA stresses that working memory is “a data structure that persists across LLM calls,” not just the prompt. Each LLM input is built from a subset of it, and the output is parsed back into it.

What is short-term memory in AI, in general?

Outside agents, “short-term memory” in AI usually means the model’s ability to use information in its current input. For transformers that’s the context window and its KV cache. For older recurrent networks such as LSTMs (long short-term memory), it’s the hidden state carried from step to step. Agent short-term memory builds on the first: the app decides what goes into each window.

What short-term memory holds

A typical agent’s short-term state includes:

• Message history: user turns, assistant replies, tool calls and tool results.
• Task state: the current goal, a plan or to-do list, progress so far.
• Retrieved context: facts or episodes pulled from long-term memory for this turn.
• Scratch results: intermediate values, drafts, extracted fields.
• Artifacts: uploaded files, documents, generated outputs. LangGraph’s docs list “uploaded files, retrieved documents, or generated artifacts” as normal parts of thread state.

Not all of this needs to be in the prompt at once. Keeping state in a structure, and choosing what to render into each call, is the main design lever.

Why short-term memory runs out

Message history grows with every turn, and tool-heavy agents grow faster. Manus reported that a typical task takes “around 50 tool calls on average” (Manus, 2025). Three problems follow:

1. Hard limits. If the history exceeds the context window, the call fails.
2. Quality drops before the limit. LangGraph’s docs note that most LLMs “still perform poorly over long contexts,” getting distracted by stale or off-topic content. Lost in the Middle (Liu et al., 2023) found accuracy is often highest when relevant information sits at the start or end of the input, and drops when it’s in the middle.
3. Cost and latency. Every token is resent and billed on every call.

How big windows are today, and how they behave when full, is covered in the context window of an LLM guide.

Techniques to manage short-term memory

Manus’s advice on offloading is worth repeating: make compression restorable. Drop a web page’s content but keep its URL; drop a document but keep its path. Then the agent can re-read it if needed. The trade-offs of each method are covered in context window limitations and solutions and LLM memory compression.

Adding short-term memory with LangChain

In LangChain v1, short-term memory is a checkpointer on the agent plus a thread_id per conversation. Summarization is a middleware. Adapted from LangChain’s short-term memory docs:

 1from langchain.agents import create_agent
 2from langchain.agents.middleware import SummarizationMiddleware
 3from langgraph.checkpoint.memory import InMemorySaver
 4
 5agent = create_agent(
 6    model="anthropic:claude-sonnet-4-5",
 7    tools=[],
 8    middleware=[
 9        SummarizationMiddleware(
10            model="anthropic:claude-haiku-4-5",
11            trigger=("tokens", 4000),   # summarize once history passes 4,000 tokens
12            keep=("messages", 20),      # always keep the last 20 messages verbatim
13        )
14    ],
15    checkpointer=InMemorySaver(),  # use PostgresSaver in production
16)
17
18config = {"configurable": {"thread_id": "user-42-session-1"}}
19agent.invoke({"messages": "hi, my name is Bob"}, config)
20agent.invoke({"messages": "write a short poem about cats"}, config)
21reply = agent.invoke({"messages": "what's my name?"}, config)
22print(reply["messages"][-1].content)  # the agent still knows it's Bob

A new thread_id starts a clean short-term memory. The same thread_id resumes it, even after a restart if you use a database-backed checkpointer such as PostgresSaver. For a plain token cap without summaries, call trim_messages(..., strategy="last", max_tokens=..., start_on="human") before each model call instead.

Short-term memory in other designs

• MemGPT / Letta. MemGPT (Packer et al., 2023) treats the context window like an operating system’s main memory and pages information in and out of external storage. Letta keeps memory files under system/ in the prompt every turn and lets the agent read other files on demand.
• Task lists as working memory. Agents that keep an explicit to-do list in state, like Deep Agents’ write_todos tool or Manus’s todo.md, are using short-term memory to stay on track across many steps.
• Observation logs. Mastra’s Observational Memory replaces raw history with background-written observations as the thread grows.

When short-term memory isn’t enough

Short-term memory ends with the thread. If a user comes back tomorrow, nothing carries over unless you save it somewhere else. That’s the job of long-term memory: extract what matters from the session, store it per user, and retrieve it in the next one. Memory servers such as Mem0, Zep and Hindsight do that step for you. Hindsight’s recall also respects a token budget (max_tokens), so what it adds back into short-term memory stays bounded; it returns more context than Mem0 by default, so set that budget to match your window. The side-by-side comparison, including how information moves between the two, is in short-term and long-term memory in agentic AI.