LLM Memory Compression: Compaction, Summaries, Tokens

9 min read

LLM memory compression explained: conversation compaction, fact extraction, prompt compression (LLMLingua), KV cache eviction and vector quantization.

LLM memory compression means fitting what a model needs to remember into fewer tokens or fewer bytes. In practice it’s five different techniques: summarizing old conversation turns (compaction), extracting facts instead of keeping transcripts, pruning tokens from prompts, evicting entries from the model’s KV cache, and quantizing stored embeddings. Each one trades some detail for cost, speed or room in the context window.

This page explains each technique, what the papers and API docs report, and when to use which.

What is LLM memory compression?

LLM memory compression is any method that reduces the size of the information an LLM application keeps or sends to the model, while preserving what later steps need. It works at three levels: the text in the prompt, the model’s internal attention cache during inference, and the vectors in an external memory store.

The most common reason is the context window. Every model has a token limit, and quality tends to drop well before it. Chroma’s Context Rot report (July 2025) tested 18 models and found that “model performance varies significantly as input length changes, even on simple tasks.” Cost is the other reason: input tokens are billed on every call, so a 100,000-token history costs the same to resend each turn. For background on limits, see the LLM context window guide.

The five kinds of LLM memory compression

TechniqueWhat gets smallerTypical savingsMain lossExamples
Compaction (summarize old turns)Conversation history in the promptDepends on summary lengthDetails not in the summaryClaude and OpenAI compaction, MemGPT
Fact extractionLong-term memory store and retrieved contextMem0: >90% token cost vs full context (vendor-run)Anything the extractor skipsMem0, LangMem, Hindsight
Prompt compressionRetrieved documents, long instructionsLLMLingua: up to 20xGrammar, sometimes key wordsLLMLingua, LLMLingua-2
KV cache evictionGPU memory during generationH2O: up to 29x throughputAttention to evicted tokensStreamingLLM, H2O
Vector quantizationEmbeddings on disk and in RAM2x (half precision) to 32x (binary)Some retrieval accuracypgvector halfvec, int8 and binary embeddings

Compaction: summarizing the conversation

Compaction replaces older turns with a summary and keeps going. It’s the oldest and most widely used form of memory compression.

The MemGPT paper (Packer et al., 2023) describes a precise version. The prompt holds a FIFO queue of messages whose first entry is “a recursive summary of messages that have been evicted from the queue.” When the prompt passes a warning threshold (70% of the window in the paper’s example), the agent gets a “memory pressure” warning so it can save important facts. At 100%, the system evicts messages (about 50% of the window), writes a new summary from the old summary plus the evicted messages, and keeps the originals in searchable recall storage.

A 2023 paper by Wang et al., Recursively Summarizing Enables Long-Term Dialogue Memory, tests the same idea on its own. The LLM first summarizes small dialogue contexts, then “recursively produces new memory using previous old memory and subsequent contexts.” The authors report more consistent responses in long conversations, and that the method complements both larger windows and retrieval.

Both major model APIs now do this server-side:

  • Claude API. Anthropic’s compaction docs say compaction “replaces the older turns of a conversation with a summary that Claude writes on the server.” It comes in two beta forms: on demand (you request the summary) and at a token threshold (the API compacts when input reaches your trigger). A related feature, context editing, clears old tool results instead of summarizing them; its clear_tool_uses_20250919 strategy defaults to triggering at 100,000 input tokens and keeping the last 3 tool uses.
  • OpenAI Responses API. The compaction guide enables it with context_management=[{"type": "compaction", "compact_threshold": 200000}]. The response includes an encrypted compaction item that is “opaque and not intended to be human-interpretable.” A standalone /responses/compact endpoint does the same on request.

A simple compaction loop

If you manage history yourself, the MemGPT pattern takes a few lines. This version keeps the last few turns word for word and folds older ones into a running summary.

 1from openai import OpenAI
 2
 3client = OpenAI()
 4WINDOW = 128_000          # model's context window, in tokens
 5FLUSH_AT = 0.7 * WINDOW   # MemGPT's example warning threshold
 6KEEP_RECENT = 6           # turns always kept word for word
 7
 8
 9def tokens(messages: list[dict]) -> int:
10    return sum(len(m["content"]) for m in messages) // 4  # rough estimate; use a tokenizer in production
11
12
13def compact(summary: str, history: list[dict]) -> tuple[str, list[dict]]:
14    if tokens(history) < FLUSH_AT or len(history) <= KEEP_RECENT:
15        return summary, history
16    old, recent = history[:-KEEP_RECENT], history[-KEEP_RECENT:]
17    transcript = "\n".join(f"{m['role']}: {m['content']}" for m in old)
18    summary = client.responses.create(
19        model="gpt-5-mini",
20        input=(
21            "Update the running summary of this conversation. Keep names, numbers, dates, "
22            "decisions and open tasks. Drop small talk.\n\n"
23            f"Current summary:\n{summary or '(none)'}\n\nNew messages:\n{transcript}"
24        ),
25    ).output_text
26    return summary, recent  # store `old` in a database too, so it stays searchable

The prompt matters more than the code. Tell the summarizer what must survive (identifiers, numbers, decisions, unresolved questions), or it will keep the narrative and drop the facts.

Fact extraction: storing meaning, not transcripts

Memory systems compress differently. Instead of summarizing a conversation as prose, they extract discrete facts and store those, then retrieve only the few that match the next query.

The Mem0 paper (Chhikara et al., 2025) measured the effect on the LoCoMo benchmark: compared with sending the full conversation, Mem0 “attains a 91% lower p95 latency and saves more than 90% token cost.” These are the vendor’s own numbers on one benchmark, but the direction holds generally: a handful of retrieved facts is far smaller than a transcript.

The cost is a second kind of loss. A summary drops details; an extractor drops whole topics it didn’t think were worth saving. Most systems keep the raw messages too, so they can be re-processed. This overlaps with memory consolidation, which merges and cleans extracted facts over time.

Hindsight shows how these layers stack. It extracts facts on write, then consolidates related facts into observations, evidence-backed beliefs that get refined rather than duplicated. Mental models go one step further: standing answers kept current in the background, so reading one is a database lookup with no retrieval or LLM call. Recall output is trimmed to a token budget you set with max_tokens. That budget matters: by Mem0’s measurement Hindsight returns 24K-44K tokens per retrieval by default against Mem0’s under 7K, so set it lower when prompt size is the constraint.

Prompt compression: pruning tokens

Prompt compression removes tokens that carry little information, leaving text that looks broken to a human but still works for the model.

Microsoft’s LLMLingua (Jiang et al., 2023) uses a small language model to decide which tokens to drop, with a budget controller and iterative token-level compression. It reports “up to 20x compression with little performance loss” across GSM8K, BBH, ShareGPT and arXiv data.

LLMLingua-2 (Pan et al., Findings of ACL 2024) reframes the task as token classification with a bidirectional encoder (XLM-RoBERTa-large or mBERT). It’s 3x-6x faster than earlier prompt compression methods and reduces end-to-end latency by 1.6x-2.9x at compression ratios of 2x-5x. Usage, from the LLMLingua README:

1from llmlingua import PromptCompressor
2
3compressor = PromptCompressor(
4    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
5    use_llmlingua2=True,
6)
7long_context = open("retrieved_docs.txt").read()
8result = compressor.compress_prompt(long_context, rate=0.33, force_tokens=["\n", "?"])
9print(result["compressed_prompt"])

rate=0.33 keeps about a third of the tokens. force_tokens lists tokens that must never be dropped. Prompt compression works best on retrieved documents and long reference text. It’s a poor fit for exact content like code, IDs or legal wording.

For coding agents, the same idea now ships as a local proxy. Headroom (78K GitHub stars, October 2026) compresses tool outputs, logs and JSON per content type, leaves source code and search results alone by default, and keeps each original so the model can fetch it back. Its README reports 21-57% fewer tokens on four synthetic scenarios.

KV cache compression: memory inside the model

During generation, a transformer stores key and value vectors for every token it has seen, the KV cache. It grows with context length and is often what limits batch size on a GPU. Compressing it is a different problem from the ones above: it saves hardware memory, not prompt tokens.

Two well-known approaches evict cache entries:

  • StreamingLLM (Xiao et al., ICLR 2024) found that keeping the KV states of the first few tokens, which act as “attention sinks,” plus a window of recent tokens lets models handle streams of 4 million tokens or more without fine-tuning. It reports up to a 22.2x speedup over sliding-window recomputation.
  • H2O (Zhang et al., 2023) observed that “a small portion of tokens contributes most of the value when computing attention scores.” Keeping recent tokens plus these “heavy hitters” (20% of the cache) improved throughput by up to 29x over some inference systems on OPT models.

Both make long streams cheap, but evicted tokens are gone for the model. Neither gives the model long-term recall of what it dropped; that still needs an external store.

Vector compression: shrinking the memory store

Agent memory stored as embeddings can get large: one million 1,536-dimension float32 vectors is about 6 GB before index overhead. Three methods cut that:

  • Half precision. pgvector’s halfvec stores 16-bit floats and supports HNSW indexes up to 4,000 dimensions, versus 2,000 for vector (pgvector README).
  • Integer and binary output. Some embedding APIs return compressed vectors directly. Voyage’s voyage-4 family supports int8, uint8, binary and ubinary output types.
  • Shorter vectors. Models trained with Matryoshka-style objectives can be truncated. OpenAI’s docs say a text-embedding-3-large vector “can be shortened to a size of 256” and still beat an unshortened ada-002 vector at 1,536 dimensions on MTEB.

The trade-offs are covered in embedding models for RAG.

How to choose a compression method

  1. Long single session? Use compaction. Keep the last few turns verbatim and store the full history somewhere searchable.
  2. Many sessions with the same user? Extract facts into a memory store and retrieve a few per turn instead of resending history.
  3. Large retrieved documents? Try prompt compression, measured on your own tasks.
  4. GPU memory limits on self-hosted models? Look at KV cache eviction or quantization in your inference server.
  5. Big vector store? Use half precision or int8 first, then binary with reranking if you need more.
  6. Always keep the raw data. Every method here is lossy. Compression should change what the model sees, not what you keep.
  7. Measure. Compare answers with and without compression on 20-50 real questions before you ship.

Compression is one way to stretch a fixed window. Another is to split memory into tiers, covered in MemoryOS explained, and a broader list of fixes is in context window limitations and solutions.