Largest Context Window Open-Source LLMs (Local, 2026)

10 min read

Open-weight LLMs with the largest context windows as of November 2026 (Llama 4 Scout, Kimi K3.1 Pro, Mistral Large 4), Ollama and llama.cpp settings, KV cache VRAM math.

The largest context window open-source LLM as of November 2026 is Llama 4 Scout, with 10M tokens. A wider group supports about 1M: Llama 4 Maverick, Kimi K3 and the new Kimi K3.1 Pro, DeepSeek V4, GLM-5.3, MiniMax-M3, Mistral Large 4 and Meta’s Muse Spark Open. Qwen3.6 and Qwen3.8 run 262,144 tokens natively. Running any of them at full length locally is limited by GPU memory, not the model card.

This page lists the open-weight models with the most context, explains why long context eats VRAM, and shows the exact settings for Ollama and llama.cpp. For hosted models, see the LLM context window comparison.

What is the largest context window open-source LLM?

Llama 4 Scout is the open-weight model with the largest context window, at 10M tokens, according to Meta’s model card. It’s a mixture-of-experts model with 109B total and 17B active parameters, released under the Llama 4 Community License. Its sibling Llama 4 Maverick (400B total, 17B active) supports 1M tokens.

A note on terms: most of these are open-weight models. You can download and run the weights, but the licenses differ. Qwen’s 27B models, Gemma 4, Mistral Small 4 and gpt-oss use Apache 2.0; DeepSeek uses MIT; Llama, Kimi, GLM and MiniMax use their own licenses with extra conditions. Read the license before you ship.

Open-weight LLM context window table

ModelParameters (total / active)Context windowLicenseIn Ollama library
Llama 4 Scout109B / 17B10MLlama 4 Communityllama4
Llama 4 Maverick400B / 17B1MLlama 4 Communityllama4
Kimi K3.1 Pro (2 Nov 2026)2.8T / 104B1,048,576 (card and config)Kimi K3.1 Pro Licensenot checked
Kimi K32.8T / 104B1MKimi K3 Licensekimi-k3 (cloud)
Mistral Large 4 (30 Sep 2026)~1T / ~52B1M (1,048,576 in config)Mistral Ecosystem Open-Weight Licensenot checked
Muse Spark Open (Meta, 15 Oct 2026)816B / 42B1MMuse Model Licensenot checked
DeepSeek-V4.1-Flash552B backboneup to 1MMITnot checked
DeepSeek-V4-Pro-0813not stated in card1,048,576 (config)MITnot checked
GLM-5.3not checked1,048,576 (config)GLM-5.3 licenseglm-5.3
MiniMax-M3~428B / ~23B1MMiniMax communitynot checked
Qwen3.8-27B27B dense262,144 native, 1M with YaRNApache 2.0qwen3.8
Qwen3.6-35B-A3B35B / 3B262,144 native, 1,010,000 with YaRNApache 2.0qwen3.6
Qwen3.6-27B27B dense262,144 native, 1,010,000 with YaRNApache 2.0qwen3.6
Mistral Small 4119B / 6.5B256KApache 2.0not checked
Mistral Medium 3.5128B dense256KModified MITnot checked
Mistral Large 3675B / 41B256KApache 2.0not checked
Gemma 4 12B, 31B, 26B MoE12B / 31B / 25.2B256KApache 2.0gemma4
Gemma 4 E2B, E4Bsmall128KApache 2.0gemma4
gpt-oss-120b, gpt-oss-20b117B / 5.1B; 21B / 3.6B131,072Apache 2.0gpt-oss

As of 8 November 2026. Sources: each model’s official Hugging Face card and config (for example Qwen3.6-27B, DeepSeek-V4.1-Flash, Kimi K3.1 Pro, Mistral Large 4), Meta’s Muse Spark Open post and the Ollama library.

Three 1M-token models joined the list since October. Kimi K3.1 Pro (Moonshot, weights published 2 November 2026) keeps K3’s 2.8T total and 104B active parameters and its 1,048,576-token window, and its card says it swaps the full-attention layers for Gated DeepSeek Sparse Attention “for more efficient long-context modeling”. Mistral Large 4 (1T total, about 52B active, 128 experts with 4 active per token) ships under the Mistral Ecosystem Open-Weight License, a modified MIT that the card says allows commercial use but requires a separate licence to run a model-as-a-service business. Muse Spark Open (816B total, 42B active) is released by Meta under the Muse Model License “with limited commercial terms”. The license column matters: three of the four newest 1M models are not under a standard open-source licence.

The models that fit on one consumer GPU are the small and mid-size ones: Gemma 4, Qwen3.6, Qwen3.8-27B, gpt-oss-20b. The 1M-token giants (Kimi K3 and K3.1 Pro, DeepSeek V4, GLM-5.3, MiniMax-M3, Mistral Large 4, Muse Spark Open) need multi-GPU servers, and in Ollama some are offered as cloud models rather than local downloads.

The real cost of long context: KV cache VRAM

The model weights are a fixed cost. The KV cache is the cost that grows with context. For every token in the window, each attention layer stores a key vector and a value vector so it doesn’t have to recompute them.

The formula for a standard attention model is:

KV cache bytes = 2 × layers × KV heads × head size × bytes per value × tokens

A worked example with Qwen3-8B

Qwen3-8B’s config lists 36 layers, 8 KV heads and a head size of 128, stored at 16 bits (2 bytes):

1def kv_cache_gib(layers, kv_heads, head_dim, tokens, bytes_per_value=2):
2    # 2 = one key and one value vector per layer, per KV head, per token
3    return 2 * layers * kv_heads * head_dim * bytes_per_value * tokens / 1024**3
4
5for ctx in (4_096, 32_768, 131_072):
6    print(ctx, round(kv_cache_gib(36, 8, 128, ctx), 2), "GiB")
7# 4096 0.56 GiB
8# 32768 4.5 GiB
9# 131072 18.0 GiB

That’s 144 KiB per token. At 131,072 tokens (Qwen3-8B’s maximum with YaRN, per its card), the cache alone needs 18 GiB. The weights add about 16 GB more at 16-bit precision (8B parameters × 2 bytes), less when quantized.

At 1M tokens it gets much worse. Qwen’s own card for Qwen2.5-7B-Instruct-1M says it needs “at least 120GB VRAM (total across GPUs)” for 1M-token sequences, and the 14B version needs at least 320 GB (Qwen2.5-1M card).

How newer models cut the cost

Model makers attack this with attention designs that store less per token:

  • Sliding window layers. Gemma 4 interleaves local attention over a 512- or 1,024-token window with full global layers, per its model card. Local layers only cache the window, not the whole context.
  • Sparse attention. MiniMax-M3 uses MiniMax Sparse Attention; its card reports 9× faster prefill and 15× faster decode than MiniMax-M2 at 1M context. DeepSeek-V4.1-Flash uses a compressed sparse attention scheme.
  • Grouped KV heads. Most current models share KV heads across query heads (Qwen3-8B has 32 query heads but 8 KV heads), which cuts the cache 4×.

Because of these designs, the simple formula overstates the cache for hybrid models. Use it for standard attention models and as an upper bound for the rest.

How to run a long context window in Ollama

Ollama doesn’t load a model’s full context by default. It picks the default from your VRAM: 4K tokens under 24 GiB, 32K from 24 to 48 GiB, and 256K at 48 GiB or more. Cloud models default to their maximum. The same page says tasks that need large context, “like web search, agents, and coding tools”, should be set to at least 64,000 tokens, which means most laptops run agents below Ollama’s own recommended floor until you raise it.

Steps to raise it:

  1. Check your VRAM and estimate the KV cache with the formula above.
  2. Set a server-wide default: OLLAMA_CONTEXT_LENGTH=64000 ollama serve.
  3. Or set it per session in ollama run with /set parameter num_ctx 64000.
  4. Or set it per request with the num_ctx option in the API.
  5. Turn on flash attention if it isn’t automatic: OLLAMA_FLASH_ATTENTION=1. Ollama’s FAQ says it “can significantly reduce memory usage as the context size grows.”
  6. Quantize the KV cache with OLLAMA_KV_CACHE_TYPE=q8_0 (about half the memory of f16) or q4_0 (about a quarter, with more precision loss at long context). This needs flash attention and applies to all models.
  7. Confirm with ollama ps. The CONTEXT column shows what was allocated, and PROCESSOR shows whether part of the model spilled to CPU.

Per-request context from Python, using the official ollama package:

 1import ollama
 2
 3with open("contract.txt") as f:
 4    contract = f.read()
 5
 6response = ollama.chat(
 7    model="qwen3.6",
 8    messages=[{"role": "user", "content": contract + "\n\nList every termination clause."}],
 9    options={"num_ctx": 65536},  # tokens to allocate for this request
10)
11print(response.message.content)

If the prompt is longer than num_ctx, it won’t fit in full, so set it above your longest expected input plus output. Ollama’s docs recommend using the model’s maximum context when you can and avoiding CPU offloading, since layers that spill to system RAM run much slower than on the GPU.

How to run a long context window in llama.cpp

llama.cpp’s server exposes the same controls as flags:

  • -c, --ctx-size N: context size. The default 0 loads the value from the model.
  • -fa, --flash-attn on|off|auto: flash attention, auto by default.
  • -ctk and -ctv: KV cache type for keys and values (f16 default; q8_0, q4_0 and others allowed).
  • -np, --parallel N: number of slots. Each slot needs its own share of the context.
  • --rope-scaling yarn, --rope-scale N, --yarn-orig-ctx N: extend a model past its native length.

Qwen’s card for Qwen3-8B gives the exact command to reach 131,072 tokens with YaRN:

1llama-server -m qwen3-8b.gguf -c 131072 -fa on -ctk q8_0 -ctv q8_0 \
2  --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768

The --rope-scaling part comes from the Qwen card; the cache flags are optional. Qwen warns that the static YaRN in common frameworks keeps the scaling factor fixed regardless of input length, “potentially impacting performance on shorter texts,” so only enable it when you need the extra length.

Free and local options for the largest context

The open-weight models above cost nothing to download, so the “free” largest context window is really a hardware question:

  • Under 24 GiB of VRAM: Ollama defaults to 4K tokens. By the formula above, a quantized 8B model plus a 64K-token q8_0 cache (about 4.5 GiB for Qwen3-8B) can fit, but measure with ollama ps.
  • 48 GiB or more: Ollama’s default rises to 256K tokens, which matches the native windows of Gemma 4 and Qwen3.6.
  • 1M tokens and up: multi-GPU servers. Qwen2.5-7B-Instruct-1M alone needs at least 120 GB at full length.

Front ends that sit on top of Ollama or llama.cpp work within whatever context the backend allocates. If a chat app “forgets” early messages, check ollama ps before blaming the model.

When long context isn’t the answer locally

Long context on a local GPU is slow to fill and expensive in memory. For most local agents, it’s cheaper to keep a short window and store history outside the model.

What the numbers say about long context vs memory

The one published head-to-head on an open model is in the Hindsight paper (arXiv 2512.12818, December 2025, run by the vendor). On LongMemEval-S, the same open 20B model (GPT-OSS-20B) scored 39.0% overall with the whole conversation history placed in its context window, and 83.6% when a memory layer retrieved the relevant parts instead. By question type, full context scored 21.1% on multi-session questions against 79.7% with memory, and 31.6% on temporal reasoning against 79.7%. Hindsight’s own runs were judged by GPT-OSS-120B, so compare them with its competitors’ numbers with care, but the full-context and memory runs here use the same model and judge. A bigger window lets the model see more; it does not make it find the right line in a long history, and the KV cache bill above grows with every token you add. Open-source memory servers such as Hindsight run fully local with Ollama, LM Studio or llama.cpp and send back only the memories that fit a token budget you set; keep that max_tokens budget well under your window, since its default retrievals are larger than Mem0’s. Our guide on how to give a local LLM memory covers that route, and context window limitations and solutions explains why accuracy also falls as the window fills.

For the basics of tokens, input and output, start with what the context window of an LLM is.