LLM Context Window History: 512 Tokens to 1M+

7 min read

How LLM context windows grew from 512 tokens in GPT-1 to about 1M today: a dated timeline of models and the arxiv papers on attention and RoPE behind it.

The LLM context window grew about 2,000-fold in seven years. GPT-1 read 512 tokens in 2018; GPT-3 read 2,048 in 2020; GPT-4 launched with 8,192 in March 2023. Then came a fast run: 100K (Claude, May 2023), 200K (Claude 2.1, November 2023), 1M (Gemini 1.5, February 2024). By October 2026 about 1M tokens is standard on frontier APIs.

That growth came from a handful of research ideas: cheaper attention, better position encodings, and ways to stretch a trained model’s window. This page gives the dated timeline and the papers, each linked to arxiv. For what a context window is and how it’s used, start with the context window of an LLM.

What is the history of the LLM context window?

The history of the LLM context window is the story of how the maximum tokens a model can read per request went from hundreds to millions, driven by attention efficiency (FlashAttention, sparse attention), position encodings that stretch (RoPE, ALiBi, YaRN), and distributed training on long sequences (Ring Attention).

It splits into three eras:

  • 2017 to 2022: the 512 to 2K era. Quadratic attention cost kept windows short. Research focused on making attention cheaper.
  • 2023: the long-context jump. Windows went from 8K to 200K in one year, while FlashAttention and RoPE-scaling papers made long windows cheaper to train and extend.
  • 2024 to 2026: the million-token era. Gemini 1.5 hit 1M, then 2M. By 2025 OpenAI and Anthropic had 1M models too, and the open question became how much of that window models actually use.

Timeline of context window sizes by model

Every row below is checked against the model’s paper, launch post or official model page.

DateModelContext windowSource
2018GPT-1512 tokensTrained on “sequences of 512 tokens” (paper)
2019GPT-21,024“increase the context size from 512 to 1024 tokens” (paper)
May 2020GPT-32,048“All models use a context window of nctx = 2048 tokens” (arxiv 2005.14165)
Feb 2023LLaMA (Llama 1)2KLlama 2 paper, Table 1 (arxiv 2307.09288)
Mar 2023GPT-48,192; plus gpt-4-32k variantOpenAI GPT-4 model page
May 2023Claude9K to 100K“from 9K to 100K tokens, corresponding to around 75,000 words” (Anthropic)
Jul 2023Llama 24K“doubled the context length” (arxiv 2307.09288)
Nov 2023GPT-4 Turbo128,000OpenAI GPT-4 Turbo page
Nov 2023Claude 2.1200,000“roughly 150,000 words, or over 500 pages” (Anthropic)
Feb 2024Gemini 1.5 Pro128K standard, 1M preview“successfully tested up to 10 million tokens” (Google)
Mar 2024Claude 3200KInputs over 1M possible for select customers (Anthropic)
Jun 2024Gemini 1.5 Pro2M for all developersGoogle Developers Blog
Apr 2025Llama 4 Scout10MUp “from 128K in Llama 3” (Meta)
Apr 2025GPT-4.11,047,576OpenAI GPT-4.1 page
Aug 2025Claude Sonnet 41M“a 5x increase” (Anthropic)
Oct 2026GPT-6 family, Claude 5.x, Gemini 3.1 Proabout 1M to 1.05MSee LLM context window comparison

A few patterns stand out:

  • The doubling era was slow. GPT-1 to GPT-3 took two years to go from 512 to 2,048.
  • 2023 was the break. In twelve months the frontier went from 8K to 200K, a 25x jump.
  • The hosted ceiling has been flat since 2025. OpenAI, Anthropic and Google all settled near 1M. Gemini 1.5 Pro’s 2M from 2024 is still larger than most current hosted models.
  • Open weights went furthest on paper. Llama 4 Scout’s 10M is the largest advertised window, though running it needs a lot of KV cache memory; see the largest context window open-source LLMs.

Key context window papers on arxiv

These are the papers most often cited for long context, in date order. Each claim below comes from the paper’s abstract.

Attention Is All You Need (June 2017)

Vaswani et al., arxiv 1706.03762 introduced the transformer. Its self-attention compares every token with every other, with cost O(n² · d) per layer. That quadratic term is the reason early windows were short, and most later papers attack it.

Transformer-XL (January 2019)

Dai et al., arxiv 1901.02860 added segment-level recurrence so a model could carry state past a fixed-length window. It reported learning dependencies “80% longer than RNNs and 450% longer than vanilla Transformers,” and evaluation “up to 1,800+ times faster” than vanilla transformers.

Longformer (April 2020)

Beltagy et al., arxiv 2004.05150 replaced full attention with local windowed attention plus a few global tokens. Its attention “scales linearly with sequence length,” which made documents of thousands of tokens practical. It’s the best-known example of sparse attention.

RoPE (April 2021)

Su et al., arxiv 2104.09864 proposed Rotary Position Embedding, which “encodes the absolute position with a rotation matrix” while capturing relative position in attention. Llama uses RoPE (the Llama 2 paper says so), as do many open models. It matters for history because almost every later window-extension trick works by rescaling RoPE.

ALiBi (August 2021)

Press et al., arxiv 2108.12409, “Train Short, Test Long,” dropped position embeddings and added a distance penalty to attention scores. A 1.3B model trained on 1,024 tokens extrapolated to 2,048, matching a model trained at 2,048 while training 11% faster with 11% less memory.

FlashAttention (May 2022)

Dao et al., arxiv 2205.14135 didn’t change what attention computes. It changed how: tiling the computation to cut memory traffic on the GPU. It reported a 3x speedup on GPT-2 at 1K length and the first transformers to beat chance on Path-X (16K tokens) and Path-256 (64K tokens). Exact attention became cheap enough to train long.

Position Interpolation (June 2023)

Chen et al., arxiv 2306.15595 showed you can extend a trained RoPE model’s window by squeezing position indices into the original range instead of extrapolating past it. LLaMA models from 7B to 65B reached 32,768 tokens with fine-tuning “within 1000 steps.”

YaRN (August 2023)

Peng et al., arxiv 2309.00071 refined RoPE scaling, needing “10x less tokens and 2.5x less training steps than previous methods.” YaRN is still how many open models advertise two numbers: a native window and an extended one.

Ring Attention (October 2023)

Liu, Zaharia and Abbeel, arxiv 2310.01889 split long sequences across devices in a ring, overlapping data transfer with compute. Context can grow “up to device count times longer,” enabling training and inference on sequences of millions of tokens.

The papers that tested whether long context works

Bigger windows raised a second question: does the model use all of it? Two papers set the agenda.

Lost in the Middle (July 2023). Liu et al., arxiv 2307.03172 found that models do best when the relevant information is at the start or end of the input and worse when it’s in the middle, “even for explicitly long-context models.”

RULER (April 2024). Hsieh et al., arxiv 2404.06654 tested 17 models on 13 tasks. All claimed at least 32K tokens; “only half of them can maintain satisfactory performance at the length of 32K.” Models near-perfect on simple needle-in-a-haystack tests still dropped sharply on harder tasks.

On the other side, the Gemini 1.5 technical report (March 2024) reported “near-perfect retrieval (>99%) up to at least 10M tokens” on its retrieval tests, and called it “a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k).” That’s the vendor’s own evaluation.

The gap between advertised and usable context is covered in context window limitations and solutions.

Why context window growth slowed at about 1M

Windows grew 500x from 2020 to 2024, then the hosted frontier stopped near 1M. The papers above point to why more length stopped being the main goal:

  • Cost per request. Every token in the window is paid for on every call. A full 1M-token prompt is expensive to resend each turn, even with caching.
  • Memory. The KV cache grows linearly with length, so serving very long contexts needs a lot of accelerator memory per request.
  • Diminishing returns. RULER and Lost in the Middle showed accuracy drops before the advertised limit. A bigger window that the model uses poorly doesn’t help much.

So the work moved to the app side: optimizing what goes into the context window with compaction, caching and retrieval, and adding external memory for anything that must persist across sessions.

How to read context window claims today

When you compare models, keep three numbers apart:

  1. Advertised window: the most tokens the API accepts.
  2. Max output: often much smaller, and it shares the window.
  3. Effective window: how far the model stays accurate on your task.

History shows the first number rising fast and the third lagging. Test at your production length before you rely on a model’s headline figure.