10M-Token Context Windows: What Fits Is Not What Works

A very common scene: a user uploads a hundred-page technical manual and asks the model to find one interface parameter limit on page 82. The document is not even that long by today’s standards, and the answer is genuinely in there. The model replies, “I could not find that information.”

Before blaming the model, rule out the ordinary suspects. The PDF parser may have dropped a table. The application layer may have truncated the input. The product may retrieve from the file instead of passing the full text at all. Only after you confirm that the target content actually entered the context, and test it placed at the beginning, the middle, and the end, can you call it a positional problem.

Once those engineering issues are cleared, a more basic fact remains: a model accepting a very long input is not the same as a model reliably finding and using every piece of information inside it.

Vendors keep racing on window size, from hundreds of thousands of tokens to millions and now tens of millions. Long-context evaluations keep repeating the same warning: maximum context length, effective context length, and economically servable context length are three different things.

Fits, retrievable, usable: three different claims

When Meta shipped Llama 4 Scout with a 10-million-token window, and newer 2026 models continued to sell long tasks, document understanding, and agent workflows, the headline number answered only one question. A context window actually bundles three separate capabilities:

1. It fits. The input is not rejected for exceeding a hard limit.
2. It can be found. Given a question, the model locates the relevant information in the long input.
3. It can be reasoned with. The model combines evidence scattered across positions into a reliable conclusion.

The published maximum answers the first layer. Developers care about the other two. If locating and reasoning do not hold up, ten million tokens is an input ceiling, not a capability.

The most common long-context test is needle-in-a-haystack: hide one special sentence in a mass of irrelevant text and check whether the model finds it. The test has value, but it is closer to exact retrieval than to full document understanding. If the question and the answer share obvious keywords, the model may not need to “understand” the context at all; it only needs to spot a salient pattern. A good single-needle score does not prove the model understood the whole document.

Nelson F. Liu et al. made the structural problem famous in Lost in the Middle: How Language Models Use Long Contexts (preprint 2023, TACL 2024). Across multi-document QA and key-value retrieval tasks, many models used information at the beginning and the end far better than information in the middle, producing the classic U-shaped curve.

Later benchmarks cut the problem finer. RULER adds multi-needle retrieval, multi-hop tracing, and aggregation. NoLiMa reduces literal overlap between question and answer. NeedleChain forces the model to reason along a chain of mutually relevant facts. Together they show that a strong single-needle score says little about dense, evidence-combining long contexts. In production, one missed paragraph can put everything that follows on the wrong foundation, no matter how polished the reasoning looks.

The opposite extreme is also wrong. Positional bias is not equally severe across all models and tasks, and “the model ignores the middle” is too crude. 2026 follow-up work still observes middle-position degradation, but the gap varies widely across models, lengths, and distractor text; some newer models narrow it in certain settings and expose it again under stronger interference.

The accurate conclusion is not “long context is marketing.” It is:

The maximum window states an input limit. It cannot replace effective-context evaluation on real tasks.

Where does the bias come from? There is no single answer

Self-attention lets each token read earlier tokens according to relevance. The longer the context, the more candidate information and distractors the model must sort through.

A popular shortcut explanation says “softmax weights sum to 1, so information is necessarily diluted.” That is not rigorous. Softmax can concentrate most of its weight on a few relevant tokens; nothing forces averaging. The real question is whether the model has learned to assign weight to the right information across long distances, heavy interference, and complex semantic relations.

Current research points to several factors that likely act together.

The first is positional attention bias inside the model. Researchers have observed that, even when content at different positions is equally relevant, the beginning and the end can receive more attention. Calibrating this bias improves middle-information utilization, which also shows it is not an immutable mathematical fate.

The second is the training distribution. In human writing, QA, and code, titles, task instructions, conclusions, and recently mentioned information genuinely carry special weight. Models may absorb those statistical regularities. That is one plausible contributor; it does not license the story that “models learned to be lazy.”

The third is position encoding, causal masking, and context extension methods. Decoder-only models generate answers from left-to-right history states; RoPE extrapolation, long-context continuation training, and local-plus-global attention mixtures all shape performance at different distances. Causal masking does not automatically mean “only recent content is remembered”; architectures and training recipes differ a lot.

Retrieval failure and reasoning failure must also be kept apart. A model may locate the right paragraph and still fail to integrate several conditions inside it, or it may be good at literal matching and lose the answer the moment the question is paraphrased. Conflating the two makes debugging miserable. A single positional curve cannot explain every long-context failure.

KV cache: the longer the context, the uglier the bill

Beyond whether the model can use the information, very long context has a direct engineering limit: memory, bandwidth, latency.

An autoregressive model reads the Keys and Values of all previous tokens to generate each new one. Inference engines cache those vectors, the KV cache, so the prefix is not recomputed at every step. Decoding gets much faster, but cache size grows linearly with token count.

A rough, transparent calculation with Llama 4 Scout’s approximate public configuration: 48 layers, 8 KV heads, 128 dimensions per head. Storing full BF16 Keys and Values for every token at every layer gives, per token:

2 (Key and Value) x 48 (layers) x 8 (KV heads) x 128 (dims) x 2 (bytes)
= 196,608 bytes
~ 192 KiB

Under this simplified model:

• 2M tokens needs roughly 393 GB;
• 10M tokens needs roughly 1.97 TB.

This is an order-of-magnitude illustration, not the precise bill of a real deployment. Actual models may use local attention, hierarchical caches, KV quantization, parallel sharding, and other optimizations, and weights, activations, communication buffers, and runtime headroom also need memory.

The bill continues. Measured against 80 GB H100s, 1.97 TB is about 25 cards of raw capacity, and real deployments need more headroom. H200s with 141 GB and B300s with 288 GB change the arithmetic, but serving a full ten-million-token context is still not a cheap single-machine task.

This is why “the vendor supports 10M tokens” does not mean every request runs ten million tokens cheaply and fast. The maximum window is a capability boundary; throughput, time-to-first-token, and concurrency cost are a separate set of metrics.

Inference systems relieve the memory pressure in different ways, and the remedies should not be conflated:

• PagedAttention manages the KV cache in blocks to cut reservation waste and fragmentation. It does not inherently delete middle tokens.
• KV cache quantization stores Keys and Values at lower precision, trading precision risk for memory.
• Local or sparse attention shrinks the history each step must touch; whether distant information survives depends on the design.
• StreamingLLM keeps a few initial attention sinks plus a recent window, which suits continuous generation and does not suit tasks that must recall any historical detail on demand.
• Offload and distributed KV caches move the cache to CPU or other devices and import transfer and scheduling costs.

Some of these only reduce wasted memory; some genuinely shrink the accessible history. They must be discussed separately.

Engineering long context anyway

Do not stare only at the model.

First, find the layer where information is actually lost. For PDFs, scans, and complex tables, check whether the parsed text contains the target at all. Then check total length after tokenization, application and gateway truncation, and where the target information finally sits in the context. If the product has file upload, find out whether it passes the full text or retrieves and summarizes first.

Second, give key content explicit structure instead of relying on position tricks. Keep system instructions and task constraints in clear, stable positions. For long documents, use section labels, page numbers, field names, and XML or Markdown boundaries. Put the question after the material and require the model to cite the source or page. Mechanically repeating the same sentence at the beginning and the end is not a universal fix and can create conflicting instructions.

Third, do not treat long context and RAG as an either-or choice. Long context alone is expensive and not always stable. For parameter lookup, clause finding, and locating code definitions, BM25, embeddings, and reranking are usually cheaper and more stable. For cross-section comparison, whole-document summarization, and tasks that need broad context, aggressive chunking can destroy global relationships. Real systems hybridize: retrieve to narrow the scope, then hand the relevant sections plus the necessary global background to the model.

Fourth, self-hosted inference must cap both window and concurrency. In vLLM, the usual knobs include --max-model-len, --max-num-seqs, --gpu-memory-utilization, KV cache precision, chunked prefill, tensor parallelism, and KV offload. Stacks differ, but one invariant holds: capping concurrency does not solve the memory of a single huge request, and adding offload can turn a capacity problem into a latency problem. Misconfigure this and you lose concurrency at best and OOM the service at worst.

Finally, build your own long-context regression tests. Record at least the model version, actual token count, target information position, task type, repetition count, and parsing result, and test exact retrieval, semantic retrieval, multi-evidence combination, and global summarization separately. That is the only way to know how much “effective context” a model really offers for your workload.

Longer windows are progress. They let models see more complete material and remove much of the information loss caused by crude chunking. But a context window is not a disk and not a reliable database. Accepting input, finding information, reasoning over it, and serving it at reasonable cost are four different thresholds.

Treating the maximum input length as reliable memory on real tasks is a premise that was wrong from the start.

References

1. Nelson F. Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL 2024: https://aclanthology.org/2024.tacl-1.9/
2. Cheng-Ping Hsieh et al., RULER: What’s the Real Context Size of Your Long-Context Language Models?: https://arxiv.org/abs/2404.06654
3. Ali Modarressi et al., NoLiMa: Long-Context Evaluation Beyond Literal Matching, ICML 2025: https://proceedings.mlr.press/v267/modarressi25a.html
4. Hyeonseok Moon, Heuiseok Lim, NeedleChain: Measuring Intact Long-Context Reasoning Capability of Large Language Models: https://arxiv.org/abs/2507.22411
5. Cheng-Yu Hsieh et al., Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization: https://aclanthology.org/2024.findings-acl.890/
6. Chuyifei Zhang et al., Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks: https://arxiv.org/abs/2605.23170
7. Meta, The Llama 4 herd: https://ai.meta.com/blog/llama-4-multimodal-intelligence/
8. Hugging Face, Llama 4 model configuration: https://huggingface.co/docs/transformers/main/en/model_doc/llama4
9. Woosuk Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention: https://arxiv.org/abs/2309.06180
10. Guangxuan Xiao et al., Efficient Streaming Language Models with Attention Sinks: https://arxiv.org/abs/2309.17453
11. vLLM, Engine Arguments: https://docs.vllm.ai/en/stable/cli/serve/