Context Windows & Context Management

The marketing numbers (1M, 10M tokens) are real. The useful capacity is not. In 2023, researchers discovered that LLMs have a U-shaped attention pattern: they reliably attend to information at the beginning and end of the context, but performance degrades for information placed in the middle. In 2025, Chroma tested 18 frontier models including GPT-5.4, Claude Opus 4, and Gemini 2.5 Pro. Every single model still showed performance degradation as input length increased. Claude models degraded the slowest. No model was immune.

• ▸NIAH (Needle in a Haystack) tests: modern models often score near 100% -- but NIAH is considered too easy. It only tests single-fact retrieval.
• ▸RULER benchmark (more rigorous): tests multi-hop reasoning and aggregation across long contexts. Almost all models fail to maintain performance at longer lengths even when they ace NIAH.
• ▸The primacy effect: the very beginning of the context (system prompt, first instructions) receives disproportionate attention.
• ▸The recency effect: the end of the context (the most recent message) also gets strong attention.
• ▸The middle gap: facts or documents placed in the middle 40-70% of a long context are the least reliably retrieved.
• ▸The effect grows with length: a 4K context shows mild degradation, a 200K context shows substantial drops for mid-context retrieval.

Context costs money proportional to size -- but the split between input and output matters a lot. Input tokens are 3-5x cheaper than output tokens across providers. And cached input tokens are up to 10x cheaper than fresh input tokens. Here's what a realistic 50K-token RAG request actually costs.

Assume: 45K input tokens (system prompt + 10 retrieved chunks + conversation history) and 5K output tokens (model response). This is a realistic mid-sized production request.

Prompt caching lets you reuse previously processed context across API calls. Instead of the model re-processing your 30K-token system prompt and knowledge base on every request, it reads from a server-side cache at a fraction of the cost. Cache hits are billed at 10% of standard input rates on Anthropic; at 50% on OpenAI (auto-applied); and through explicit context caching on Gemini.

The question is not 'how do I fit more in context?' -- it is 'what should be in context at all?' More context is not free. It costs money, it takes latency, and past a point it hurts quality. Here are the strategies ordered by when to reach for them.

Treat your context window as a budget with fixed allocations for each category. Silent overflow -- where the model drops content to fit -- is a silent quality bug. Explicit budgeting prevents it.

The 1M-token context window exists. But 'fits in context' and 'reliably understood in context' are different things. Here is what the research and production experience actually shows -- without fabricated percentages.

• ▸RAG: 3-8 highly relevant chunks (2K-6K tokens) consistently outperforms 30+ marginally relevant chunks (30K+ tokens). More chunks means more noise and more mid-context content that the model under-weights.
• ▸Conversation history: summarize after roughly 15-20 turns. Beyond this, the model reliably attends to recent turns and underweights earlier ones -- so preserving verbatim history wastes tokens you could use for recent context.
• ▸Code context: providing a full file is often better than isolated snippets because the model needs structural context (imports, class definitions). But providing 50 files rarely helps for a single-function task.
• ▸Long-form analysis (legal, research): models can reliably extract specific facts from very long documents. They struggle to synthesize insights across the full document from a single pass. For synthesis tasks, break it into chunks.
• ▸The practical ceiling: diminishing returns set in well below the model's maximum context. The correct ceiling depends on your task -- measure quality at different context sizes rather than assuming more is better.