Business IEA Says Data Center Power Demand Could Double to 1,000 TWh by 2030, With AI Supplying Three Quarters of the Growth Chatbots ChatGPT Now Transcribes as You Talk: OpenAI Removes the 10-Minute Cap and Streams Words Live on Every Plan Business Ex-Alibaba Executive's Everful Raises $48.5 Million to Put AI Agents in Charge of Sourcing From 30,000 Factories AI Agents Google Built a Mirror Dimension of the Internet to Train Its Chrome AI Agents Without Breaking the Real Web Productivity Canva Finally Adds an Off Switch for Its Generative AI Features, and a Filter to Hide AI-Made Templates Business Anthropic Begins Trading Today: 980 Million Shares at $83, a $1.5 Trillion Valuation and the Largest IPO on Record News Google Books Blue Origin New Glenn to Launch TPU-Carrying AI Satellites in Early 2029 Security Hackers Used the AI Penetration Tool ARTEX to Disrupt European Commission Systems for Over Ten Days Business IEA Says Data Center Power Demand Could Double to 1,000 TWh by 2030, With AI Supplying Three Quarters of the Growth Chatbots ChatGPT Now Transcribes as You Talk: OpenAI Removes the 10-Minute Cap and Streams Words Live on Every Plan Business Ex-Alibaba Executive's Everful Raises $48.5 Million to Put AI Agents in Charge of Sourcing From 30,000 Factories AI Agents Google Built a Mirror Dimension of the Internet to Train Its Chrome AI Agents Without Breaking the Real Web Productivity Canva Finally Adds an Off Switch for Its Generative AI Features, and a Filter to Hide AI-Made Templates Business Anthropic Begins Trading Today: 980 Million Shares at $83, a $1.5 Trillion Valuation and the Largest IPO on Record News Google Books Blue Origin New Glenn to Launch TPU-Carrying AI Satellites in Early 2029 Security Hackers Used the AI Penetration Tool ARTEX to Disrupt European Commission Systems for Over Ten Days

Context Windows Keep Growing - Here Is What Actually Breaks

Context Windows Keep Growing - Here Is What Actually Breaks

Bigger context does not mean better recall. Models still lose information in the middle of long inputs, and cost scales with everything you paste in.

Advertised context lengths now run into the millions of tokens. In practice, dumping an entire archive into a prompt is one of the fastest ways to get a worse answer at a higher price.

Three effects that show up immediately

  • Position bias. Information at the very beginning and very end of a long input is recalled more reliably than material buried in the middle.
  • Attention dilution. As the input grows, relevance per token falls. Irrelevant passages crowd out the ones that matter.
  • Cost and latency. Both scale with input size, and long prompts make streaming feel sluggish.

What works instead

Retrieval still earns its keep. Split documents into coherent sections, index them, and fetch the handful of passages that match the question. A well-tuned retrieval step routinely outperforms a naive full-document paste, and it costs a fraction as much.

Where full context genuinely helps is when the task depends on the whole document — a contract, a specification, a novel. Even then, structure the input: headings, explicit section markers and a short summary at the top measurably improve results.

A useful mental model

Treat context as a budget rather than a container. Every passage you include should be able to justify its place. If you cannot say why a paragraph is there, removing it will usually help.

Comments (0)

Log in to join the discussion

Log In

No comments yet