Why more context can make AI slower, costlier — and less reliable
There’s a moment in almost every AI rollout where someone on the team says some version of: “Let’s just give it everything. More context can’t hurt.”
It’s an understandable instinct. We’ve spent two years being told that bigger context windows are the upgrade — 8K to 32K to 200K to a million tokens, each jump treated like a milestone. So the logic follows naturally: if the model can hold more, why not feed it more? Every document, every tool response, every prior message in the conversation. Let the model sort it out.
Except it turns out the model doesn’t sort it out. It drowns in it.
A team at Microsoft recently ran a quiet experiment that’s worth more attention than it’s gotten. They were trying to get GPT-5 to reliably itemize hotel expenses inside Dynamics 365 — a task that sounds boring, which is exactly why it matters. This is the kind of grinding, repetitive work enterprises actually need agents to handle all day, not the flashy demo tasks that get the attention. The agent had to read forms, call tools, and keep track of its own progress across a long, multi-step process.
The obvious approach — full conversation history, every tool call, every response, hold nothing back — got the job done 71% of the time. Not bad. But it burned through nearly 1.5 million tokens and took over 14 hours to run the benchmark. The failures weren’t random, either: the agent would lose track of what it had already done, act on outdated information, or get tangled in its own history.
Then they tried something almost rude by comparison: they made the agent forget most of what it had seen. Instead of full history, it got only the last five tool calls, plus a short, running summary of everything before that. Not more context — a fraction of it, refreshed and compressed.
Completion jumped to 91.6%. Token use dropped by nearly two-thirds. Runtime dropped by roughly 60%.
Read that again, because it’s the part that should sting a little if you’ve been assuming context windows are the whole story: the agent got dramatically better by remembering less, not more.
Losing the forest for the trees
You know the phrase — “you can’t see the forest for the trees.” It’s usually used to describe people, but it turns out to describe language models with unsettling precision. Give an agent everything, and everything starts to look equally important. The tool call from twelve steps ago carries the same weight as the one from thirty seconds ago. The model isn’t reasoning about relevance anymore; it’s wading through noise, and somewhere in that wading, it loses the thread of what it’s supposed to be doing.
This is easy to miss because “context” sounds like an unambiguous good. More information should mean better decisions — that’s how it works for people, mostly. But a model doesn’t have judgment about what to skim and what to dwell on unless you build that judgment in. Left alone, it treats a stale error message from an hour ago with the same seriousness as the instruction you just gave it. Humans do this too, if we’re honest — the mind that’s holding onto every open tab, every unresolved thread, every “just in case” fact, isn’t the mind that makes the sharpest call in the moment. Clarity has always come from what you leave out, not just what you keep in.
The genuinely interesting part of the Microsoft result isn’t “less is more” as some kind of maxim — it’s what got left out and what got kept. Pruning alone, just chopping the history down to the last few tool calls, already got the agent from 71% to 79%. That’s decent, but it’s a bit reckless — you’re just throwing things away. The real jump came from adding a compact summary of everything that got pruned. The agent didn’t lose the plot of the task; it lost the clutter, while keeping a lightweight memory of where things stood. That distinction — between discarding and distilling — is doing almost all of the work in that 91.6% number.
Memory versus useful memory
This is the shift worth sitting with: we’ve been treating “context” as a storage problem — how much can we fit — when it’s actually a curation problem — what’s worth keeping. Those are completely different engineering challenges, and most teams building agents right now are still solving the first one.
If you’re building or buying agentic systems, this shows up in a few concrete places:
- Stale-state errors — an agent acting confidently on a tool response that’s ten steps old and no longer true, because nothing flagged it as stale.
- Verbose tool outputs treated as gospel — enterprise systems return walls of JSON, and if none of it gets filtered, the agent re-parses the same noise every turn, at real cost, with real risk of losing the thread.
- The assumption that a bigger context window is a substitute for good context management — it isn’t. A million-token window filled with unfiltered history is not more capable than a 50,000-token window with the right five things in it. It’s just more expensive to be confused.
None of this is exactly new — anyone who’s debugged a long-running agent has felt it in their gut: watching it lose the plot around turn thirty, then repeat a mistake it already made. What’s new is having a clean, controlled number attached to the intuition. 71% to 91.6%, same model, same task, radically less to hold onto.
The part worth remembering
Worth saying plainly: this is one study, one narrow task, one model family behind the headline number. Nobody should read it as proof that every agent everywhere should aggressively forget things. The authors are careful about that themselves — they frame it as strong evidence for a category of enterprise tool-use work, not a universal law. Good research usually comes with that kind of restraint, and it’s a decent sign that the number is real rather than a marketing artifact.
But the underlying shift it points to feels bigger than the benchmark. For the last two years, the AI conversation has mostly been about scale — bigger models, bigger windows, bigger everything. This is a quiet signal that the next real edge won’t come from how much an agent can hold, but from how well it decides what to let go of.
There’s an old bit of advice that gets handed to writers, and it applies here better than it has any right to: “kill your darlings.” Not everything you’ve gathered deserves to survive into the final draft. The agents that win the next round won’t be the ones with the longest memory. They’ll be the ones that know what to forget.
