Most agents I’ve debugged didn’t get dumber over a long task because the model got worse. They got dumber because the context window filled up with everything that had ever happened and nobody decided what still deserved to be there. The window was being used as a memory — an append-only log of the run — when it should have been run as a cache: a bounded, curated store where every token has to keep earning its place.
That reframing is the whole post. If you treat the window as memory, your only verb is append, and the window grows until latency, cost, and accuracy all degrade together. If you treat it as a cache, you get the verbs that actually matter: admit, evict, summarize, reorder. This is a walk through why the memory framing fails — with the cost arithmetic and the accuracy failure mode — and a concrete context manager that does the caching for you.
Why “append everything” fails on three axes at once
The append-only habit is seductive because it’s simple and it works in the demo. The task is short, the window never fills, and keeping the full history means you never have to decide what to drop. Then the task gets long, and three things go wrong together.
Cost. With most APIs you pay for the full input on every call. In an agent loop that reuses the whole transcript each turn, input tokens accumulate quadratically over the run, not linearly. Say each step adds ~1,500 tokens (a model turn plus a tool result) and you resend the whole history every step. Step 1 bills ~1,500 input tokens; step 20 bills ~30,000; step 40 bills ~60,000. Sum it and a 40-step run bills roughly 1500 × (40 × 41 / 2) ≈ 1.23M input tokens — versus the ~60K that are actually live at the end. You paid ~20× the final-window size just in accumulated re-sends. This is the same multiplicative trap I worked through for failures in Retry budgets: the per-step number feels small and the integral is what bites.
Prompt caching softens the re-send cost — Anthropic’s prompt caching bills cached input tokens at a fraction of the base rate — but caching only helps the stable prefix. The moment you edit, reorder, or summarize earlier context (exactly what a memory-as-log approach avoids doing), you invalidate the cache from the edit point forward. So the append habit and the cache discount are in tension: the thing that keeps your prefix stable enough to cache is not mutating history, and the thing that keeps your window small is mutating history. You have to choose per-region, which is itself a caching decision.
Latency. Time-to-first-token scales with input length because the model has to prefill the whole prompt before it decodes. A window that’s 10× larger doesn’t cost 10× on output, but the prefill tax is real and it’s paid every turn. Long-running agents feel sluggish for a reason that has nothing to do with the model’s “thinking.”
Accuracy. This is the one people miss, and it’s the worst. More context is not monotonically better. The “Lost in the Middle” work (Liu et al., 2023) showed that models retrieve information best when it sits at the start or end of a long input and measurably worse when the relevant fact is buried in the middle — performance on a multi-document QA task degraded as relevant content moved toward the center of a long context. Every irrelevant token you leave in the window is a token competing for attention with the ones that matter, and a place for the relevant fact to get buried. A bloated window doesn’t just cost more; it answers worse. That’s how you get the confident-but-wrong outputs that are so hard to catch.
Put together: append-everything makes the agent slower, more expensive, and less accurate as the task grows. Three axes, same root cause.
The cache framing: four verbs, one budget
A cache has a fixed size and an eviction policy. When something new comes in and there’s no room, the policy decides what leaves. Apply that to the window:
- Admit — does this new content earn a place at all? A 4,000-token tool result that’s 95% boilerplate shouldn’t enter the window raw.
- Evict — when you’re over budget, what leaves? Oldest-first (FIFO) is the naive default; usually you want something smarter that protects pinned content.
- Summarize — the cache-specific superpower a plain LRU cache doesn’t have: instead of dropping old turns, compress them. Ten steps of exploration become three sentences of “here’s what I learned and ruled out.”
- Reorder — given “Lost in the Middle,” put the highest-value content at the edges. Pin the task definition and the live working state at the top and bottom; let the compressible middle be the middle.
The budget is a token count you pick deliberately — not the model’s max context, but the working set you’ve decided keeps quality high. Bigger is not the goal; right-sized is.
A context manager that runs the window as a cache
Here’s a minimal manager. Blocks are typed and can be pinned; the budget is in tokens; eviction summarizes the oldest non-pinned blocks instead of dropping them outright. It’s deliberately small — real code you can lift and adapt, not a framework.
from dataclasses import dataclass, field
from typing import Callable, Literal
Role = Literal["system", "task", "history", "tool_result", "working_state"]
@dataclass
class Block:
role: Role
text: str
pinned: bool = False # never evicted or summarized
tokens: int = 0
def __post_init__(self):
# Swap in your model's real tokenizer; ~4 chars/token is a rough stand-in.
self.tokens = self.tokens or max(1, len(self.text) // 4)
@dataclass
class ContextCache:
budget: int # target working-set size, in tokens
summarize: Callable[[list[Block]], str] # your compaction call (an LLM call)
blocks: list[Block] = field(default_factory=list)
def admit(self, block: Block, max_raw: int = 1200) -> None:
# Admission control: don't let a huge low-signal result in raw.
if block.tokens > max_raw and not block.pinned:
block = Block(block.role, self.summarize([block]), tokens=0)
self.blocks.append(block)
self._evict_to_budget()
def _evict_to_budget(self) -> None:
while self._total() > self.budget:
victims = self._oldest_evictable()
if not victims:
break # everything left is pinned; stop and let the caller decide
summary = self.summarize(victims)
i = self.blocks.index(victims[0])
self.blocks[i:i + len(victims)] = [
Block("history", summary, tokens=0)
]
def _oldest_evictable(self, n: int = 3) -> list[Block]:
# Compress the oldest run of non-pinned history/tool blocks.
run = [b for b in self.blocks
if not b.pinned and b.role in ("history", "tool_result")]
return run[:n]
def _total(self) -> int:
return sum(b.tokens for b in self.blocks)
def render(self) -> list[Block]:
# "Lost in the Middle" ordering: pinned edges, compressible middle.
pinned_top = [b for b in self.blocks if b.pinned and b.role in ("system", "task")]
pinned_bot = [b for b in self.blocks if b.pinned and b.role == "working_state"]
middle = [b for b in self.blocks if not b.pinned]
return pinned_top + middle + pinned_bot
Wiring it into a loop, the discipline is: the task and the live working state are pinned (they must never fall out and they sit at the edges); raw tool results pass through admission control so a giant log gets compressed on the way in; and when the working set exceeds budget, the oldest exploration gets summarized in place rather than dropped, so the agent keeps the lesson without the transcript.
cache = ContextCache(budget=8000, summarize=my_summarizer)
cache.admit(Block("system", SYSTEM_PROMPT, pinned=True))
cache.admit(Block("task", task_description, pinned=True))
for step in range(max_steps):
working = Block("working_state", state.render(), pinned=True)
# keep exactly one live working-state block pinned at the bottom
cache.blocks = [b for b in cache.blocks if b.role != "working_state"]
cache.admit(working)
response = model.call(messages=to_messages(cache.render()))
cache.admit(Block("history", response.text))
result = run_tool(response.tool_call)
cache.admit(Block("tool_result", result)) # summarized on admit if huge
Two things this buys you immediately. The working set stays near 8K tokens no matter how long the run goes, so cost and latency stay flat instead of climbing. And the highest-value content — the task and the current state — is always at the edges where the model attends to it best, instead of getting buried under step 30’s stack trace.
The one that’s actually subtle: summarize lossily and on purpose
The hard part isn’t the plumbing above; it’s the summarize function, because a summary is a lossy compression and what you choose to lose is the whole game. A summary that keeps the wrong thing is worse than eviction — it looks like signal and isn’t.
The rule I’ve landed on: summaries should preserve decisions and dead ends, not narration. For an agent doing a task, the compressible past is mostly “here’s what I tried and what it told me.” The valuable residue is: what did we learn, what did we rule out, what constraints did we discover. The disposable part is the play-by-play. A good compaction of ten debugging steps is “Ruled out the auth layer (tokens valid) and the DB (query returns correct rows). The bug reproduces only with region=eu-west. Have not yet checked the serializer.” — three facts and a pointer, not a transcript.
Get this wrong in the other direction — summarize away a failed approach without noting it failed — and you’ve built the exact conditions for the agent to try it again. That’s not hypothetical; it’s a close cousin of the “stuck but busy” loop drift I wrote about, and a bad compaction policy is one of its quieter causes. The cache doesn’t just save money; done right, it’s part of how you keep the agent oriented.
What I’d do
- Set a token budget below the model’s max and hold it. The working set is a number you own, not a limit the API hands you. Right-sized beats bigger.
- Give every write to the window a verb. Not just append. Ask: admit raw, admit summarized, or not at all?
- Compress tool results at the boundary. Big low-signal results (logs, file dumps, API responses) get summarized on the way in, not left raw to rot in the middle. This pairs with designing tools whose outputs are already legible — see designing tools an LLM won’t misuse.
- Pin the task and the live state to the edges. Cheapest accuracy win available given how models attend to long inputs.
- Make your summarizer preserve decisions and dead ends. Test it directly: can the agent reconstruct what it already ruled out from the summary alone? If not, your compaction is lossy in the wrong place.
- Watch caching tension. Prompt caching rewards a stable prefix; compaction mutates history. Decide per-region which matters more; don’t invalidate a 30K-token cached prefix to save 500 tokens.
The window is the one resource the agent spends on literally every call. Treat it like memory and it fills with whatever happened. Treat it like a cache — with a budget and an eviction policy you chose — and it holds what the task needs. That’s not a nice-to-have; on a long run it’s the difference between an agent that stays sharp and one that gets slower, pricier, and quietly wrong at the same time.