51 posts tagged with "context-engineering"

Nobody decides to give an agent forty tools. It happens the way a garage fills up. You wire in a search tool, then a database reader, then someone on the team ships a Slack integration, then the ticketing MCP server gets installed because it was one config line away. Each addition is individually reasonable. Nobody ever removes anything, because removing a tool feels like taking away a capability, and taking away a capability feels like a regression.

Six months later your agent has a tool belt with three tools it uses constantly, a dozen it uses occasionally, and a long tail of two dozen it has technically never selected in production. That long tail is not free. It is not even cheap. Every unused tool in the catalog is actively making the agent worse at choosing among the ones that matter.

There's a specific kind of argument you can only have with a language model. You paste in your code. It rewrites a working call into one that hasn't existed since a major version ago. You correct it. It apologizes, agrees with you, and on the next turn does the exact same thing. You are not fighting ignorance. You are fighting a confident, well-rehearsed memory of a different version of your stack — and that memory was reinforced by more training examples than your correction will ever carry.

This is the failure mode I've come to think of as fighting the prior. The model's parametric knowledge — everything it absorbed during training — contains the popular, the outdated, or simply the different version of the framework you're actually using. When your context and its prior disagree, the prior often wins. And unlike a plain hallucination, this one is dangerous precisely because it's plausible: the deprecated API used to be correct, so the code looks right, passes a casual read, and sometimes even compiles.

Every team building agents eventually hits the same wall. The system prompt started at 400 tokens. Then someone added the database migration checklist. Then the postmortem template, the deploy runbook, the style guide for customer emails. Eighteen months later it's a 9,000-token monolith that nobody dares to edit, because changing the line about rollback procedures somehow degrades the agent's tone in support tickets. You've built the prompt equivalent of a single 50,000-line main.c — and everyone is statically linking against it.

The instinct is to reach for RAG: chunk the runbooks, embed them, retrieve on demand. That fails in a subtler way. RAG is built to retrieve facts, and facts degrade gracefully when fragmented — three out of five relevant chunks about your billing model still tells the agent most of what it needs. Procedures don't degrade gracefully. A database migration runbook retrieved at 60% is not 60% useful; it's a production incident. Steps 1 through 4 without step 5 ("verify replication lag before cutting over") is worse than no runbook at all, because the agent now acts with confidence it hasn't earned.

Here is a failure you have probably watched happen without naming it. Your orchestrator reads the user's request and briefs a worker agent. The worker runs a dozen tool calls, digests the outputs, and reports back a tidy summary. The orchestrator folds that summary into a brief for the next worker, who does the same. Five hops later, the system delivers a confident final answer — and it violates a constraint the user stated plainly in the second sentence of their request. Nobody dropped the constraint on purpose. Each hop just compressed the context a little, in a direction nobody chose, and the compressions compounded.

This is the telephone game, and multi-agent systems play it constitutively. Every handoff in your pipeline is a lossy compression step: the orchestrator's brief is a paraphrase of the user, the worker's report is a paraphrase of its tool outputs, and the final answer is a paraphrase of paraphrases. The question is not whether information gets lost — it does, measurably — but whether you have decided what is allowed to be lost, or left that decision to the sampling temperature.

Your agent answered a customer's billing question with last quarter's pricing, and the postmortem will blame the model. It shouldn't. The prompt was assembled correctly, the retrieval scored well, the model reasoned soundly over everything it was given — and everything it was given was true three days ago. Somewhere between the CRM export, the docs sync, and the vector index rebuild, "current state of the world" quietly became "state of the world as of Tuesday," and nothing in your stack was measuring the difference.

Data engineers solved this class of problem years ago. A downstream dashboard consuming ten upstream tables gets lineage, freshness checks, and an on-call rotation that pages when the nightly job slips. The context window your agent consumes is the same thing — a materialized view joined from docs, tickets, code, CRM, and memory — except nobody owns the join, nothing measures its staleness, and when it serves yesterday's truth the failure gets filed as "the model hallucinated."

For two decades, documentation was where good intentions went to die. You wrote the README during the first sprint, when the architecture was clean and your enthusiasm was high. Nobody read it. By the third sprint it was lying about the build command, and by the sixth it described a service that had been deleted. Documentation was a tax everyone agreed to pay and nobody actually paid — a moral imperative with no feedback loop. Write bad docs and nothing happened. Write no docs and nothing happened either, because the senior engineers carried the architecture in their heads.

Then we pointed coding agents at our repositories, and the feedback loop arrived overnight. The README is now the single highest-leverage file you own — not because anyone gave a motivational talk about documentation hygiene, but because the quality of that file now visibly determines whether your agent ships correct code or confidently hallucinates an architecture that no longer exists.

This is the documentation renaissance, and it has almost nothing to do with the documentation we used to write.

A cache that is being written to but never read from is not a cache. It is a logging system with extra latency, billed by the kilobyte. And the cruelest version of this failure mode is the one where the cache looks healthy from every angle except the one that matters: the set calls succeed, the get calls return quickly, the keys are well-formed, the values are valid, the TTLs are sensible. The only thing wrong is that no get call ever finds the key a previous set call wrote, because a single field in the key changes every time it is computed.

This is the story of a debugging session that added a timestamp to a cache key "so I can tell which cache entry I'm looking at," and the system that quietly paid for fourteen extra LLM calls per conversation for two weeks before anyone noticed.

A user asks: "Does this qualify as a 'transfer' under article 28?" Forty turns later, the model gives an answer to a different question. The transcript shows the model answered the question it was given. The user is reading a complaint that reads like a hallucination. Both are right. The model never saw the user's question — it saw your summarizer's polite translation of it: "user asked about article 28 applicability."

The word "transfer" was the question. The summarizer threw it away because the summarizer's loss function was tuned to preserve facts, not wording, and the rubric never learned the difference between paraphrasing the topic and paraphrasing the constraint. Topic was preserved. Constraint became fog.

This failure mode is structural, not anecdotal. Any application that compresses long conversations with a model-generated summary has a second model in the critical path — one whose quality contract is usually treated as a token-budget knob rather than as a piece of product logic. That asymmetry is where the bug lives.

A long-running agent generates a plan at step 3. The plan reads something like: "if the order returned by get_order in step 1 has status shipped, send the customer a tracking email; otherwise open a refund ticket." The agent confidently picks the email branch. The customer never received a tracking number, because the order was actually in pending. You go to the trace expecting to find a hallucination. What you find is worse: the step-1 tool result is no longer in context. The pruner evicted it between step 2 and step 3 — it ranked low on recency and there was a 12KB transcript to make room for. The plan still ran. The branch was still chosen. The decision now points at evidence that does not exist.

This is not a model failure in the usual sense. The model produced a syntactically valid plan, executed it in order, and made a branch decision. The branch was made against a fact that used to be in context and is not anymore. The chain of thought encoded the condition (if status == "shipped"); the actual status got dropped on the way to the step that needed it. The plan looks deterministic, but it has been quietly cut loose from its evidence.

A user asked our support agent: "Why was invoice INV-2025-08-44719 charged twice on April 3rd?" Forty-five minutes and eighteen tool calls later, the agent confidently reported back: there was no evidence of any duplicate billing on the account that quarter. The user, understandably, escalated. When we replayed the trace, the answer became obvious. The agent had compacted its conversation at turn nine. The summary said the user was "asking about a duplicate charge in early April." It did not contain the string "INV-2025-08-44719." Every subsequent tool call — the ledger lookup, the chargeback API query, the audit log scan — was issued against a paraphrased intent, not the literal invoice number the user typed.

The bug was not in the tools. It was not in the model's reasoning. It was that our context manager had a contract with every downstream component, and nobody had written it down. The contract said: "I will preserve meaning." The components needed: "I will preserve strings."

A user opens your long-session agent and says, in turn 3, "I'm vegetarian and on a tight budget." The conversation continues. Eleven turns later, the pruner runs. It counts tokens, finds turn 3 old and short, and drops it to keep the window inside budget. Turn 14 asks, "what should I cook tonight?" The model, looking at a window where the constraint no longer exists, recommends a $40 ribeye. The user reads this as the agent getting worse, opens the satisfaction survey, and rates the session a 2.

Nothing in your stack will report a memory failure. The token-budget dashboard will show the window staying healthily under the cap. The latency dashboard will be green. The eval suite — which scores single-turn answers against a held-out set — will report no regression. The only signal that the agent's competence dropped is a thumbs-down rating that your product team will attribute to "model variance." It will not be model variance. It will be a pruning heuristic doing exactly what it was tuned to do, on the wrong objective.

A user types "actually, I meant fifty, not fifteen," hits the pencil icon on their last message, and edits it. The UI does what good UIs do: it shows them the corrected message, fades out the old one, scrolls the assistant's stale reply into a struck-through ghost, and presents a clean conversation that reads as if the original mistake never happened. The user, satisfied, sends the next turn. The agent answers using fifteen.

The bug is not in the model. The model received exactly what the server sent it, and the server sent it the original message, the original assistant response, the regret, the edited message, and the new request — all concatenated, all in order, all live. The user is having a conversation they edited. The agent is having a conversation that was never edited. The two transcripts diverge at turn three and never reconcile, and every subsequent turn pays interest on the gap.