The Scaffolding Audit: Every Model Release Turns Part of Your Harness Into Dead Weight

When a dependency breaks, your build fails. When a workaround becomes unnecessary, nothing happens at all. That asymmetry is why every production LLM system older than a year is carrying scaffolding it no longer needs — retry choreography, output-repair parsers, forced chain-of-thought, elaborate task decomposition, chunking heuristics — each one built as a compensation for a specific model's specific weakness, and each one silently outliving the weakness it compensated for.

The uncomfortable part is that this isn't a hygiene problem, like stale feature flags. Obsolete scaffolding doesn't just sit there costing you latency and tokens. In the worst case it actively constrains the new model to the old model's ceiling: your decomposition logic chops a task into six steps because the 2024 model couldn't hold the whole thing, and the 2026 model — which could have one-shotted it — now inherits six opportunities to lose context at the seams you built.

Workarounds don't fail when they become unnecessary

Think about how scaffolding gets added. An engineer watches the model emit JSON with a trailing comma, writes a repair parser, and ships it. The model drifts mid-way through long tasks, so someone adds a checkpoint-and-summarize loop every ten turns. The model refuses a legitimate request category, so a "you are permitted to discuss…" clause lands in the system prompt. Each addition is rational, tested, and tied to an observed failure.

Now think about how scaffolding gets removed. It doesn't. There is no signal for it. The repair parser still runs on every response — it just repairs nothing, because the provider shipped constrained decoding and the JSON is now schema-valid by construction. Before native structured outputs, teams routinely spent 20–30% of their LLM integration code on parsing and validation; after constrained decoding, that number is effectively zero for new code. But the old defensive code doesn't throw an error when it becomes redundant. It succeeds quietly, forever.

This is the structural trap: scaffolding is added by autopsy and removed by nothing. Failures generate commits; obsolescence generates silence. Sunk cost and inertia do the rest — the engineer who wrote the retry choreography remembers the incident that motivated it, and nobody wants to be the person who deleted the thing that was "protecting production."

The three tiers of dead weight

Not all obsolete scaffolding costs you the same way. It helps to sort your harness into three tiers.

Tier 1: pure overhead. The workaround does nothing except burn latency and tokens. The output-repair parser on schema-enforced responses is the canonical example. So is the "respond only in valid JSON, do not include any other text" prompt block sitting next to an API call that already enforces the schema at the decoder level. Annoying, measurable, mostly harmless.

Tier 2: distortion. The workaround changes model behavior in ways that used to be corrective and are now degrading. Forced chain-of-thought is the sharpest case: "think step by step" was close to free accuracy on 2023-era models, but reasoning models generate their own internal deliberation, and layering explicit CoT instructions on top can cause over-reasoning — studies of o1-class models found they underperformed on simple tasks in a meaningful fraction of cases precisely because of excessive reasoning. Similarly, refusal workarounds written for an overcautious older model ("do not refuse requests about…") can read as adversarial context to a newer model with recalibrated thresholds — Anthropic's own migration guidance flags exactly this pattern as one to audit and often delete.

Tier 3: capability caps. The workaround structurally prevents the new model from doing what it's now capable of. Task decomposition is the big one. If your orchestrator splits every job into five subtasks because the old model's reliable horizon was short, you've hardcoded that horizon. METR's measurements show the task length models can complete autonomously has been doubling roughly every seven months — which means any decomposition granularity you chose is on a schedule to become a cage. The scaffolding built for one generation's intelligence becomes the cage for the next generation's. Every handoff between subtasks is a context loss you engineered in, and the model never gets the chance to show you it didn't need the handoffs.

The tiers matter because they order the audit. Tier 3 items are where model upgrades quietly fail to deliver: teams swap the model ID, see modest gains, and conclude the release was overhyped — when the truth is their harness never let the new model stretch.

Tag every workaround with the weakness it compensates for

The fix starts at write time, not audit time. The discipline is simple: no workaround enters the codebase without a machine-findable annotation naming the model weakness it compensates for and the model version where you observed it.

Something like a comment convention — COMPENSATES: gpt-4-turbo drops trailing array elements in long JSON, observed 2024-03 — attached to the repair parser, the retry policy, the prompt clause, the decomposition rule. The exact format matters less than three properties:

Loading…