19 posts tagged with "technical-debt"

Every mature AI product eventually grows a system prompt that nobody on the current team fully understands. It started as forty tokens of plain English, and twenty months later it is a 4,000-token wall of conditional clauses, refusal templates, formatting rules, persona reinforcements, edge-case warnings, and one peculiar sentence about Tuesdays that nobody can explain. Each line was added in response to a specific failure: a customer complaint, a Slack ping from legal, a regression caught by an eval, a one-off bug that surfaced during an investor demo. The engineer who wrote line 37 has rotated to another team. The engineer who wrote line 112 was a contractor whose Notion doc was archived. The eval suite covers maybe a third of the behaviors the prompt is asserting, and nobody is sure which third.

So the prompt becomes load-bearing in the worst possible way: it works, the team knows it works, and the team has stopped touching it. Engineers who should be iterating on the prompt route their changes around it instead — adding a post-processing filter here, a few-shot wrapper there, a parallel "v2 prompt" feature-flagged off in case anyone ever finds the courage to A/B test the replacement. The prompt has stopped being software and has become a relic. And once that happens, the prompt is no longer the lever you use to improve the product. It's the constraint shaping it.

Three months after shipping a document summarization feature, a team at a mid-size company runs a prompt improvement. The new prompt scores better on the five examples they tested manually. They deploy it Friday afternoon. Monday morning, their Slack is full of user reports: summaries are now truncating half the document and presenting the truncated version as complete. The feature looked fine. The change passed review. Nobody noticed because there was no evaluation — no golden test set, no regression baseline, no automated check. The ratchet had been turning silently for months.

This is eval debt in its most recognizable form. The team didn't skip evaluations because they were careless. They skipped them because writing evaluations for AI features is harder than it sounds, the feature shipped fast and looked good, and nobody wanted to slow down a team with momentum. Now they're paying the compound interest.

In 2022, a team spent three months fine-tuning a BERT-based classifier to categorize customer support tickets. It was a genuine win — 94% accuracy where their old rule-based system topped out at 70%. Two years later, the same classifier runs on aging infrastructure, requires a specialist to retrain whenever categories shift, and gets beaten on a fresh benchmark by a zero-shot prompt to a frontier model. Nobody wants to touch it. The engineer who built it left. The current team is afraid that deprecating it will break something. The feature is frozen.

This is the frozen feature trap. It's one of the quieter forms of AI technical debt, and it's accumulating across the industry as teams discover that what looked like a moat was actually a hole they've been shoveling money into.

Your AI feature shipped cleanly. The documentation looked good: input schema, expected outputs, a worked example. Three months later, a model update arrives silently. The outputs shift. Your docs are wrong but nobody knows it yet — because they still look right.

This is the core of AI documentation debt, and it compounds faster than any other kind of technical debt because the failure is invisible until a user finds it.

Your codebase has the same authentication logic implemented three different ways, and nobody on the team wrote any of them. A quick git blame shows the same engineer on all three files, but ask that engineer and they'll tell you they just accepted what the AI suggested and it "looked right." The anti-pattern didn't spread because someone was lazy. It spread because an AI model with no memory of your existing auth module generated plausible-looking implementations every time someone opened a new file and asked for help.

This is the copy-paste contagion, and it's structurally different from the classic copy-paste problem you already know how to fight.

A feature ships at 92% pass rate. The launch deck celebrates it. Twelve months later the same feature is at 78% — no incident report, no failed deploy, no single change to point at, just a slow erosion that nobody owned watching for. The team blames "hallucinations" or "user behavior shift," picks a junior engineer to investigate, and sets a quarterly OKR to "improve quality." The OKR misses. The feature ships an apologetic dialog telling users the AI sometimes makes mistakes. Six months after that, it's deprecated and replaced with a new version that ships at 91% pass rate, and the cycle starts again.

This isn't bad luck. It's the second clock that AI features run on, the one that nobody marks on the release calendar at launch. Conventional software has feature decay too — dependency drift, codebase rot, the slow accumulation of half-applied refactors — but those decay on a clock the engineering org already understands and budgets for. AI features have all of that, plus a parallel set of decay sources that conventional amortization assumptions don't model: model deprecations, vendor weight rotations, distribution shift in user inputs, prompt patches that compound, judge calibration drift, and the quiet aging of an eval set that no longer represents what production traffic looks like.

The architectural realization that has to land — before the next AI feature ships, not after — is that AI features have a non-zero baseline maintenance cost. The feature isn't done when it launches. It's enrolled in a maintenance schedule it can't escape, and the team that didn't budget for that schedule is going to discover it the hard way.

Engineering leaders are comfortable with the idea that code rots. Dependencies need updating, infrastructure has lifecycle management, certificates expire on a calendar nobody disputes. Yet the prompt repository gets treated as a write-once-read-many artifact — even though it defines how your product talks to a probabilistic engine that ships behavior changes every six weeks.

The system prompt tuned six months ago against the model that was current then is still in production. The few-shot examples chosen against a tokenizer that has since changed are still being injected on every call. The reranker prompt was tuned against an embedding endpoint the vendor deprecated last quarter. Nobody scheduled a review. Nobody is going to.

This is not a hypothetical failure mode. When one team migrated their prompt suite — meticulously stabilized against GPT-4-32k — to GPT-4.1 and GPT-4.5-preview, only 95.1% and 97.3% of their regression tests passed. A 3-5% silent quality regression is not a rounding error in production; at any non-trivial scale it is a customer-visible degradation that nobody on the team intentionally shipped. And those are the teams that even had a regression test suite. The median team's "regression test" is whatever vibes the on-call engineer formed during the last incident.

The category we are missing is prompt asset depreciation: a maintenance discipline that treats every production prompt as a depreciating asset with a known lifespan, not a constant.

When an agent writes a function and a comment in the same diff, the comment is not documentation. It is a paraphrase of the code at write-time, generated by the same model from the same context, and it is silently wrong the first time the code shifts. The function gets refactored, an argument changes type, an early-return gets added, the comment stays. By next quarter, the comment is encoding a specification that no longer matches the code, and the next reader trusts the comment because the comment is easier.

This is an old failure mode — humans-edit-code-comments-stay-stale — but agents accelerate it across three dimensions at once. Comment volume goes up because agents add a doc block to every function whether it needs one or not. The comments are grammatically perfect, so reviewers don't flag them as low-quality. And the comments paraphrase the code in different terms than the code actually executes, so they look like documentation but encode a second specification that drifts independently of the first.

Every team that ships an AI feature without evals tells themselves the same story: we'll add measurement later, after we find product-market fit, after the prompt stabilizes, after the next release. Six months later, the prompt has been touched by four engineers and two product managers, the behavior is load-bearing for three customer integrations, and the team discovers that "adding evals later" means reconstructing intent from production logs they never structured for that purpose. The quarter that was supposed to be new features becomes a quarter of archaeology.

This isn't a planning mistake. It's a compounding one. The team that skipped evals to ship faster is the same team that will spend twelve weeks rebuilding eval infrastructure from incomplete traces, disagreeing about what "correct" meant in February, and quietly removing features nobody can prove still work. The cost of catching up exceeds the cost of building in — not by a little, but by a multiplier that grows with every prompt edit that shipped without a regression check.

The pitch is seductive: describe the behavior in English, the model emits the code, ship it. Prompts become the source, artifacts become the target, and the LLM sits between them like gcc with a friendlier front-end. If that framing held, the rest of software engineering — review, refactoring, architecture — would be downstream of prompt quality. It does not hold. And the codebases built on the assumption that it does start failing in a pattern that is now boring to diagnose: around month six, nobody can explain why a particular function looks the way it does, and every incremental change produces a wave of duplicates.

The compiler metaphor is the root cause, not vibe coding, not model quality, not prompt skill. It is a category error that quietly excuses teams from doing the work that keeps a codebase coherent over years. When you believe the model is a compiler, the generated code is an implementation detail, the same way assembly is an implementation detail of a C program. When you are actually running a team of non-deterministic, context-limited collaborators, the generated code is the asset — and the prompts are closer to Slack messages than to source.

In March 2026, a major e-commerce platform lost 6.3 million orders in a single day — 99% of its U.S. order volume gone. The cause wasn't a rogue deployment or a database failure. An AI coding tool had autonomously generated and deployed code based on outdated internal documentation, corrupting delivery time estimates across every marketplace. The company had mandated that 80% of engineers use the tool weekly. Adoption metrics were green. Engineering discipline was not.

This is what vibe coding at scale actually looks like. Not the fast demos that ship in four days. The 6.3 million orders that vanish on day 365.

In a controlled randomized trial, developers using AI coding assistants predicted they'd be 24% faster. They were actually 19% slower. The kicker: they still believed they had gotten faster. This cognitive gap — where the feeling of productivity diverges from actual delivery — is the early warning signal of a failure mode that plays out over months, not hours.

The industry has reached near-universal AI adoption. Ninety-three percent of developers use AI coding tools. Productivity gains have stalled at around 10%. The gap between those numbers is not a tool problem. It is a compounding debt problem that most teams don't notice until it's expensive to reverse.