842 posts tagged with "ai-engineering"

Text-to-SQL demos never die on syntax. The model writes fluent SQL — better than most junior analysts, honestly — and the query runs, and a number comes back. The demo dies three weeks later, in production, when the CFO notices that the agent's "Q2 revenue" doesn't match the board deck. Not because the SQL was malformed, but because the warehouse contains three defensible definitions of revenue — bookings, recognized, and net-of-refunds — and the model confidently picked one. Just not the one finance uses.

This is the failure mode that matters, and it's invisible to every benchmark you've seen. The fix isn't a better model or a longer prompt. It's a piece of infrastructure most data teams already half-built and then abandoned: the semantic layer. The metrics definitions you wrote for BI dashboards — dbt metrics, LookML, cube definitions — turn out to be the missing tool contract for data agents. The teams shipping reliable agents figured out that the build order is inverted from what everyone assumed: semantic layer first, agent second.

Your design system survived the last decade because humans absorbed it slowly. New engineers learned the button variants through Figma files, PR nitpicks, and the one designer who always caught the wrong shade of gray. That absorption pipeline had a throughput of maybe a few components per engineer per week — slow enough that a design team could police the edges by hand.

Coding agents just broke the pipeline. An agent fleet can generate fifty slightly-wrong buttons before lunch: each one plausible, each one using a hex value that's two shades off, a padding that's 14px instead of your 16px scale step, a border radius invented on the spot. No designer reviews at that speed. And the instruction you reached for first — "follow the style guide" in the system prompt — decays exactly like every other soft instruction: it loses statistical weight as the context window fills, and the model reverts to the generic CSS patterns it learned from a million other people's codebases.

The fix is not a better prompt. It's a category change: your design system has to stop being documentation that humans interpret and become a contract that machines enforce. Documentation asks. Compilers refuse.

Count the readers of your stack traces. For most internal tools, the answer used to be "one tired engineer, occasionally." Today the highest-volume reader of your error output is almost certainly a language model inside a retry loop. Coding agents read your linter warnings, your CLI usage strings, your API error bodies, and your test failures thousands of times a day — far more often than any human ever will. And unlike the human, the agent takes every word literally.

That changes what an error message is. It is no longer documentation of a failure. It is an instruction injected into the context window of the next attempt — a prompt you wrote months ago, now steering fleets of agents you've never met. A precise error converges the loop in one retry. A vague or misleading one sends the agent spiraling: wrong fixes, --no-verify workarounds, hallucinated flags, burned tokens. If you maintain a tool, a service, or a build system, you are already doing prompt engineering. You're just doing it in your error strings, and probably by accident.

The budget meeting for a fine-tuning project always prices the wrong thing. Teams estimate the data pipeline, the training runs, the eval passes — a one-time investment with a clear finish line. Then the model ships, the accuracy chart goes up and to the right, and everyone moves on. Six months later an email arrives: the base model your adapter is welded to has a retirement date. Nothing about your system changed. Everything about its foundation did.

This is the part nobody prices in: a fine-tune is not a product you finished. It is a fork of someone else's codebase, and every base-model release is an upstream rebase you didn't schedule. Anyone who has carried private patches against a fast-moving open-source project knows exactly how this story goes — the fork is cheap to create and expensive to keep.

Here is a postmortem pattern that is becoming a genre. The primary model was healthy all night. Latency was flat, token throughput normal, provider status page green. And yet every user request failed for forty minutes — because the safety classifier sitting in front of the model timed out, and the middleware wrapped that timeout in a generic exception, and the exception handler returned a refusal. Your model didn't go down. Your gate went down, and the gate was wired to fail closed by an engineer who never thought of it as a decision.

The uncomfortable truth is that most teams run a second machine-learning system in production without admitting it. The moderation classifier, the jailbreak detector, the PII scrubber, the topical filter — each one is a model, with its own latency distribution, its own error rates, its own training-data assumptions quietly rotting under drift, and its own failure modes. But because it's called a "guardrail," it gets treated like a config file: set once, never monitored, absent from the dashboard, missing from the on-call runbook. You would never ship your primary model without an SLO. Most teams ship their guardrail without even a health check.

Ask a coding agent to build a React component and it writes idiomatic, hook-shaped, accessibility-annotated code on the first try. Ask the same agent to use your in-house ORM — the one your platform team has maintained for six years, the one with excellent docs and a hundred internal consumers — and it hallucinates methods that don't exist, invents configuration options from some other library, and confidently ships code that compiles against an API it made up.

The difference isn't quality. Your ORM might be better-designed than half the open-source libraries the model handles flawlessly. The difference is training data. React has millions of public repositories behind it; your framework has zero. In the vocabulary of natural language processing, your internal framework is a low-resource language — and every consequence NLP researchers documented for low-resource languages now applies to your codebase.

There is a class of AI-generated bug that passes code review almost every time, and it isn't the hallucinated function or the fabricated package. It's the perfectly idiomatic code — idiomatic for the version of your stack that existed when the model's training data was frozen. The model writes a tailwind.config.js for a project running Tailwind v4, reaches for a class component lifecycle method in a hooks codebase, or calls an API that was deprecated three minor versions ago and removed in the one your lockfile actually pins. Nothing about the code looks wrong. It looks like code from a well-regarded tutorial. The tutorial is just from 2024.

Call it the training-cutoff bug class: defects that exist not because the model reasoned badly, but because the model's knowledge of your dependencies has a timestamp and your lockfile doesn't care. An ICSE 2025 study of seven code models across eight popular Python libraries found deprecated-API usage rates of 25–38% in plausible completions — and when the surrounding code already contained outdated patterns, that rate climbed to 70–90%. These aren't rare edge cases. They're the default failure mode of asking a frozen artifact to write for a moving target.

Rename a database column and watch what happens. The compiler catches every query builder. The ORM migration catches the model class. The type checker catches the API serializer. Integration tests catch the two services that read the field over the wire. Every consumer of that column gets flagged — except one. Your system prompt, which contains a carefully formatted description of the table "so the model understands the data," keeps confidently describing a column that no longer exists. No compiler error. No failing test. No deprecation warning. Just a model that starts generating queries against a schema from three sprints ago, and a dashboard that slowly fills with malformed SQL.

Prompts have foreign keys. They reference database schemas, tool signatures, enum values, API shapes, and few-shot examples copied from production data — and none of those references participate in refactoring. Every fact you paste into a prompt is a join against a table that your tooling doesn't know exists. When the referenced artifact changes, nothing cascades. The prompt just rots in place.

Your vault does everything right. Secrets are encrypted at rest, access is logged, rotation is automated, and nothing ever touches disk in plaintext. Then your agent calls a debugging tool, the tool prints an environment dump to stdout, the framework helpfully feeds stdout into the model's context — and your database credential is now part of a prompt. From that moment, the vault's guarantees are fiction. The credential exists in the trace your observability platform captured, in the prompt cache your provider keyed on that prefix, in the memory store your agent writes to between sessions, in the eval fixture someone snapshotted from production traffic, and in the provider's retention logs. Five persistence layers, none of which your secrets manager knows exist.

This is not a hypothetical. A large-scale study of over 17,000 published agent skills found that information exposure through logging accounted for 73.5% of all credential security issues — dwarfing hardcoded secrets at 18.2% — precisely because agent frameworks capture console output directly into the LLM context window. The old failure mode was a developer committing a key to GitHub. The new one is a tool response committing a key to a context window, and the context window has no git revert.

Six weeks after launch, your quality dashboard starts sagging. Thumbs-down rates creep up, task completion drifts down, and the on-call channel fills with screenshots of bad responses. The team does what teams do: they diff the prompts (unchanged), check the model version (pinned), audit the retrieval index (fresh), and bisect the deploy history (nothing shipped). Everyone concludes the model provider silently degraded the model. The provider, of course, insists nothing changed.

Everyone is looking in the wrong place. Nothing in the system changed. The users did.

Launch-week metrics assume launch-week users. But people adapt to an AI product within weeks, and they adapt in ways that systematically break the assumptions baked into your prompts, your evals, and your launch benchmarks. Your model is frozen. Your users are not. The gap between them is a form of drift that most teams don't instrument for at all — and it produces the most confusing incident pattern in AI engineering: metric decay with no deploy.

Everyone worries about model collapse as an internet-scale problem: AI slop floods the web, the next generation of foundation models trains on it, and quality decays in a slow civilizational feedback loop. That framing is comforting because it makes collapse someone else's problem — a thing that happens to OpenAI and Anthropic, on a timescale of years, mitigated by armies of data-cleaning PhDs.

Here is the uncomfortable version: the same feedback loop is already running inside your company, and it converges much faster than the internet-scale one. Every logged completion that gets labeled as a "gold example," every model-written document that lands in your RAG corpus, every LLM-judged eval that promotes an LLM-generated answer — each is a small act of training on your own exhaust. You don't need nine generations of recursive pretraining to feel it. In a production system that mines its own logs for few-shot examples and fine-tuning data, generation two ships next quarter.

The most celebrated interaction pattern of the last two years — text that materializes word by word, as if the machine were thinking out loud — is, for a screen reader user, closer to noise than to language. Every token your model emits is a DOM mutation. Plug a naive aria-live region into that stream and the screen reader will try to announce each mutation as it lands, producing a stuttering, overlapping torrent that resets mid-sentence dozens of times per second. The feature that makes your product feel alive is the same feature that makes it unusable for the people who depend on assistive technology.

This is accessibility debt, and generative UIs accrue it faster than any interface pattern before them. The reason is structural: traditional web content is static and predictable, so you can reason about it once and ship. A generative interface changes on every interaction — one prompt returns a list, the next returns a table, the next streams 600 words of prose followed by a tool-call widget. There is no fixed DOM to audit, no stable tab order to verify, no single snapshot that represents "the page." The accessibility contract has to hold across an infinite space of generated outputs, and almost nobody is testing for that.