842 posts tagged with "ai-engineering"

Every team shipping an LLM feature eventually holds the same meeting. Someone asks whether the new model — or the new prompt, or the new retrieval tweak — is good enough to ship. The senior engineer who spent the weekend testing says it feels sharper. The PM says a customer complained about exactly this last week. The skeptic on the team pulls up a transcript where the old version was clearly better. Forty-five minutes later, nobody has changed their mind, and the decision gets made by whoever talks last or outranks the room.

That meeting is a symptom. It recurs because the team is trying to settle an empirical question — did this change make the system better or worse? — with anecdotes, and anecdotes don't converge. You can stack them all day. The reason the debate never ends is that there is no shared instrument that everyone agrees to be bound by. The meeting that should have been an eval is the meeting where you discover you're missing one.

The quarter your team adopted coding agents, two things happened that nobody put on the same slide. Throughput went up — more pull requests, more merged code, more tickets closed. And your three most senior engineers got slower. Not lazy-slower. Drowning-slower. Their own commits dried up, their calendars filled with review, and the one-on-one refrain shifted from "here's what I shipped" to "I spent the week unblocking everyone else."

This is the seniority inversion. The people whose time was supposed to get more leveraged by AI are the ones who got buried by it. And it isn't a motivation problem or a tooling gap. It's a structural consequence of what agents actually do to the shape of work: they make generation cheap and verification expensive, and verification is the one task you can't hand to a junior or to another agent.

A coding agent on your team opens a pull request that calls parseUserToken() four times across two files. The function does not exist in the repository, has not existed for nineteen days, and was replaced by decodeSessionClaim() in a commit your engineers all remember reviewing. The agent did not invent the name. It read the name from its semantic index — a vector store rebuilt from a working copy that was twenty-one days behind main. The agent's edit step, by contrast, ran git pull at session start and operated on fresh code. Two views of the same repository, three weeks apart, and the agent confidently bridged them with code that does not compile against anything real.

This is the failure mode that doesn't announce itself. The agent ran. The tests appeared to pass. The PR landed. The first reviewer noticed only because a stubbed-out function shared a name with an unrelated helper and tripped the linter. By then the agent had spent a full sprint writing against a phantom version of the codebase, and no one on the team — including the agent — had any signal that something was wrong.

Your residency audit traced every outbound request from the tenant's traffic, watched it terminate on a hostname in Frankfurt, and signed off. The audit was correct about everything it measured. It was also looking at the wrong layer. The request went to the EU. The bytes that satisfied the request — the cached prefix the provider hashed and pulled from the nearest available node — lived in us-east-1. Your regional endpoint promised you a destination. The cache promised nothing, because the cache was a different product, governed by a different SLA, designed for cost rather than for compliance.

The customer's auditor caught it. Not yours. A different vendor's incident report mentioned that prompt cache placement was decoupled from inference region, and the customer's GRC team asked the obvious follow-up question: where do our prefixes go? The contract amendment to close the gap took ninety days. The renewal got suspended. The team that wrote the integration had done nothing wrong by the documentation they were handed.

A security-conscious customer pulled their tenant's audit export, opened the JSON, and read the verbatim refusal policy, retrieval pipeline structure, and a handful of internal product identifiers from a field called llm.request.system. No exploit. No prompt injection. No jailbreak. Just a log field your platform team added six months earlier so engineers could correlate prompt versions with incidents — surfaced through a feed your enterprise team had separately opened to tenants for SOC 2 reasons.

The disclosure happened during a normal Wednesday afternoon. Your security team got paged by the customer, not by an alert. The incident timeline doesn't show a deploy on the day of the leak — the misconfiguration shipped on the day the audit feed expanded its field allowlist, which was a different team, a different sprint, and a different ticket. Both reviewers signed off on what they were looking at. Neither was looking at the composition.

Your judge prompt scored four axes on a 1–5 scale: helpfulness, clarity, empathy, accuracy. You averaged them. Your weekly dashboard trended up for six months. Your support queue trended in the opposite direction the whole time, and nobody connected the two until a customer escalation forced a manual audit and you discovered the model had learned a posture your product could not afford.

The posture was hedged wrongness. A softened wrong answer — "there are a few ways to think about this, one common view is X" where X is incorrect — scored 4.2 on your composite. A blunt correct answer — "no, X is wrong, the answer is Y" — scored 3.8. The judge wasn't broken. The rubric wasn't obviously broken either. Each axis was defensible in isolation. The aggregation was the bug.

The eval dashboard had been climbing for three sprints. Quality up six points on the hard slice, up nine on the regression slice, up twelve on the slice the support team had hand-curated from last quarter's worst tickets. The team shipped a model promotion off the back of it. Two days later, a customer asked a question that looked nothing like anything in the eval set, and the answer was worse than what they had been getting six months ago.

The forensic was quick once someone thought to run it. The prompt engineers had been working out of the same repo as the eval team. They had found the curated examples — the painstaking ones, the ones where someone had argued for an hour about the correct phrasing of the ideal answer — and over a few sprints they had copy-pasted the strongest of them as few-shot demonstrations into the production system prompt. The dashboard kept going up because the model was being graded on inputs it had seen verbatim at inference time. Nobody flagged it. Nobody owned the boundary between "the examples we measure quality against" and "the examples we ship in the prompt." Both teams were doing exactly the job they had been hired to do.

A user submits a question. Your model produces an answer that is confidently wrong in a very specific way: the format is perfect, the reasoning is well-structured, and a particular qualifier — one that does not apply to this question at all — appears in exactly the place a similar qualifier appeared in example three of your system prompt. Not a hallucination. Not a prompt injection. The model did precisely what the examples taught it to do, on a question those examples were never meant to cover.

This is the failure mode that few-shot prompting actively encourages and that most eval suites are structurally blind to. Your examples are not neutral demonstrations of "what good looks like." They are case law. The model selects the closest match by surface tokens and applies the precedent — including its constraints — to whatever case is in front of it.

Your application opens a long conversation with a forty-thousand-token system prompt and a full tool inventory. Turn 1 pays the prefix at write rates and the provider's KV cache warms up. Turn 2 arrives ninety seconds later. You assume it's a cache hit. Sometimes it is. Sometimes the same forty thousand tokens land on your invoice again at uncached prices, and nothing in your code changed between turn 1 and turn 2.

The thing that changed was somebody else's traffic. The KV cache is shared infrastructure. Your tenant was colocated on a serving node that, in the ninety seconds between your two turns, took on enough other tenants to evict your prefix from memory. The provider's dashboard will describe this as "cache pressure." Your finance team will describe it as a line item that doubled. Both descriptions are accurate. Neither is in your code.

The security review said the agent acts "on behalf of" the user. The OAuth token said something else, and the audit log agreed with the token.

A small distinction in language did a lot of architectural work nobody noticed. "On behalf of" is the language a security review reaches for when it wants to capture an arrangement where the agent is a delegate, recognizable as a delegate, and constrained by being a delegate. "Act as" is the runtime behavior when the agent holds a token indistinguishable from the user's own and is therefore the user as far as every downstream system can tell. These two phrases describe completely different threat models. A typical enterprise OAuth integration ships the second one and prices it as the first.

A user opens yesterday's conversation. Next to their support agent's reply sits a broken-image placeholder where their uploaded receipt used to be. The reply confidently quotes "the charge of $47.32 dated March 14 at the merchant Coffee Tribunal." The user has no way to check whether that quote is accurate, because the evidence the model worked from is now a 403 from your object store. They file a hallucination ticket. Your eval suite did not catch it because the model was, at the time of the call, exactly right.

This is a story about retention mismatch, not about model quality. Your transcript outlived its grounding. The grounding was a presigned URL with a fifteen-minute clock, and the claim about the grounding is text that will sit in your database for years. When the asset clock and the claim clock run at different speeds, every correctly-grounded multimodal answer eventually looks like fabrication to whoever revisits it.

The eval set is a file. It is also, secretly, a theory of what the AI feature is for. The two are not the same thing, and the team that confuses them has built a quality gate whose calibration depends on a single human's working memory. When that human leaves, the file stays and the theory walks out the door.

This is the failure mode you don't see in the org chart. You scoped a prompt engineering role. You hired someone good. They shipped the v1 prompts, looked at the thin benchmark, and rewrote it into something rich — a taxonomy of failure modes, weights per category, a labeling rubric that disambiguates edge cases. The eval set became the contract for "is this model good enough to ship." Six quarters later you discover that the contract is unreadable by anyone except the person who wrote it.