639 posts tagged with "llm"

Your best AI engineer can recite Claude's XML formatting preferences from memory. They know that Claude Opus refuses to generalize implicit instructions, that few-shot examples actually hurt performance on o1-series models, and that Azure OpenAI imposes an extra 8–12 seconds of latency versus the direct API in some regions. This expertise took months to accumulate. It also represents one of the most underappreciated risks in AI engineering today.

When a provider deprecates a model or silently shifts behavior, that knowledge doesn't transfer. It vanishes. And teams that built their systems — and their institutional competence — around a single model family often discover this the hard way.

When you write a system prompt, you have a user in mind. Maybe it's the competent professional asking a focused question in clear English. Maybe it's someone who sends a short, well-scoped request that fits neatly inside your prompt's assumptions. You test against examples that feel representative, tune until the outputs look good, and ship.

Then you see production traffic.

The real population of queries your system prompt must handle is not the median case you designed for. It's a distribution — some narrow, many diffuse — with a long tail of edge cases that expose every assumption baked into your instructions. For most production systems, somewhere between 15% and 30% of real queries fall into categories the prompt handles poorly. The unsettling part: most of these failures are silent. Your system returns a 200, the user gets an answer that looks plausible, and the failure never surfaces in your logs.

When two microservices diverge in their API assumptions, your integration tests catch it before production does. When two agents diverge in their prompt assumptions, you find out when a customer gets contradictory answers—or when a cascading failure takes down the entire pipeline. Multi-agent AI systems fail at rates of 41–87% in production. More than a third of those failures aren't model quality problems; they're coordination breakdowns: one agent changed how it formats output, another still expects the old schema, and nobody has a test for that.

The underlying problem is that agents communicate through implicit contracts. A research agent agrees—informally, in someone's mental model—to return results as a JSON object with a sources array. The orchestrating agent depends on that shape. Nobody writes this down. Nobody tests it. Six weeks later the research agent's prompt is refined to return a ranked list instead, and the orchestrator silently drops half its inputs.

Most teams discover their system prompt has a weight problem the same way — a cost review, a latency spike, or an engineer who finally reads the thing end to end. What they find is typically a 2,000-token document that grew organically over six months, with three versions of "be concise" scattered across different sections, instructions that reference a product workflow that was deprecated in February, and a dozen rules that the model visibly ignores on every run. The prompt is large. Most of it isn't doing anything.

This is the prompt credit assignment problem: figuring out which instructions in a multi-thousand-token system prompt actually drive model behavior, and which are just dead weight that burns tokens and dilutes attention. The bad news is that most teams skip this entirely — they add instructions when behavior breaks and never subtract. The good news is there is a repeatable engineering discipline for it.

In 2023, "prompt engineer" was one of the most searched job titles in tech. LinkedIn was full of engineers rebranding their profile summaries. Job postings promised six-figure salaries for people who knew how to coax GPT-4 into behaving. What the job descriptions didn't say was that many of the skills they listed were already on borrowed time — and that the engineers who noticed the difference between durable and decaying skills would end up in very different places by 2026.

The prompt engineering career trap is not that the field went away. It's that it changed so fast that skills built over 12 months became liabilities by the 18-month mark. Engineers who invested heavily in the wrong layer and ignored the right one found themselves holding expertise in things the next model revision made irrelevant.

There is a certain kind of engineering debt that never shows up in your metrics. You accumulate it every time someone adds a sentence to the system prompt to fix a one-off complaint — a phrase like "never discuss competitor products" or "always respond in a formal tone" — and then nobody ever verifies whether the model actually enforces it. Over months, the prompt grows to 800 tokens. It sounds authoritative. It contains multitudes. And maybe a third of it does nothing.

Prompt mutation testing is the practice of finding out which third. The technique borrows its name from classical mutation testing in software engineering: systematically introduce small, deliberate faults into your code to determine whether your test suite would actually catch them. Here, you introduce deliberate perturbations into your system prompt — remove a clause, contradict a rule, substitute a critical keyword with a near-synonym — and measure how much the model's output actually changes. Instructions that survive perturbation without affecting behavior are decorative. Instructions that break things when touched are load-bearing.

A team at a product company built a brainstorming assistant for their marketing department. They added RAG over their document corpus — campaign briefs, brand guidelines, competitor analyses — figuring the richer context would produce better ideas. Usage dropped within three weeks. The qualitative feedback: outputs felt "too safe," "too predictable," "like it just remixed our existing stuff." They removed retrieval from the brainstorming feature. Ideas improved. Engagement recovered.

This pattern repeats more often than practitioners admit. Retrieval-augmented generation has become the default architecture for grounding LLM outputs in facts, and for factual tasks it earns that default. But for generative tasks — ideation, creative writing, novel solution generation — adding a retrieval layer can silently cap the ceiling of what your model produces. Not because retrieval is broken, but because it's working exactly as designed.

When legal asks "why did the AI deny this loan application?", your chain-of-thought trace is the wrong answer. It doesn't matter that you have 1,200 tokens of step-by-step reasoning. What they need is a sentence that holds up in a deposition — and right now, most engineering teams have no idea how to produce it.

This is the stakeholder explanation gap: the distance between what engineers understand about model behavior and what regulators, executives, and legal teams need to do their jobs. Closing it requires a distinct architectural layer — one that most production AI systems never build.

Most teams treat their system prompts the way early web developers treated CSS: paste something that works, modify it carefully to not break anything, commit it to a config file, and hope nobody touches it. Then a new team member "cleans it up," a model upgrade subtly changes behavior, and three weeks later a user files a bug that nobody can reproduce because nobody knows what the prompt actually said last Tuesday.

This isn't a workflow problem. It's a category error. System prompts aren't configuration — they're software interfaces. And until engineering teams treat them as such, the LLM features they build will remain fragile, hard to debug, and impossible to scale.

A surprising number of AI teams default to extended thinking on every query once they gain access to an o3-class or Claude extended thinking model. The logic seems obvious: smarter reasoning equals better outputs, so why not always enable it? The problem is that this reasoning fails to account for a basic fact of how test-time compute scaling works in practice. Extended thinking dramatically improves performance on a specific class of tasks, degrades quality on others, and can inflate your inference costs by 5–30x across the board. The teams getting the most value from these models treat the reasoning budget as an explicit decision — one with the same weight as model selection or prompt engineering.

This post lays out the task taxonomy, the cost structure, and the routing decision framework that distinguishes teams who use thinking budgets strategically from teams who are just paying a premium for an illusion of quality.

A team ships a customer-facing AI assistant. They price it at $49/month per seat, targeting 70% gross margins based on a spreadsheet that assumed "average 500 tokens per query." Three months later, finance flags that their heaviest users are consuming 15,000 tokens per session. The pricing model collapses not because the feature failed, but because the product team priced something they didn't yet understand.

This isn't a failure of forecasting. It's a structural problem: the cost basis of an LLM-powered product is fundamentally unlike anything traditional SaaS pricing was designed to handle. Every API call has unpredictable and material token cost. The inputs vary wildly by user, task, and time of day. The outputs compound in ways that only show up weeks later on your cloud bill. And once you layer in agentic patterns — tool calls, multi-turn reasoning, subagent orchestration — a single user interaction can cost $0.02 or $20 depending on what the model decides to do.

Most teams building AI agents discover the same problem on their fifth sprint: the agent can't reliably pick the right tool anymore. At ten tools, it mostly works. At twenty, accuracy starts to slip. At fifty, you're watching the agent call search_documents when it should call update_record, and the logs offer no explanation. The usual reaction is to tweak the tool descriptions — add more context, be more explicit, rewrite the examples. This occasionally helps. But it misses the root cause: flat embedding retrieval is architecturally wrong for large tool inventories, and better descriptions cannot fix an architectural problem.

Tool selection is retrieval, and retrieval has known scaling limits. Understanding those limits — and the structured metadata patterns that work around them — is what separates agent systems that hold up in production from ones that require constant babysitting.