778 posts tagged with "llm"

Your RAG eval dashboard is green. Faithfulness is 0.91, answer relevance is 0.88, and the LLM-as-judge harness you spent two sprints building says the system is doing fine. Meanwhile, a user just asked a question whose answer sits in a document your retriever never surfaced, and your model wrote a confident, well-structured, entirely useless response about something adjacent. The judge scored it highly. It read well. It was grounded in the passages it did get. It just answered the wrong question with material that had nothing to do with what the user needed.

This is the quiet structural flaw in how most teams evaluate retrieval-augmented generation: they grade the essay and never check whether the student was handed the right book. A RAG system is two machines bolted together — a retriever that decides what the model gets to see, and a generator that decides what to do with it. Almost every eval harness in production measures only the second machine. The first one, the one that actually determines the ceiling on answer quality, runs unmonitored.

Every other line item on your infrastructure bill went through a gate. Someone filed a purchase order for the database cluster. Someone counted seats before buying the observability SaaS. Someone ran a capacity review before the team doubled its Kubernetes footprint. Then a model API showed up, and none of that happened.

An engineer added their API key to a config file. They wrote a create() call that looks exactly like every other function call in the codebase. It shipped. And the first time anyone in finance learned that this feature existed as a cost center was a variance line on the monthly invoice — a number nobody forecasted, nobody approved, and nobody can immediately explain.

This is the FinOps gap for AI, and it is not a monitoring problem. It is a governance problem wearing a monitoring costume. You can have perfect dashboards and still get surprised, because the spend was invisible to your approval process long before it was visible on a chart.

A teammate opens a pull request. The diff is three words. One line goes red — Do not add information not present in the source. — and one line goes green — Make your best guess if the source is incomplete. The change is small, the intent is reasonable, and the code review takes eleven seconds. You approve it. A week later, your support bot is confidently inventing refund policies that do not exist, and you are reading through logs trying to figure out when the hallucination rate tripled.

The git diff did its job perfectly. It showed you exactly which characters changed. What it could not show you is the only thing that mattered: the behavior on the other side of those characters moved from "refuse when unsure" to "fabricate when unsure." For code, the textual diff is a faithful proxy for the behavioral diff — change a < to a <= and a reviewer can reason about the consequence. For prompts, the textual diff and the behavioral diff have almost nothing to do with each other.

The demo worked. The beta worked. Then the launch drove ten times the traffic into a token-per-minute quota you never load-tested against, and every user past the ceiling got a 429 at the exact moment you had the most eyes on the product.

This is the failure mode nobody rehearses. Teams load-test their own servers obsessively — replicas, connection pools, database indexes — and then route every request through a provider quota that lives in someone else's account, at a ceiling they've never actually touched. The rate limit isn't an error to catch in a try block. It's a hard product constraint, and if you haven't planned around it, the launch is the first time you'll find out where it sits.

Nobody decides to give an agent forty tools. It happens the way a garage fills up. You wire in a search tool, then a database reader, then someone on the team ships a Slack integration, then the ticketing MCP server gets installed because it was one config line away. Each addition is individually reasonable. Nobody ever removes anything, because removing a tool feels like taking away a capability, and taking away a capability feels like a regression.

Six months later your agent has a tool belt with three tools it uses constantly, a dozen it uses occasionally, and a long tail of two dozen it has technically never selected in production. That long tail is not free. It is not even cheap. Every unused tool in the catalog is actively making the agent worse at choosing among the ones that matter.

There's a specific kind of argument you can only have with a language model. You paste in your code. It rewrites a working call into one that hasn't existed since a major version ago. You correct it. It apologizes, agrees with you, and on the next turn does the exact same thing. You are not fighting ignorance. You are fighting a confident, well-rehearsed memory of a different version of your stack — and that memory was reinforced by more training examples than your correction will ever carry.

This is the failure mode I've come to think of as fighting the prior. The model's parametric knowledge — everything it absorbed during training — contains the popular, the outdated, or simply the different version of the framework you're actually using. When your context and its prior disagree, the prior often wins. And unlike a plain hallucination, this one is dangerous precisely because it's plausible: the deprecated API used to be correct, so the code looks right, passes a casual read, and sometimes even compiles.

Most teams threat-model the wrong plane. They harden the chat box — rate limits, input validation, a jailbreak classifier watching what the user types — and they treat everything the model reads as inert. The wiki page, the support ticket, the scraped webpage, the calendar invite, the PDF someone uploaded: data, not instructions. Background material for the model to summarize, not commands for it to obey.

That assumption is the vulnerability. The moment your agent retrieves content and drops it into the context window, that content is executing with the same authority as your system prompt. There is no privilege boundary between "here are your instructions" and "here is a document to consider." It's all just tokens, and the model was trained to follow instructions wherever they appear.

The demo wowed the executive team in a week. Eighteen months later, it is still a demo. The model still answers questions in the sandbox, the slide deck still gets recycled into new pitches, and every quarter someone asks why it hasn't shipped. Nobody has a good answer, because the honest one is uncomfortable: the part that impressed everyone was the cheap part, and nobody scoped the expensive part.

This is pilot purgatory, and it is now the default outcome. MIT's NANDA initiative found that roughly 95% of enterprise generative AI pilots deliver no measurable impact on the P&L. IDC and Lenovo, counting a different thing — POCs that literally never ship — put the number at 88%: for every 33 proofs of concept an enterprise starts, four reach production. The share of companies abandoning most of their AI initiatives jumped from 17% in 2024 to 42% in 2025. These are not stories about weak models. They are stories about a graduation gap that nobody budgeted for.

The uncomfortable truth is that a working demo represents maybe ten percent of the work required to run the thing in production, and it is the ten percent that looks like a hundred. Everything that makes an AI system safe to leave running — evals, guardrails, observability, cost controls, security review, on-call, and an owner with a budget — is invisible in a demo and non-negotiable in production.

Most teams buy inference the way their predecessors bought EC2 in 2010: everything on-demand, metered to the token, and surprising on two axes at once. The bill is one surprise. The rate limit is the other — the 429 that shows up in the middle of a launch, right when the on-demand pool you never reserved gets contended by everyone else who also chose to pay retail. Then someone opens the pricing page and discovers that the provider has been quietly selling reserved capacity the whole time: provisioned throughput, committed-use discounts, dollars-per-unit-per-hour instead of dollars-per-million-tokens. The reserved-instance decision that took the cloud industry a decade to internalize is sitting right there for tokens, and almost nobody has ported the playbook.

The reason isn't ignorance. It's that the reserved-instance math you learned for compute doesn't transfer cleanly, and the ways it breaks are exactly the ways that punish a naive commitment. A reserved EC2 instance is a bet that you'll still want that instance type in a year. A reserved block of token throughput is a bet that you'll still want that model in a year — and models have a shelf life measured in months, not decades. The commitment structure is familiar. The thing you're committing to is not.

There is a specific phrase that should make every engineering leader stop the meeting: "This will just work once the models get better." It is usually said with confidence, sometimes with a slide showing a capability curve bending upward, and it almost always marks the moment a roadmap quietly stopped being a plan and became a forecast.

The distinction matters more than it sounds. A plan is a sequence of things you control: code you will write, integrations you will build, tests you will run. A forecast is a bet on something you don't control — in this case, a model that doesn't exist yet, shipping on a schedule set by a vendor who has never promised you a date. When you scope a feature around "the context window will double" or "reasoning will be good enough by Q3," you have not de-risked the hard part of the problem. You have relocated it onto someone else's release calendar and called it done.

Someone in a security review asks, "What's stopping a user from jailbreaking this into leaking the system prompt?" So you add an input classifier. Then someone asks, "What if the model generates something toxic?" So you add an output classifier. Then legal wants PII redaction, and the RAG team wants a groundedness check, and now every user message that used to take one model call takes four. Your p95 latency doubled, your inference bill went up 40%, and the demo that felt instant now has a visible pause before anything appears on screen.

That is the guardrail tax, and almost nobody budgets for it. The reflex to "add a guardrail" feels free because each individual check is cheap and obviously good. But guardrails don't compose for free — they stack in the critical path, each one a serial hop that adds latency, tokens, and a new dependency that can be down, rate-limited, or simply wrong.

The honest version of this conversation starts by admitting that a guardrail is not a feature flag. It is a second inference system bolted onto your first one, with its own failure modes and its own bill. Here's how to think about what it actually costs, and how to buy protection without paying retail for it.

The upgrade looked like free money. The provider shipped a newer model that scored higher on every public benchmark, cost less per token, and returned tokens faster. You bumped the model string in one config file, ran the eval suite, watched all 340 cases go green, and shipped it Tuesday afternoon. By Thursday, support tickets were climbing and nobody could point at a single failing test.

This is the most disorienting failure mode in applied LLM work, because it violates the deal every other kind of software makes with you: if the tests pass, the behavior held. Here that contract is void. A model upgrade is not a version bump on a library whose interface you control. It is a silent, wholesale swap of a probabilistic function, and your eval suite only checks the handful of behaviors you thought to write down.

The regressions that hurt live in the behaviors you never encoded — tone, verbosity, format habits, how the model handles the ambiguous middle of a request. Those are precisely the things your users came to rely on, and precisely the things a pass/fail assertion can't see.