778 posts tagged with "llm"

Everyone worries about model collapse as an internet-scale problem: AI slop floods the web, the next generation of foundation models trains on it, and quality decays in a slow civilizational feedback loop. That framing is comforting because it makes collapse someone else's problem — a thing that happens to OpenAI and Anthropic, on a timescale of years, mitigated by armies of data-cleaning PhDs.

Here is the uncomfortable version: the same feedback loop is already running inside your company, and it converges much faster than the internet-scale one. Every logged completion that gets labeled as a "gold example," every model-written document that lands in your RAG corpus, every LLM-judged eval that promotes an LLM-generated answer — each is a small act of training on your own exhaust. You don't need nine generations of recursive pretraining to feel it. In a production system that mines its own logs for few-shot examples and fine-tuning data, generation two ships next quarter.

Ask a large language model what time it is and you will get a confident answer that is almost certainly wrong. Not because the model is broken, but because there is no clock inside it. A transformer is a stateless text-completion engine: it maps tokens to tokens. Nowhere in that pipeline does a signal arrive that says "it is now 14:32 UTC." The current moment is not something the model perceives — it is something you have to hand it, every single turn, or it will invent one from the stale sediment of its training data.

This is the quiet failure that surfaces at the worst possible moments. Your agent believes it is Monday because the session opened on Monday, and it keeps believing that on Tuesday, on Wednesday, right up until it schedules a "tomorrow morning" reminder for a day that has already passed. It reasons about "the last 24 hours" of logs using a now that froze hours ago. It converts a meeting time across time zones and lands an hour off because it assumed the wrong side of a daylight-saving boundary. None of these look like hallucinations in the classic sense. The output is fluent, plausible, and internally consistent. It is just anchored to a moment that no longer exists.

A model gives a customer a wrong, expensive answer. You open the post-mortem, paste the exact prompt back into the exact same endpoint with temperature=0, and you get a different answer. Not a worse one, not a better one — a different one. The bug you are supposed to root-cause refuses to reproduce on demand, and the one knob everyone told you guarantees reproducibility just lied to your face.

This is the moment most teams discover that "set temperature to zero" is folklore, not an engineering control. Temperature zero changes how the model samples — it forces greedy decoding, always taking the highest-probability token. It says nothing about whether the probabilities themselves come out the same twice. And in a production serving stack, they frequently don't.

There is a specific kind of regret that only happens on AI teams. You spend a quarter building an elaborate workaround — a multi-step prompt chain, a custom reranker, a hand-tuned tool-routing layer — and ship it. It works. Then six weeks later a new model drops, does the whole thing natively in one call, and your quarter of work is now dead weight you have to rip out. The feature didn't fail. The floor moved.

This is the structural problem with planning AI features in 2026: the substrate you're building on is improving faster than your release cycle. From 2023 into mid-2025, frontier labs shipped on a roughly six-month cadence. By Q1 2026 that compressed to substantive releases roughly every four weeks, with stretches where five labs shipped inside a thirteen-day window. The thing you're standing on moves under you between planning and launch.

Ask any engineering leader whether they're locked into a model vendor and they'll point you at the abstraction. "We route everything through a gateway. Swapping providers is a config change." The endpoint is one line. The base URL is an environment variable. On paper, migration is a Tuesday afternoon.

Then they try it. They flip the config to a different model family, the integration tests stay green, and production quietly falls apart. The JSON that always parsed now arrives wrapped in a markdown fence. The classifier that hit 94% drops to the low eighties. A prompt that ran clean for a year starts refusing one request in twenty for reasons no one can reproduce. The endpoint swapped in seconds. The behavior did not come with it.

This is the lock-in nobody budgets for. It doesn't live in your contract or your SDK. It lives in your prompts — thousands of small accommodations your team made, one at a time, to the quirks of a single model family. You didn't choose a model. You married one, and the prenup is every prompt you've ever shipped.

A customer support transcript leaves a server in Frankfurt, gets concatenated into a prompt, and crosses the Atlantic to a GPU in Virginia. The model thinks for 800 milliseconds. A response comes back. From the user's perspective, nothing happened — the chat just worked. From your regulator's perspective, you transferred personal data to a third country, and you may not be able to name the legal basis for it.

This is the part of LLM adoption that demos hide. Prototypes call api.openai.com and ship. Then a procurement questionnaire from a German bank, a French hospital, or your own legal team asks a question that the prototype never had to answer: where does the inference happen, and who can compel access to it? "The provider is SOC 2 compliant" is the reflexive answer, and it is the wrong one — it answers a question about the provider's internal controls, not about which jurisdiction's courts can reach into your prompts.

A team I talked to recently spent three engineer-weeks shaving their average prompt from 4,000 tokens to 2,600. They were proud of it — a clean 35% reduction, real numbers, a nice graph in the deck. Then someone did the arithmetic the other direction. The savings came to roughly $1,800 a month. The three engineer-weeks they spent getting there cost something like $25,000 in fully-loaded salary. At that monthly run rate, the optimization pays for itself in about fourteen months — assuming the prompt never changes, the model never gets cheaper, and those engineers had nothing more valuable to build.

None of those assumptions held. The prompt changed twice in the next quarter. The model they were on dropped its input price by 40% on its own. And the feature those engineers didn't ship that month was the one the biggest customer had been asking about.

A clinical decision support tool has exactly one job: help a doctor reason about symptoms, medications, and treatment plans in the same vocabulary the doctor would use in their own notes. That means the agent has to talk about overdose thresholds, drug interactions, contraindications, and dosing by name. None of that is optional. It is the product.

So it is a special kind of failure when the model interrupts a consult to say it cannot provide medical advice without professional oversight — to a professional, mid-consult, about a patient in front of them. The doctor didn't change. The patient didn't change. The prompt didn't change. What changed was a safety filter the team doesn't own, tuned against a population the team isn't part of, updated on a schedule the team doesn't control.

This is the vendor moderation seam, and it is one of the least-discussed boundaries in applied AI. Most teams treat the provider's safety layer as a fixed property of the model, like its context window. It isn't. It's a policy — a living one — and right now it's your product's policy whether you wrote it or not.

Eighteen months ago, "we built our own retrieval pipeline" was a reasonable thing to say in an architecture review. You had a chunking strategy, an embedding model you'd benchmarked, a vector store you'd tuned, a re-ranker, and a context-packing heuristic that took three engineers a quarter to get right. That stack was a real piece of differentiated infrastructure. Today, the same capability is a single hosted tool call: upload files to a vector store, attach it to a request, and the provider parses, chunks, embeds, stores, retrieves, and re-ranks — all behind one API. The three-engineer-quarter is now a config object.

This is the uncomfortable pattern of building AI products right now. The line between what you build and what you buy is not fixed. It moves — and it moves in one direction. Every few months a provider ships a primitive that absorbs a layer you used to own: memory, retrieval, structured output, tool routing, even multi-step orchestration. The infra you were proud of last quarter becomes the thing a competitor gets for free this quarter, with better defaults and a lower latency floor because it runs inside the provider's own datacenter.

The instinct is to treat this as a threat to defend against. That's the wrong frame. The provider absorbing commodity infra is doing you a favor — it deletes maintenance you never wanted. The real question is whether the thing you chose to build sits above or below the rising waterline. Most teams never make that call explicitly. They build whatever the demo needed, and then discover a year later that 70% of their codebase is reimplementing a primitive the provider now offers, while the 30% that's actually defensible got starved of attention.

Most finance teams filed their first LLM bill in the wrong drawer. It arrived during the experimentation phase, when a handful of engineers were prototyping with an API key, and it looked like exactly what it was at the time: research and development. A few thousand dollars a month to figure out whether the technology worked. So it went into R&D, mentally and sometimes literally, and nobody thought harder about it.

Then the feature shipped, usage climbed, and the same line item that was a rounding error in Q1 became the fastest-growing cost on the cloud invoice by Q4. The problem was never the dollar amount. The problem was that the cost had quietly changed categories — from a fixed bet on building something to a variable cost of serving every customer who used it — and the mental model hadn't moved with it.

That misclassification is the single most expensive accounting mistake in AI products right now, and it's not really an accounting mistake at all. It's a forecasting one.

A RAG agent answers a customer's regulatory question with a tidy paragraph and a citation. The verification layer fetches the URL, sees a 200 OK, ticks the box, and ships. Six months later a compliance audit pulls the transcript, clicks the same link, and finds a page that now says the opposite of what the agent quoted. The URL is fine. The quote is fine in the transcript. The two no longer match. The customer's compliance officer asks whether the agent fabricated the quote, and the team cannot prove it didn't, because the only surviving evidence of what the URL used to say is the agent's own assertion of what it said.

This is not a hallucination in the usual sense. The model retrieved real content, faithfully extracted a real sentence, and emitted a real URL that still resolves. Every link-checker on earth would call this citation valid. The audit fails anyway, because the verification layer was measuring the wrong property. Reachability is not fidelity. A URL is a pointer to a mutable document under someone else's editorial control, and the moment the document changes, every transcript that quoted it becomes a hallucination report waiting to happen.

A user submits a question. Your model produces an answer that is confidently wrong in a very specific way: the format is perfect, the reasoning is well-structured, and a particular qualifier — one that does not apply to this question at all — appears in exactly the place a similar qualifier appeared in example three of your system prompt. Not a hallucination. Not a prompt injection. The model did precisely what the examples taught it to do, on a question those examples were never meant to cover.

This is the failure mode that few-shot prompting actively encourages and that most eval suites are structurally blind to. Your examples are not neutral demonstrations of "what good looks like." They are case law. The model selects the closest match by surface tokens and applies the precedent — including its constraints — to whatever case is in front of it.