778 posts tagged with "llm"

Your CI pipeline is a fortress. Container images are signed and verified before deploy. Every npm and PyPI dependency resolves against a lockfile with pinned hashes. Commits require signed tags. And then, somewhere in your model-serving startup script, there's a line that downloads a multi-gigabyte blob from a model hub or an S3 bucket and loads it into memory — no signature check, no hash verification, no record of who produced it. The single artifact that most decides what your product actually does is the one artifact your supply-chain tooling has never heard of.

This isn't a hypothetical gap. Security researchers have pulled hundreds of malicious models off public hubs — models that execute attacker code the moment you load them, models crafted specifically to slip past the scanners the hubs run. The tooling to close the gap now exists: safe serialization formats, an industry signing specification, admission controllers that reject unsigned weights. Most teams just haven't noticed that "model file" belongs in the same mental category as "unaudited binary from the internet."

Somewhere in your codebase there is a config file that says something like model: small-and-cheap under the test profile and model: frontier under production. It felt responsible when someone added it — why burn frontier-model tokens on CI runs that fire twenty times a day? But that one line quietly repealed a rule your team has followed for fifteen years without thinking about it: the environment you test in should behave like the environment you ship to.

The twelve-factor methodology called this dev/prod parity, and we got so good at it that we stopped noticing it. Docker gave us bit-identical runtimes. Infrastructure-as-code gave us identical topology. Then we put a language model in the middle of the request path and reintroduced the exact gap we spent a decade closing — except this time the divergent component isn't a database version. It's the part of the system that makes the decisions.

Somewhere in your codebase there is a gateway that lets you swap claude-sonnet for gpt-5 by changing one string. Your architecture review praised it. Your CTO sleeps better because of it. And it is quietly costing you 90% cache discounts, schema-enforced outputs, and the reasoning-effort knobs that separate a great production model from a mediocre one.

That is the unadvertised price of the unified LLM API. Every abstraction layer that promises "swap providers in one line" delivers that promise by projecting every provider onto the subset of features they all share — and the features that fall outside that subset are precisely the ones where providers compete hardest. Prompt caching semantics, structured output enforcement, extended thinking budgets, server-side tool execution: these are the levers that determine your real cost and quality curve, and your gateway may be silently dropping them on the floor.

Pull up your token dashboard and ask one question about every workload on it: was a human waiting for this response? For most teams running agents in production, the honest answer is no for half the bill or more. Eval suites, embedding backfills, nightly report generation, bulk classification, overnight code migration, summarization of yesterday's tickets — none of it has a user staring at a spinner. Yet nearly all of it flows through the interactive endpoint, at full price, competing for the same capacity as the requests that actually are latency-critical.

Every major provider will run that deferrable work for half the cost. OpenAI's Batch API, Anthropic's Message Batches, and Gemini's batch mode all price asynchronous jobs at a flat 50% discount in exchange for a 24-hour completion window. The discount requires no negotiation, no committed spend, no engineering heroics. It requires only that you admit, in your architecture, that some work can wait — and most teams have never made that admission, because nobody made deferral a design decision.

We have seen this movie before. Spot instances offered 60–90% off cloud compute for years while most teams kept everything on on-demand, not because the savings were unreal but because using them forced an uncomfortable question: which of our workloads can tolerate interruption? Teams that answered it built checkpointing and cut their compute bills by more than half. Teams that didn't kept paying the "everything is urgent" tax. The batch tier is the same fork in the road, except the axis is latency tolerance instead of interruption tolerance — and agentic workloads, which burn 5–30x more tokens per task than chatbots, make the price of not choosing much steeper.

For fifty years, data pipelines enforced their contracts by dying. An upstream team renamed a column, the downstream parser threw, the job crashed, someone got paged at 2 a.m., and by morning the contract was either fixed or formally renegotiated. Nobody designed this as a governance mechanism. It just fell out of the fact that rigid code cannot process input it doesn't expect. The crash was the enforcement. The pager was the audit trail.

Then we put an LLM in the consumer seat, and the breakage stopped crashing.

A model reading a malformed record doesn't throw a parse error. It copes. A missing field becomes a plausible guess. A renamed field becomes a slightly wrong interpretation. A unit change — cents to dollars, UTC to local — becomes a confident answer that's off by a factor the model never mentions. The pipeline runs green end to end, the dashboards stay quiet, and the broken contract surfaces three weeks later as a diffuse quality complaint that nobody can bisect. We didn't remove the failure. We removed the signal.

Most enterprise LLM contracts get negotiated backwards. Legal spends three weeks on the limitation-of-liability section — which will almost certainly never be invoked — while the clause that determines whether your product survives next year gets waved through in the boilerplate: the vendor's right to deprecate the model you built on, with notice defined as "commercially reasonable efforts to inform you."

Ask anyone who had production traffic on a model that got a retirement date. The migration wasn't a legal event; it was an engineering fire drill — re-running eval suites, re-tuning prompts that silently behaved differently, and explaining to customers why the assistant's tone changed overnight. No indemnity clause covers that. But a deprecation-notice clause, negotiated up front, converts the fire drill into a scheduled project.

This is a checklist of the clauses that actually move risk for an AI buyer, roughly ordered by how often they bite — and, just as important, a calibration of which levers vendors will genuinely move on versus the ones where you're burning negotiation capital for nothing.

Ask an infrastructure team what happens if the primary database goes down and you will get a rehearsed answer: replicas, failover runbooks, RTO and RPO numbers someone signed off on. Ask the same team what happens if the model API goes down and you will usually get a shrug and a link to the provider's status page. That asymmetry made sense in 2023, when the LLM powered an experimental sidebar. It stopped making sense the day your support flow, your search ranking, your code review bot, and your onboarding assistant all started routing through one vendor's inference endpoint.

The model API is now a tier-0 dependency for a lot of products — revenue-critical, sitting in the request path next to the database — but most disaster-recovery plans still treat it like a nice-to-have integration. The result is a familiar incident shape: the provider degrades, every AI feature in the product throws the same spinner, on-call stares at a status page they can't influence, and nobody can answer the only question that matters: what is this product supposed to do right now?

Somewhere, right now, an AI assistant is explaining your product to a prospect. It is quoting a price you changed eighteen months ago, recommending an integration you sunset last quarter, and suggesting an API endpoint that returns 410 Gone. You will never see this conversation. No analytics event fires. No session recording exists. The prospect either believes the wrong answer and files a confused support ticket, or believes the wrong answer and quietly buys from the competitor the model mentioned in the same breath.

This is not a hypothetical future problem. AI referrals already account for meaningful traffic — up to 5–8% for some technology and e-commerce sites — and the answers behind those referrals are generated from whatever the model absorbed about you, whenever it absorbed it. Your marketing team spent a decade learning to monitor brand search, review sites, and social mentions. Almost nobody is monitoring the surface that is growing fastest: what the models say when someone asks about you.

Every latency playbook you own was written for systems where the work per request was roughly constant. A database lookup takes what it takes. An image resize scales with pixel count, which you know in advance. Even classic LLM completions had a knowable cost envelope: input tokens in, bounded output tokens out. Reasoning models quietly deleted that assumption. When the model decides at runtime how long to think — and it decides based on how hard the problem turns out to be — response time stops being a property of your infrastructure and becomes a property of the question.

The consequence shows up first in your percentiles. Teams running reasoning models in production report p99 latencies spiking three to five times above p50, not because a host got slow or a cache went cold, but because one request in a hundred happened to be genuinely difficult. Your autoscaler, your timeout policy, and your SLO dashboard were all tuned for a world where that spread meant something was broken. Now it means the system is working as designed — and every tool you have for managing the tail is pointed at the wrong cause.

Every security team has a mental model of typosquatting: an attacker registers requets and waits for someone to fat-finger requests. It works, but it's a scattershot bet on human clumsiness. Slopsquatting is worse, because the "typo" isn't random. Language models invent plausible-but-nonexistent package names in predictable, repeatable patterns — and attackers can query the same models you use, harvest the names they invent, and register exactly those packages on PyPI and npm. The hallucination becomes a preorder. Your coding agent, running with autonomous install rights, is the customer who picks it up.

This isn't hypothetical. The largest study of the phenomenon generated 2.23 million code samples across sixteen models and found that 19.7 percent of recommended packages didn't exist — 205,474 unique fabricated names. And when a security researcher registered one of the most commonly hallucinated Python packages as a harmless empty shell, it was downloaded more than 30,000 times in three months and ended up in the install instructions of a major tech company's open-source repository. The supply chain attack that vibe coding made possible is already running its proof of concept.

When a dependency breaks, your build fails. When a workaround becomes unnecessary, nothing happens at all. That asymmetry is why every production LLM system older than a year is carrying scaffolding it no longer needs — retry choreography, output-repair parsers, forced chain-of-thought, elaborate task decomposition, chunking heuristics — each one built as a compensation for a specific model's specific weakness, and each one silently outliving the weakness it compensated for.

The uncomfortable part is that this isn't a hygiene problem, like stale feature flags. Obsolete scaffolding doesn't just sit there costing you latency and tokens. In the worst case it actively constrains the new model to the old model's ceiling: your decomposition logic chops a task into six steps because the 2024 model couldn't hold the whole thing, and the 2026 model — which could have one-shotted it — now inherits six opportunities to lose context at the seams you built.

Pull up your request logs from the last time you hit sustained 429s. You will probably find something odd: the errors don't arrive as a steady stream. They arrive in waves — a burst of 429s, a quiet gap, a bigger burst, another gap. The provider's quota didn't change between waves. Your traffic didn't spike. What you are looking at is your own retry logic, synchronized against itself. Every client that failed at second zero computed the same backoff delay, slept the same duration, and woke up at the same instant to fail together again.

This is the thundering herd, and the punchline is that the standard fix — exponential backoff — does not prevent it. Deterministic exponential backoff organizes the herd. It takes a crowd of clients that failed at roughly the same moment and marches them forward in lockstep: everyone retries at 1 second, then everyone at 2, then everyone at 4. The load spikes get farther apart, but each spike is just as tall as the first. If the spike is what triggered your rate limit, you have built a metronome that re-triggers it forever.