135 posts tagged with "mlops"

There is a specific kind of outage that never shows up as an outage. The service stays green, latency is flat, error rates are zero, and search quietly starts returning garbage. This is what happens the moment you point a new embedding model at an index built by the old one. Nothing crashes. The results just stop making sense.

The reason is geometry. An embedding model doesn't assign fixed coordinates to a concept — it defines a space, and the same sentence lands in a completely different location depending on which model drew the map. A vector produced by last year's model and a query embedded by this year's model are not "close" or "far." They are measured against different rulers. Cosine similarity between them is a number, and the number is meaningless.

So when someone files a ticket titled "upgrade to the new embedding model," they have not filed a config change. They have filed a full data migration that happens to be disguised as a one-line diff. Treat it like a library bump and you ship the silent outage.

The most dangerous eval set is the one that still passes. A regression suite that goes red gets attention: someone opens the failing case, argues about it, fixes the bug or updates the expectation. A green suite gets trust. And trust is exactly what a rotted eval set does not deserve, because the score stayed green not because your system is good but because the test stopped resembling what your users actually do.

This is the quiet failure mode of AI evaluation. You build a golden dataset — a few hundred carefully labeled cases that represent the job your product does. It earns its keep for a quarter. Every deploy runs it, every score lands green, and everyone sleeps well. Meanwhile the product ships three new features, enterprise traffic climbs from ten percent of queries to forty-five, and users start phrasing requests in ways nobody on your team wrote down eighteen months ago. The eval set doesn't know about any of it. It keeps grading the model on a distribution that no longer exists.

The demo wowed the executive team in a week. Eighteen months later, it is still a demo. The model still answers questions in the sandbox, the slide deck still gets recycled into new pitches, and every quarter someone asks why it hasn't shipped. Nobody has a good answer, because the honest one is uncomfortable: the part that impressed everyone was the cheap part, and nobody scoped the expensive part.

This is pilot purgatory, and it is now the default outcome. MIT's NANDA initiative found that roughly 95% of enterprise generative AI pilots deliver no measurable impact on the P&L. IDC and Lenovo, counting a different thing — POCs that literally never ship — put the number at 88%: for every 33 proofs of concept an enterprise starts, four reach production. The share of companies abandoning most of their AI initiatives jumped from 17% in 2024 to 42% in 2025. These are not stories about weak models. They are stories about a graduation gap that nobody budgeted for.

The uncomfortable truth is that a working demo represents maybe ten percent of the work required to run the thing in production, and it is the ten percent that looks like a hundred. Everything that makes an AI system safe to leave running — evals, guardrails, observability, cost controls, security review, on-call, and an owner with a budget — is invisible in a demo and non-negotiable in production.

Your CI pipeline is a fortress. Container images are signed and verified before deploy. Every npm and PyPI dependency resolves against a lockfile with pinned hashes. Commits require signed tags. And then, somewhere in your model-serving startup script, there's a line that downloads a multi-gigabyte blob from a model hub or an S3 bucket and loads it into memory — no signature check, no hash verification, no record of who produced it. The single artifact that most decides what your product actually does is the one artifact your supply-chain tooling has never heard of.

This isn't a hypothetical gap. Security researchers have pulled hundreds of malicious models off public hubs — models that execute attacker code the moment you load them, models crafted specifically to slip past the scanners the hubs run. The tooling to close the gap now exists: safe serialization formats, an industry signing specification, admission controllers that reject unsigned weights. Most teams just haven't noticed that "model file" belongs in the same mental category as "unaudited binary from the internet."

The budget meeting for a fine-tuning project always prices the wrong thing. Teams estimate the data pipeline, the training runs, the eval passes — a one-time investment with a clear finish line. Then the model ships, the accuracy chart goes up and to the right, and everyone moves on. Six months later an email arrives: the base model your adapter is welded to has a retirement date. Nothing about your system changed. Everything about its foundation did.

This is the part nobody prices in: a fine-tune is not a product you finished. It is a fork of someone else's codebase, and every base-model release is an upstream rebase you didn't schedule. Anyone who has carried private patches against a fast-moving open-source project knows exactly how this story goes — the fork is cheap to create and expensive to keep.

Here is a postmortem pattern that is becoming a genre. The primary model was healthy all night. Latency was flat, token throughput normal, provider status page green. And yet every user request failed for forty minutes — because the safety classifier sitting in front of the model timed out, and the middleware wrapped that timeout in a generic exception, and the exception handler returned a refusal. Your model didn't go down. Your gate went down, and the gate was wired to fail closed by an engineer who never thought of it as a decision.

The uncomfortable truth is that most teams run a second machine-learning system in production without admitting it. The moderation classifier, the jailbreak detector, the PII scrubber, the topical filter — each one is a model, with its own latency distribution, its own error rates, its own training-data assumptions quietly rotting under drift, and its own failure modes. But because it's called a "guardrail," it gets treated like a config file: set once, never monitored, absent from the dashboard, missing from the on-call runbook. You would never ship your primary model without an SLO. Most teams ship their guardrail without even a health check.

A rollout team ships a new model behind a percentage flag. The flag bucket is computed as hash(user_id) % 100, the canary is buckets 0–4, the lift on per-user engagement is large and stable for two weeks, and the team ramps to 20%, then 50%, then global. The lift evaporates somewhere between 50% and global, and the post-mortem traces it back to the canary cohort. The treatment didn't move the metric. The canary arm was a different population.

The team thought it had been sampling users. It had been sampling IDs.

The incident timeline reads cleanly. At 9:02 your platform team pushed prompt-template@v38 to the config service. At 11:14 your dashboards showed everything green. At 16:51 someone in support flagged a spike in escalations. At 17:03 you opened the eval suite, found a regression score of 0.34, and rolled back. The post-mortem says "caught in eight hours, no customer harm beyond the 0.04% who saw it." Engineering leadership applauds the response time.

It is wrong. The regression was caught in zero hours. The eval suite running at 17:03 was the same eval suite running at 09:03. It had been pointed at v37 the entire time. The harness loaded the template from your config service at process startup, cached the rendered prompts as Python objects in module-level scope, and never reread the source. Your live traffic moved to v38 at 9am. Your eval moved at 17:03, when someone restarted the worker pool to "rerun the regression." Eight hours of customer interactions ran against a prompt that no eval had ever scored, while the eval kept grading a prompt that no production request was using.

A fine-tune is not a file. It is the closure of a pipeline over a training set, and the team that ships the file without the closure has built a production dependency whose source code is in someone else's head. The day that person leaves with two weeks of notice and a clean handoff document is the day your bus factor on a revenue feature drops to zero and nobody notices, because the weights are still in the registry and the registry tag is still stable and the model still serves traffic. The reckoning shows up later, in a routine base-model migration that should have taken a sprint and takes a quarter instead.

The pattern is consistent across teams I have watched run into it. An ML engineer spends six months iterating on a fine-tune — data curation, hyperparameter sweeps, behavioral patches evaluated by feel against a held-out set. The final adapter weights get pushed to the model registry with a tag. The training pipeline that produced those weights is a notebook on the engineer's laptop, with hard-coded paths and floating dependencies that resolved to whatever was the latest version on the day each cell was last executed. The team accepts the handoff at face value because the weights work and the eval scores are good and the registry tag is stable. Eighteen months later, the engineer departs. Six months after that, a base-model migration requires regenerating the adapter against an updated base, the notebook runs and produces weights that score three points lower and regress visibly on the hardest customer segment, and the team spends four months trying and failing to reproduce the original artifact.

A platform team I know spent two quarters building a model registry. It had everything the org chart asked for: a promotion workflow from dev to staging to prod, a CODEOWNERS-style approval matrix, lineage tracking, eval-score gates, a deprecation policy with a 30-day window, and a Backstage tile that showed which version of every model was live in which service. They cut a launch announcement, ran a brown bag, and added a row to the compliance binder.

Six months later, the highest-traffic agent in the company was running on a model card whose "owner" field still pointed at someone who had left, whose eval score was from a benchmark the team had since deprecated, and whose "approved by" name was the platform tech lead — who had never used that agent, never read its eval set, and had pressed approve at 11:43pm on a Thursday because the producer had pinged him in DMs saying the launch was tomorrow.

The registry was not broken. The promotion gates fired. The audit log was intact. Everything the launch announcement had promised was true. And the org had less real oversight of its production models than it had had eighteen months earlier, when the same decisions were made by an ML engineer reading the eval output by hand before pasting the model URI into a config file.

A RAG pipeline ships with a reranker score threshold of 0.4. Anything below gets dropped from the prompt. Six months in, a routine index rebuild swaps the embedding model for a newer checkpoint in the same family — a transparent upgrade, the change log says. Two days later answer relevance falls 6%. The team blames the LLM, runs a model bake-off, finds no candidate that recovers the loss, and spends a quarter chasing a regression that lives in none of the models they were comparing.

The regression lives in the gate. The reranker — untouched, same checkpoint, same weights — is now scoring a different candidate set. The new embeddings pull different chunks into the top-50, the reranker scores them lower on its own calibration, and the gate at 0.4 drops 37% more candidates than it did the week before. The number 0.4 didn't change. What 0.4 meant changed.

A thumbs-up button is the cheapest signal you will ever instrument. It is also one of the most dangerous, because nothing about it announces that it is reshaping the distribution your eval set is supposed to represent. The button is collected as a positive — the curation pipeline reads it as quality — and six months later the eval is dominated by examples chosen by a cohort that does not include the customers most likely to churn.

The failure rarely shows up as a regression. It shows up as a divergence: weekly eval trends up, the enterprise tier's NPS slides, and the team only diagnoses the gap when a churned account names the specific kind of question their team kept getting wrong. The eval set has no examples shaped like it. The signal you were optimizing was real. It was just measuring the wrong distribution.