63 posts tagged with "mlops"

A team swaps GPT-4o for a newer model on a Tuesday afternoon. By Thursday, support tickets are up 30%, but nobody can tell why — the new model is slightly shorter with responses, refuses some edge-case requests the old one handled, and formats dates differently in a way that breaks a downstream parser. The team reverts. Two sprints of work, gone.

This story plays out constantly. The problem isn't that the new model was worse — it may have been better on most things. The problem is that the team released it with the same process they'd use to ship a bug fix: merge, deploy, watch. That works for code. It fails for LLMs.

You get paged at 2am. Users are reporting garbage output. You SSH in, check logs, stare at traces — everything looks structurally fine. The model is responding. Latency is normal. But something is wrong with the answers. Then the question lands in your incident channel: "Which prompt version is actually running right now?"

If you can't answer that question in under thirty seconds, you have a prompt versioning problem.

Prompts are treated like configuration in most early-stage LLM projects. A product manager edits a string in a .env file, a developer pastes an updated instruction into a hardcoded constant, someone else pastes a slightly different version into a staging Slack channel. Eventually the versions diverge, and nobody has a complete picture of what's running where. The experimentation-phase casualness that got you to launch becomes a liability the moment you have real users.

Most teams reach for fine-tuning too early or too late. The ones who fine-tune too early burn weeks on a training pipeline before realizing a better system prompt would have solved the problem. The ones who wait too long run expensive 70B inferences on millions of repetitive tasks while accepting accuracy that a fine-tuned 7B model could have beaten—at a tenth of the cost.

The decision is not about which technique is "better." It's about matching the right tool to your specific constraints: data volume, latency budget, accuracy requirements, and how stable the task definition is. Here's how to think through it.