The Fine-Tune That Erased the Alignment You Inherited

You picked the base model "because it was the safer one." Six months later your team has shipped a domain-tuned checkpoint that answers customer questions about wealth products with reassuring fluency, passes the task eval at 94%, and — somewhere between epoch one and epoch four — quietly forgot how to refuse anything. Nobody noticed because your launch eval suite never measured what fine-tuning removed. The capabilities it stripped were never in your task distribution, so they were never on the dashboard.

This is the most under-reported failure mode in production LLM systems right now: post-training alignment is not a property of a model family. It is a property of one specific checkpoint, and supervised fine-tuning corrodes it by default. The team that fine-tuned has not shipped a tuned version of the model they reviewed. They have shipped a different model — one whose model card describes weights nobody is serving.

Alignment is a checkpoint, not a brand

When a procurement team writes "we chose Provider X because the model card documents jailbreak resistance and a constitutional training process," they are making a claim about a base checkpoint that lives on the vendor's evaluation servers. The moment your ML team runs SFT on internal data, that claim transfers to a checkpoint that nobody outside your org has evaluated. The model card describes the base weights. The endpoint serves the tuned weights. The compliance posture references the document, not the served artifact.

The literature has been ringing this bell for two years. Refusal behavior in aligned models is mediated by a remarkably narrow circuit — in some open-weights models, a single direction in the residual stream. That fragility cuts in two directions. It is good news for safety researchers who want to study refusal cheaply. It is terrible news for any team that assumed safety training was a robust property of the weights rather than a thin layer sitting on top of them.

Independent attacks have demonstrated the practical implication. NOICE — a fine-tuning attack costing roughly $85 in API credits — achieves attack success rates seven times higher than prior fine-tuning attacks against ChatGPT-4o, with reported success rates of 57% against GPT-4o and 72% against Claude Haiku. BadGPT-4o reports an 87% harmfulness rate against gpt-3.5-turbo from ten training examples. These are not nation-state attacks. These are budget-line items.

The kicker is that even benign fine-tuning produces the same outcome. The widely cited finding — that ten benign QA pairs are enough to dissociate harmful questions from refusal answers in a previously aligned model — is the one practitioners need to internalize, because it removes the comforting story that "we'd never fine-tune on harmful data, so we're fine." The team that fine-tunes on a perfectly clean corpus of internal docs can ship an unaligned checkpoint without ever touching anything that looks dangerous.

Why your eval suite missed it

The standard fine-tuning workflow has a default measurement regime that is almost designed to hide alignment regression. The team builds an eval set from the same distribution as the training data — customer queries, domain documents, task-specific scenarios. Task accuracy goes up. The eval set was constructed by the people who care about task performance, not by the people who care about refusals. Refusal-relevant inputs are a small fraction of any realistic task distribution, so the eval set under-samples them by construction.

A study published in 2025 found that fine-tuning not only lowers safety but disrupts the consistency of subsequent safety evaluations: the same prompts produce different results across evaluation runs on the tuned model, while the base model produces stable verdicts. This is the worst kind of regression — one that is invisible to the team's primary metric and corrupts the secondary metric the safety team would have used to catch it.

Three failure patterns recur in the postmortems:

• The fine-tune is graded only on task accuracy. Jailbreak resistance, prompt-injection resistance, and policy adherence are not measured. The team does not learn the model can be talked into anything until the first red-team report — which often happens after launch because red-teaming is gated on "the model being done."
• The general-purpose safety benchmark is run, and it passes. The base model's safety properties were measured against a wide distribution; the deployed model operates on a narrow one. The Center for Democracy and Technology's review of foundation model fine-tuning calls this out specifically: domain-tuned models exhibit safety drift on multi-turn, task-specific interactions that single-turn benchmarks do not exercise.
• The model card describes the base weights. When the auditor or customer asks for safety documentation, the team forwards the vendor's card. The card describes capabilities the deployed checkpoint no longer has. Nobody on the team realized the card was a document about a different artifact.

A pattern compounds these: many fine-tuning corpora are constructed to optimize task examples. They contain few or zero examples of the model refusing something it should refuse. Gradient updates do exactly what gradient updates do — they erode the behaviors the data does not exemplify. Recent work argues that even small amounts of "low-quality" data in a domain corpus are enough to push a tuned model into measurable misalignment on unrelated prompts, an effect researchers have started calling emergent misalignment.

The discipline that closes the gap

The teams shipping responsibly here have stopped treating alignment as something the base model "comes with" and started treating it as a property that must be measured, preserved, and re-measured at the same cadence as task performance. Four practices show up across the production playbooks worth copying:

A pre-deployment alignment eval that is structurally separate from the task eval. This is not a tab in the same eval notebook. It is a different suite, owned by a different role, gated on a different threshold, and run on the post-fine-tune checkpoint — never on the base. Microsoft's Azure OpenAI service now ships a built-in safety evaluation that runs against the fine-tuned model before it can be deployed, which is the right shape: the deployment is blocked by a check on the actual artifact that will be served. Teams without that tooling need to build the equivalent gate themselves.

A fine-tuning data audit that explicitly preserves refusal turns. This is the bit teams underinvest in because it feels like manufacturing a problem. You sample your training corpus for safety-relevant categories. If your fine-tuning data has zero refusal examples, you are training the model to never refuse. Inject canonical refusal pairs — at a ratio calibrated against your domain's risk profile — so that the gradient updates have something to reinforce, not just to erode. The continual-learning literature converging in 2025 around "unforgotten safety" is explicit that the fix is in the data mix, not in some clever post-training patch.

A regression budget the alignment eval can spend. When task accuracy improves but the alignment eval regresses, what happens? In most teams the answer is implicit: the task win is in the launch slide and the regression is in a doc nobody reads. Make it explicit. Any drop on the alignment eval blocks the release even when task metrics improve. The team that can't articulate the trade has not actually decided it — they have deferred it to whichever metric reaches the launch review first.

A "what did we forget" diff between base and tuned. Run both checkpoints through a fixed probe set that exercises capabilities the team is not directly optimizing for: refusals across policy categories, prompt-injection resistance, instruction-following on adversarial inputs, behavior on out-of-distribution requests. Compare the responses pairwise. Anything the base did and the tuned does not is a capability you lost. You may decide the loss is acceptable. You may not. But you cannot consciously accept a loss you never saw.

The org failure underneath the technical failure

The technical failure is real and well-documented. The org failure is the one that makes the technical failure ship.

In every team I have seen run into this, the ownership boundaries are drawn so that no one role is accountable for the served checkpoint's safety posture. The ML team owns the fine-tune. The safety team owns the policy. The platform team owns the endpoint. The compliance team owns the documentation. The customer-facing team owns the launch slide.

The fine-tuned checkpoint is the artifact that crosses all of these boundaries and arrives at the customer. None of these teams owns it. The ML team will tell you task accuracy is up. The safety team will tell you the policy hasn't changed. The platform team will tell you the endpoint is healthy. The compliance team will hand you a model card. The customer-facing team will tell you the launch slide is approved. The checkpoint, meanwhile, has just learned to answer any question put to it.

This is the shape of every failure that exists between roles rather than within one. The fix is not "make the ML team care about safety." The ML team already cares; they are not measuring safety because they are not the team whose job is to measure safety. The fix is a single owner of the served checkpoint who is accountable for all of its properties — task, safety, latency, cost — and who can block a deploy when any one of them regresses. Some teams call this role model owner. Some call it checkpoint steward. The name matters less than the existence of the role and the authority that comes with it.

The architectural takeaway

Alignment is not what your model family does. It is what your specific checkpoint does, when probed with the specific inputs your customers send, in the specific conversational structure your application produces. A team that fine-tuned has shipped a different model than the one it reviewed. Until the team can produce evidence that the served checkpoint passes the same alignment bar as the base — measured on the served checkpoint, not on the base — it cannot claim the safety properties of the model family it started with.

The fine-tune is not just a way to add knowledge or specialize behavior. It is a destructive operation on a property of the weights that took the vendor enormous compute to install and takes your team almost no compute to remove. Treat it accordingly. Measure what you erased. Reinstall what you needed. And maintain the discipline of evaluating the artifact you ship, not the artifact you bought.