Retiring an Embedding Model: Re-indexing Millions of Vectors Without Taking Search Down

There is a specific kind of outage that never shows up as an outage. The service stays green, latency is flat, error rates are zero, and search quietly starts returning garbage. This is what happens the moment you point a new embedding model at an index built by the old one. Nothing crashes. The results just stop making sense.

The reason is geometry. An embedding model doesn't assign fixed coordinates to a concept — it defines a space, and the same sentence lands in a completely different location depending on which model drew the map. A vector produced by last year's model and a query embedded by this year's model are not "close" or "far." They are measured against different rulers. Cosine similarity between them is a number, and the number is meaningless.

So when someone files a ticket titled "upgrade to the new embedding model," they have not filed a config change. They have filed a full data migration that happens to be disguised as a one-line diff. Treat it like a library bump and you ship the silent outage.

Why a Model Swap Is a Migration, Not a Version Bump

The intuition that trips people up is that embeddings feel like a setting. You changed text-embedding-old to text-embedding-new in one place, the code compiles, the API accepts the call. Every instinct says this is the same class of change as swapping a JSON parser.

It isn't, because the old model's output is durably stored in your database. Every one of those hundreds of millions of vectors is a frozen artifact of a model you're trying to retire. The query path moved; the data didn't. Until the entire corpus is re-embedded with the new model, your index is a haunted house — full of coordinates from a map nobody uses anymore.

This is also why you can't do a partial or lazy migration where you "re-embed on read." Search doesn't read one document at a time; it compares a query against the whole space at once. A single old vector in a top-k result isn't a small error — it's an apples-to-rulers comparison that can outrank genuinely relevant new vectors. The space has to be internally consistent, which means the migration is all-or-nothing at the level of what a query sees, even if the re-embedding itself happens gradually in the background.

Once you accept that framing, the rest of the playbook writes itself. You don't edit a value in place. You build a second index, fill it, prove it's better, and cut over atomically — the same discipline you'd apply to any schema migration on a table you can't afford to lock.

The Dual-Index Cutover

The workhorse pattern is blue-green for vectors, and it has four moving parts.

Build the new index alongside the old one. Provision a second collection (or a second named vector inside the same collection, if your database supports it) configured for the new model's dimensions and distance metric. The old index keeps serving every query. Nothing user-facing has changed yet.

Turn on dual writes. From this moment, every new document and every update embeds through both models and writes to both indexes. This is the step people skip, and skipping it guarantees a broken migration: while you spend days re-embedding the backlog, live traffic keeps mutating the corpus. Without dual writes, the new index is stale the instant you finish backfilling it — missing every document created during the migration window. Dual writes freeze the delta at zero.

Backfill the history in the background. Scroll through the existing corpus in batches, re-embed each record with the new model, and upsert it into the new index. This is the long pole — hours to days depending on scale — but it runs at low priority, decoupled from the serving path. A useful discipline is insert-only backfill: never overwrite a record that dual writes already populated with a fresher vector, or you'll clobber new data with stale re-embeds.

Flip reads atomically. When the new index is fully populated and validated, switch the query path in one motion. The cleanest mechanism is an alias: your application queries search-current, which is a pointer, and you repoint it from the old index to the new one. The swap is instantaneous, there's no in-between state where half your traffic sees each index, and rollback is the same operation in reverse.

One caveat worth internalizing: the clean version of this pattern assumes inserts and upserts. If your workload does hard deletes or partial updates mid-migration, dual writes need extra logic to keep both indexes consistent, or you pause those operations for the duration. Decide this before you start, not when you notice the counts diverging.

Loading…