Zero-Downtime Embedding Model Migration: How to Re-Index a Production RAG System Safely

A new embedding model ships with better multilingual recall and half the price per token. You change one line, model="old-embedder" to model="new-embedder", redeploy, and retrieval quality quietly collapses. Queries are now embedded by the new model, but every stored document vector still comes from the old one. The two live in different vector spaces, so cosine similarity between them is noise.

This is the core constraint: embeddings from different models are not comparable, even when the dimensions match. Migrating means re-embedding your whole corpus and switching query-time and index-time models together. Done carelessly, that is an outage. Done well, nobody notices. This post walks through a dual-index migration you can run on any vector store.

Why you cannot just swap the model

Three things break when the embedder changes:

Query and document vectors drift apart. Queries use the new model, documents the old one. Retrieval returns plausible-looking garbage rather than errors, so monitoring based on exceptions will not catch it.
Dimensions and normalization may differ. A 1536-dim index will reject a 1024-dim vector outright, which is the lucky case. Same-dimension models are the dangerous ones.
Score distributions shift. Similarity thresholds, rerank cutoffs and "no good match" logic tuned on the old model's scores are now miscalibrated. A threshold of 0.78 can mean very different things across models.

Every one of these is invisible in a unit test. All of them show up as "the answers got worse" a week later.

The migration pattern: build beside, compare, cut over

The safe approach never mutates the live index. You build a second index next to it and move traffic only after evidence says it is better.

Phase	What happens	Reversible?
1. Provision	Create a new collection/index, named with the model version	Yes, delete it
2. Backfill	Re-embed the corpus into the new index in batches	Yes
3. Dual-write	New documents and updates go to both indexes	Yes
4. Shadow eval	Run real queries against both, compare offline	Yes
5. Canary	Serve a small % of traffic from the new index	Yes, flip flag
6. Cutover	New index becomes primary	Yes, until old index is deleted
7. Decommission	Delete the old index after a soak period	No

Only the last step is irreversible, and you control when it happens.

Step 1: Name indexes by model, not by purpose

Stop calling your collection documents. Include the embedder and version so two generations can coexist:

# config.py
EMBEDDERS = {
    "v1": {"model": "text-embedding-3-small", "dim": 1536, "collection": "docs_te3small_1536"},
    "v2": {"model": "your-new-embedder",       "dim": 1024, "collection": "docs_new_1024"},
}

# Which generation each path uses is controlled by flags, not code edits
ACTIVE_READ = "v1"
ACTIVE_WRITE = ["v1", "v2"]  # dual-write during migration

Store the embedder id as metadata on every chunk as well. When you find a stray vector six months from now, you will want to know which model produced it.

Step 2: Backfill with a resumable job

Backfilling a corpus is a long batch job. It will get rate limited, crash, or be restarted. Make it idempotent and checkpointed, keyed on a content hash so unchanged chunks are never paid for twice.

import hashlib, json, time
from pathlib import Path

CHECKPOINT = Path("backfill_checkpoint.json")

def chunk_key(chunk) -> str:
    return hashlib.sha256(chunk["text"].encode()).hexdigest()

def load_done() -> set[str]:
    return set(json.loads(CHECKPOINT.read_text())) if CHECKPOINT.exists() else set()

def backfill(chunks, embed_fn, upsert_fn, batch_size=64):
    done = load_done()
    pending = [c for c in chunks if chunk_key(c) not in done]
    for i in range(0, len(pending), batch_size):
        batch = pending[i : i + batch_size]
        for attempt in range(5):
            try:
                vectors = embed_fn([c["text"] for c in batch])
                break
            except Exception:
                time.sleep(2 ** attempt)  # crude backoff; use your SDK's retry in prod
        else:
            raise RuntimeError("batch failed after retries")
        upsert_fn(batch, vectors)  # upsert by chunk id => safe to re-run
        done.update(chunk_key(c) for c in batch)
        CHECKPOINT.write_text(json.dumps(sorted(done)))

Two practical notes. First, use your provider's batch endpoint if it offers one, since a one-off backfill is exactly the latency-insensitive workload those are priced for (check the provider docs for current discounts). Second, run the backfill at low priority so it does not starve live query traffic of rate limit.

Step 3: Dual-write so the new index never goes stale

The backfill takes hours or days. During that window, documents are being added and edited. If you only write to the old index, the new one is stale the moment the backfill finishes.

def ingest(chunk):
    for gen in ACTIVE_WRITE:
        cfg = EMBEDDERS[gen]
        vec = embed(cfg["model"], chunk["text"])
        upsert(cfg["collection"], chunk["id"], vec, {**chunk["meta"], "embedder": gen})

def delete(chunk_id):
    for gen in ACTIVE_WRITE:
        remove(EMBEDDERS[gen]["collection"], chunk_id)  # deletes must dual-write too

Do not forget deletes. A document removed from the old index but left in the new one is a data-governance bug: it can resurface in answers after the cutover.

Step 4: Shadow evaluation before any user sees the new index

This is the step teams skip, and the one that justifies the whole exercise. Replay a sample of real production queries against both indexes and compare. You need a golden set to score against, so if you do not have one yet, build it first. See our guide on retrieval evaluation with golden sets for recall, MRR and nDCG.

def shadow_compare(queries, golden, k=10):
    rows = []
    for q in queries:
        old = search("v1", q, k)
        new = search("v2", q, k)
        rows.append({
            "q": q,
            "recall_old": recall_at_k(old, golden[q], k),
            "recall_new": recall_at_k(new, golden[q], k),
            "overlap": len({d.id for d in old} & {d.id for d in new}) / k,
        })
    return rows

Look at three things beyond the average:

Per-segment recall. A model can win overall and lose badly on one language, one document type, or short queries. Slice by those.
Overlap. Low top-k overlap with equal recall means the two models retrieve different but equally valid chunks. That is fine, but it tells you downstream prompts will see different context.
Regressions. List the queries where new is worse than old. Read twenty of them by hand. Metrics hide failure modes that reading exposes immediately.

Also re-tune any similarity threshold on the new model's score distribution. Plot score histograms for known-relevant and known-irrelevant pairs and pick the cutoff again. Do not carry the old number over.

Step 5: Canary and cutover behind a flag

Once shadow evals pass, route a small share of live traffic to the new index. Hash on user or session id so a given user gets a consistent experience:

import zlib

def read_generation(user_id: str, canary_pct: int) -> str:
    bucket = zlib.crc32(user_id.encode()) % 100
    return "v2" if bucket < canary_pct else ACTIVE_READ

Step the percentage 1 → 10 → 50 → 100, watching end-to-end signals rather than just retrieval: answer thumbs-up rate, citation click-through, "I don't know" rate, and latency. A new embedder with higher dimensionality can raise vector search latency and memory, so check p95 and your index size too. If you already run shadow and canary rollouts for LLM features, reuse that machinery.

Rollback is flipping ACTIVE_READ back. Because dual-write kept both indexes current, this is instant and lossless. That property is the entire reason for the dual-write phase.

Comparison: migration strategies

Strategy	Downtime	Cost	Risk	Use when
Swap model in place	Silent breakage	Lowest	Very high	Never in production
Maintenance-window re-embed	Minutes to hours	Low	Medium	Small internal corpus, tolerant users
Dual-index with canary	None	2x storage during migration	Low	Default for production
Alias swap (blue/green collection)	None	2x storage	Low	Vector DB supports atomic aliases
Matryoshka truncation, same model family	None	Minimal	Low	Only changing dimension within one model, not changing model

Many vector databases support collection aliases, which makes the final cutover a single atomic pointer swap. Check your store's docs for the exact feature name.

Common pitfalls

Re-embedding only the documents. Queries must use the new model too. Put the model id in one shared config used by both the ingest and query paths.
Forgetting query-time prefixes. Some embedding models expect different instruction prefixes for queries versus documents. Read the model card, and keep the new model's format.
Changing the chunker at the same time. If you change chunk size and embedder together, you cannot attribute the quality change to either. One variable per migration.
Cached query embeddings. A semantic or embedding cache keyed on query text will serve old-model vectors against the new index. Include the embedder id in the cache key, or flush it at cutover.
Deleting the old index too early. Keep it for at least one full traffic cycle, such as a week, so you can roll back when a slow-burning regression appears.
Ignoring cost. Re-embedding a large corpus is a real line item. Estimate tokens times price before you start, and use content hashes to skip unchanged chunks.

Pre-cutover checklist

New collection named with embedder id and dimension
Backfill complete, with count parity against the old index
Dual-write and dual-delete live for new traffic
Golden-set recall at least equal overall, and no segment regression you cannot explain
Similarity thresholds re-tuned on the new score distribution
Embedding and semantic caches keyed by embedder id
Canary dashboard with answer-quality and latency signals
Rollback is a flag flip, and it has been tested
Old index retained for a soak period before deletion

Next steps

Do a dry run before you need one. Even if you are not migrating today, make the embedder id a first-class config value, store it on every chunk, and write the backfill job once. When the next model ships, which it will, you will be running a checklist instead of improvising under pressure.

If you want hands-on practice building and evaluating production RAG systems with experienced mentors, have a look at the Joinloop engineering cohort. Product managers deciding when a retrieval upgrade is worth the cost and risk can explore the PM cohort.