Fine-Tuning Embedding Models for RAG: A Practical Guide with Sentence Transformers

Most RAG debugging happens in the wrong place. The answer is wrong, so the team rewrites the prompt, adds a reranker, raises top_k from 5 to 20, and switches to a bigger generator. Recall barely moves, because the problem sits one step earlier: the embedding model doesn't know that "TAT breach" and "SLA violation" mean the same thing in your support tickets, or that "Form 16" and "salary TDS certificate" should land close together.

General-purpose embedding models learn from web-scale data, which is exactly what makes them weak on your internal vocabulary, product names, and abbreviations. A reranker can reorder what the retriever returns, but it can't recover a document that never made the candidate list.

Fine-tuning an open embedding model on your own corpus is one of the cheapest, highest-leverage fixes in a RAG stack. On narrow domains it's common to see double-digit gains in recall@10 from a few thousand training pairs and under an hour on a single GPU. This guide walks through the whole loop: build training data without human labelling, mine hard negatives, train with the right loss, and prove the gain with an eval you trust.

When fine-tuning is (and isn't) the right fix

Measure before you train. Fine-tuning helps when retrieval misses documents that a domain expert would consider obviously relevant. It doesn't help when the right document doesn't exist, is split badly, or is buried in a PDF your parser mangled.

Symptom	Likely cause	Fix first
Right doc exists, not in top 50	Embedding doesn't understand domain vocabulary	Fine-tune embeddings
Right doc in top 50, not in top 5	Ranking precision	Add a reranker
Right doc retrieved, answer still wrong	Chunk lacks context, or generation issue	Contextual chunking, prompt fixes
Exact IDs / codes / SKUs missed	Dense models are weak on exact tokens	Hybrid search (BM25 + dense)
Right doc isn't in the index at all	Ingestion or parsing	Fix the pipeline

If most failures sit in the first row, keep reading.

How embedding fine-tuning works

An embedding model maps text to vectors so that relevant pairs end up close together. Fine-tuning for retrieval uses contrastive learning: you show the model (query, relevant passage) pairs and train it to pull those together while pushing unrelated passages apart.

The workhorse loss is Multiple Negatives Ranking Loss (MNRL). For a batch of N (query, positive) pairs, each query treats its own positive as the correct answer and the other N−1 positives in the batch as negatives. It's effectively an N-way classification problem per query, and you get negatives for free. Two consequences follow:

1. Batch size matters. More in-batch negatives make the task harder and the signal richer. 64–256 is a reasonable range; cached variants of the loss let you go higher without running out of GPU memory.
2. Duplicates hurt. If two pairs in one batch share a positive, the model gets penalised for a correct match. Use a no-duplicates batch sampler.

Random in-batch negatives are usually easy to tell apart. Hard negatives, passages that look relevant but aren't, teach the model the fine distinctions that matter in your domain.

Step 1: Generate training pairs from your corpus

You rarely have labelled (query, passage) pairs, but you do have the passages. Use an LLM to write the questions a real user would ask that each chunk answers. This is the same "synthetic query" trick behind many retrieval benchmarks.

# pip install openai datasets
import json, random
from openai import OpenAI

client = OpenAI()  # any OpenAI-compatible endpoint works

PROMPT = """You are generating search queries for a retrieval system.
Given the passage below, write {n} distinct questions a real user might type
that this passage answers. Vary the phrasing: use jargon in one, plain language
in another, and make one short and keyword-like. Do not copy sentences verbatim.
Return JSON: {{"queries": ["...", "..."]}}

Passage:
{passage}"""

def synth_queries(passage: str, n: int = 3) -> list[str]:
    resp = client.chat.completions.create(
        model="gpt-4.1-mini",  # use whichever cheap, capable model you have
        messages=[{"role": "user", "content": PROMPT.format(n=n, passage=passage)}],
        response_format={"type": "json_object"},
        temperature=0.7,
    )
    return json.loads(resp.choices[0].message.content)["queries"]

chunks = [json.loads(l) for l in open("chunks.jsonl")]  # {"id": ..., "text": ...}
pairs = []
for c in chunks:
    for q in synth_queries(c["text"]):
        pairs.append({"anchor": q, "positive": c["text"], "chunk_id": c["id"]})

# Split by chunk, not by pair, so eval passages never appear in training
ids = list({p["chunk_id"] for p in pairs})
random.seed(42); random.shuffle(ids)
eval_ids = set(ids[: int(0.1 * len(ids))])
train = [p for p in pairs if p["chunk_id"] not in eval_ids]
test = [p for p in pairs if p["chunk_id"] in eval_ids]

Two details that matter more than they look:

• Split by chunk ID. If the same passage shows up in train and test under different questions, your eval measures memorisation, not generalisation.
• Mix in real queries. If you have search logs or thumbs-up signals, add those pairs. Synthetic queries tend to be cleaner and more literal than what users actually type.

Step 2: Establish a baseline eval

Build the evaluator before you train anything. Sentence Transformers ships an InformationRetrievalEvaluator that reports recall@k, MRR, and NDCG over a query set and a corpus.

# pip install -U sentence-transformers
from sentence_transformers import SentenceTransformer
from sentence_transformers.evaluation import InformationRetrievalEvaluator

# Use the FULL corpus so the eval reflects real retrieval difficulty
corpus = {c["id"]: c["text"] for c in chunks}
queries = {f"q{i}": p["anchor"] for i, p in enumerate(test)}
relevant = {f"q{i}": {p["chunk_id"]} for i, p in enumerate(test)}

evaluator = InformationRetrievalEvaluator(
    queries=queries, corpus=corpus, relevant_docs=relevant,
    name="domain-test", accuracy_at_k=[1, 5, 10], ndcg_at_k=[10],
)

base = SentenceTransformer("BAAI/bge-base-en-v1.5")
print(evaluator(base))

Write the baseline numbers down. You'll compare against them, and you'll also run this eval on the current production model (including a hosted API model, if that's what you use) so the fine-tune has to beat what you already ship.

Step 3: Mine hard negatives

Use the base model itself to find passages that rank high for a query but aren't the positive. These are the confusable neighbours your model needs to learn to separate.

from datasets import Dataset
from sentence_transformers.util import mine_hard_negatives

train_ds = Dataset.from_list([{"anchor": p["anchor"], "positive": p["positive"]} for p in train])

hard_ds = mine_hard_negatives(
    train_ds,
    base,
    corpus=list(corpus.values()),
    range_min=10,        # skip the top 10: some are likely unlabelled true positives
    range_max=50,
    num_negatives=1,
    relative_margin=0.05,  # negative must score at least 5% below the positive
    use_faiss=False,
)
# Columns: anchor, positive, negative

The range_min and margin settings guard against the most common hard-negative failure: false negatives. In a real corpus, several chunks often answer the same question. If you label one of them as a "negative", you teach the model to push relevant content away. Skipping the top few hits and requiring a score margin removes most of these. Check the exact parameter names against the version you install, since the helper's signature has changed across releases.

Step 4: Train with MNRL + Matryoshka

MatryoshkaLoss wraps your main loss so the first 64, 128, 256... dimensions of each embedding are useful on their own. You can then truncate vectors in production, cutting storage and search cost, and lose little accuracy.

from sentence_transformers import (
    SentenceTransformer, SentenceTransformerTrainer, SentenceTransformerTrainingArguments,
)
from sentence_transformers.losses import CachedMultipleNegativesRankingLoss, MatryoshkaLoss
from sentence_transformers.training_args import BatchSamplers

model = SentenceTransformer("BAAI/bge-base-en-v1.5")

inner = CachedMultipleNegativesRankingLoss(model, mini_batch_size=32)
loss = MatryoshkaLoss(model, inner, matryoshka_dims=[768, 512, 256, 128, 64])

args = SentenceTransformerTrainingArguments(
    output_dir="bge-base-domain",
    num_train_epochs=3,
    per_device_train_batch_size=128,   # large batch = more in-batch negatives
    learning_rate=2e-5,
    warmup_ratio=0.1,
    bf16=True,                          # set False if your GPU lacks bf16
    batch_sampler=BatchSamplers.NO_DUPLICATES,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="eval_domain-test_cosine_ndcg@10",
)

trainer = SentenceTransformerTrainer(
    model=model, args=args, train_dataset=hard_ds, loss=loss, evaluator=evaluator,
)
trainer.train()
print(evaluator(model))
model.save_pretrained("bge-base-domain/final")

A base-size model with a few thousand pairs trains in minutes to an hour on one consumer or cloud GPU. The cached MNRL variant decouples the effective batch size from GPU memory, so 128+ fits on a 24 GB card.

One easy-to-miss detail: many embedding models expect an instruction prefix on queries (BGE, E5, and others each have their own convention). Whatever you use during training, use the exact same prefix at inference. Check the model card.

Step 5: Ship it without breaking retrieval

A fine-tuned model produces vectors in a different space. You can't mix old and new embeddings in the same index.

1. Re-embed the whole corpus into a new index or collection.
2. Run the eval again against the new index, plus a sample of real production queries graded by a human or an LLM judge.
3. Shadow or A/B the traffic. Route a slice of queries to the new index and compare downstream metrics such as answer acceptance and escalation rate, not just recall.
4. Version the model and the index together, so a rollback swaps both at once.

Model and approach comparison

Option	Domain quality	Cost to run	Effort	Good fit when
Hosted embedding API, off the shelf	Good on general text	Per-token fees	None	Prototypes, broad content
Open model, off the shelf	Good on general text	Your infra	Low	Data residency, cost control
Open model, fine-tuned	Best on your vocabulary	Your infra	Medium (1–2 days)	Narrow domain, measurable misses
Fine-tuned + reranker	Best overall	Higher latency	Medium-high	High-stakes answers
Linear adapter on API embeddings	Modest gains	API fees + tiny model	Low	You're locked into a hosted model

The last row deserves a mention: if you can't leave a hosted embedding API, you can still train a small linear transform applied to query embeddings. It captures less than a full fine-tune, but it needs no re-embedding of the corpus.

Common pitfalls

• Leaky eval splits. Splitting by pair instead of by passage inflates results. Always hold out whole chunks.
• Evaluating on a tiny corpus. Retrieving from 500 test passages is far easier than from 500,000. Evaluate against the full corpus.
• False-negative hard negatives. Mining from the top 1–3 results often grabs genuinely relevant passages. Skip the top ranks and enforce a margin.
• Overfitting to synthetic phrasing. LLM-written questions are grammatical and literal. Mix in real queries, and generate keyword-style and misspelled variants.
• Catastrophic forgetting. Heavy training on a narrow set can hurt general queries. Keep epochs low (1–3), use a small learning rate, and include a general-domain eval slice.
• Prefix mismatch. Training with "query: " and serving without it quietly erases much of the gain.
• Forgetting to re-embed. Querying an old index with a new model returns noise, often without any error.

Checklist before you call it done

• Failure analysis shows retrieval misses, not parsing or chunking problems
• Train/test split by chunk ID, eval over the full corpus
• Baseline numbers recorded for the base model and the current production model
• Hard negatives mined with a rank offset and margin
• Recall@10 and NDCG@10 improved on held-out data and on real queries
• No regression on a general-domain sanity set
• Corpus re-embedded, model and index versioned together
• Shadow or A/B traffic shows downstream improvement

Next steps

Start small: pick one domain where users complain that "it can't find the doc", generate 2,000–5,000 synthetic pairs, and run the baseline eval today. If the base model already gets recall@10 above 0.95, spend your time on reranking or chunking instead. If it sits at 0.7, a fine-tune is likely the cheapest big win left in your stack.

After that, look at pairing the fine-tuned model with hybrid search and a reranker, and wiring the retrieval eval into CI so the next model swap, or the next corpus change, can't quietly regress it. Library APIs move quickly, so check the Sentence Transformers docs for the current signatures before you pin versions.

If you want to build and ship retrieval systems like this with feedback from practitioners, our AI engineering cohort takes you from RAG fundamentals to production evals over a series of hands-on projects.