Reranking for RAG: When Vector Search Isn't Enough

Most RAG pipelines retrieve the top 5 or 10 chunks by cosine similarity and feed them straight into the prompt. This works fine as a demo. It breaks in production, quietly, because the chunk that actually answers the question is sitting at rank 7 or 8 — close enough in embedding space to show up in a top-20 pull, but not close enough to make the top-5 cut that goes into the context window.

The fix isn't a better embedding model. It's a second-stage reranker that scores the shortlist against the actual query, not just against a compressed vector.

Why Vector Search Alone Misranks Results

Embedding models compress a passage of text into a single fixed-size vector, then compare it to the query's vector with cosine similarity. That compression is the problem. A 512-dimension vector has to represent every possible thing someone might ask about that passage — it optimizes for retrievability across many queries, not precision for this query.

Concretely, this causes two common failure modes:

1. Semantic drift. A chunk about "refund processing time" and a chunk about "refund eligibility criteria" can land close together in embedding space because they share vocabulary, even though only one answers "how long does a refund take?"
2. Length and dilution bias. Long chunks that touch many topics tend to have "average" embeddings that are moderately similar to a lot of queries, out-competing a short, precisely relevant chunk.

Bi-encoders (what standard embedding models are) encode the query and the document independently and compare vectors afterward. A cross-encoder reranker instead runs the query and each candidate document through the model together, letting attention layers directly compare tokens across both. That's why cross-encoders are dramatically more accurate at ranking — and also why you can't use them for the first retrieval pass over millions of documents. They're too slow to run against your whole corpus.

The standard architecture is a two-stage retrieval pipeline:

Query → Bi-encoder retrieval (top 50-100, fast, approximate)
      → Cross-encoder reranking (top 5-10, slow, precise)
      → LLM context window

You use the cheap, fast method to cast a wide net, then the expensive, accurate method to pick the best few from that net.

Implementation: Adding a Reranker to an Existing Pipeline

Assume you already have a vector store (Pinecone, Weaviate, pgvector, whatever) returning candidates. Here's how to bolt a reranker on top, using Cohere's rerank API as the example — the pattern is identical for other providers.

import cohere
from typing import TypedDict

co = cohere.Client(api_key="YOUR_COHERE_KEY")

class Chunk(TypedDict):
    id: str
    text: str
    metadata: dict

def retrieve_and_rerank(
    query: str,
    vector_store,
    top_k_retrieve: int = 50,
    top_k_final: int = 6,
) -> list[Chunk]:
    # Stage 1: cheap, wide retrieval from the vector store
    candidates = vector_store.similarity_search(query, k=top_k_retrieve)

    # Stage 2: precise reranking over the candidate set
    rerank_response = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=[c["text"] for c in candidates],
        top_n=top_k_final,
    )

    # rerank_response.results gives back indices into `candidates`,
    # sorted by the reranker's relevance score
    reranked = [
        {**candidates[r.index], "score": r.relevance_score}
        for r in rerank_response.results
    ]
    return reranked

The critical detail: retrieve wide, rerank narrow. Pulling only the top 5 from the vector store and reranking those 5 defeats the purpose — you need the reranker to see chunks that the bi-encoder scored lower but that are actually more relevant. A top_k_retrieve of 40-100 and top_k_final of 5-10 is a reasonable starting range; tune it against your own eval set.

Self-hosted alternative

If you don't want to depend on an external API, sentence-transformers ships open-weight cross-encoders that run locally on a GPU (or CPU for low volume):

from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank_local(query: str, candidates: list[Chunk], top_k: int = 6) -> list[Chunk]:
    pairs = [(query, c["text"]) for c in candidates]
    scores = model.predict(pairs)  # returns a float per pair

    scored = sorted(
        zip(candidates, scores), key=lambda x: x[1], reverse=True
    )
    return [{**c, "score": float(s)} for c, s in scored[:top_k]]

ms-marco-MiniLM-L-6-v2 is small (22M params) and fast enough to rerank 50 candidates in well under 100ms on a modern GPU, and still usably fast on CPU for low-QPS internal tools. For higher accuracy at more compute cost, BAAI/bge-reranker-v2-m3 is a strong open-weight option that handles multilingual queries well.

Measuring the Improvement

Don't ship a reranker on faith — measure it. The metric that matters is NDCG@k (normalized discounted cumulative gain) or, more simply, hit rate: for a labeled eval set of (query, correct chunk) pairs, what fraction of the time does the correct chunk end up in your final top-k, before vs. after reranking.

def hit_rate(eval_set: list[tuple[str, str]], retrieve_fn, k: int = 6) -> float:
    """eval_set is a list of (query, correct_chunk_id) pairs."""
    hits = 0
    for query, correct_id in eval_set:
        results = retrieve_fn(query, top_k_final=k)
        if any(r["id"] == correct_id for r in results):
            hits += 1
    return hits / len(eval_set)

# Compare
baseline = hit_rate(eval_set, retrieve_only, k=6)
with_rerank = hit_rate(eval_set, retrieve_and_rerank, k=6)
print(f"Vector-only hit rate: {baseline:.2%}")
print(f"Reranked hit rate: {with_rerank:.2%}")

In practice, teams commonly see hit-rate improvements of 10-25 percentage points at k=5 when moving from vector-only to two-stage retrieval, with the effect being largest on queries that use different phrasing than the source document (paraphrase gap) and on corpora with many semantically similar-but-distinct chunks (FAQ-style content, policy documents, legal text).

Comparison: Reranking Options

Option	Latency (50 docs)	Cost	Accuracy	Self-hosted
Cohere Rerank v3.5	~150-300ms	$2/1K searches	High	No
Voyage rerank-2	~150-300ms	$0.05/1M tokens	High	No
bge-reranker-v2-m3 (local)	~200-400ms (GPU)	Compute only	High, multilingual	Yes
ms-marco-MiniLM-L-6-v2 (local)	~50-100ms (GPU)	Compute only	Medium	Yes
LLM-as-reranker (e.g. GPT-5-mini scoring)	~1-3s	$$ (token cost)	Highest, but slow	No
No reranking (vector-only)	~0ms extra	Free	Baseline	—

The LLM-as-reranker approach — literally prompting a model to score or reorder candidates — gets the best accuracy in absolute terms but is too slow and expensive for most real-time paths. It's worth using for offline eval-set labeling or for low-QPS, high-stakes queries where a 2-second latency hit is acceptable.

Common Pitfalls

• Reranking too small a candidate set. If your vector store only returns 5 candidates and you rerank those 5, you've added latency without adding recall. Retrieve at least 5-10x your final k.
• Ignoring chunk length in the reranker input. Cross-encoders have token limits (often 512 tokens). If your chunks are longer, they get silently truncated before scoring — verify this isn't happening by checking the model's max sequence length against your chunk size.
• Not reranking on the same text you'll pass to the LLM. If you rerank a cleaned/summarized version of a chunk but pass the raw chunk to the LLM, the ranking and the actual content can diverge in ways that are hard to debug.
• Skipping the eval set. Reranking is not free — it adds latency and cost. Without a hit-rate measurement, you can't tell if it's actually helping your corpus or if your bi-encoder was already good enough.
• Reranking on every request regardless of candidate quality. If stage-1 retrieval already returns high-confidence results (e.g., vector score above some threshold), you can skip reranking for that request and save the latency. Not every query needs the second pass.
• Forgetting metadata filters happen before retrieval, not after reranking. Apply hard filters (tenant ID, date range, permissions) at the vector-store query stage — reranking a candidate set that includes documents the user isn't allowed to see is a correctness and security bug, not just a ranking one.

Next Steps

Start by building the hit-rate eval set — even 30-50 labeled (query, correct chunk) pairs from real user questions is enough to tell you whether reranking moves the needle on your specific corpus. Add the two-stage pipeline behind a flag, measure before/after, and only ship it once you can point to the number.

If you're building RAG systems and want to go deeper on retrieval architecture, evals, and production patterns like this one, Joinloop's AI Engineering cohort covers this hands-on with real pipelines, not slides.