Late Interaction Retrieval with ColBERT and ColPali: When Token-Level Matching Beats One Vector per Document

Most retrieval pipelines compress each chunk into a single vector. A 768- or 1536-dimension embedding has to summarise a paragraph, a table, and a code snippet into one point in space. Queries that hinge on a specific detail, such as a part number, a version string, or a rare term in the middle of a long passage, often lose that detail in the compression.

Late interaction keeps more of the signal. A ColBERT-style model stores one vector per token. At query time, each query token finds its best match among the document's tokens, and the best matches are summed. The comparison happens late, after both sides are encoded separately, which is where the name comes from. This post explains the scoring rule, shows runnable code for it, and gives you a checklist for deciding whether the storage cost is worth paying.

The problem: one vector per chunk is a lossy summary

A bi-encoder embeds a query and a document into a single vector each, then ranks by cosine similarity. It is fast because the document side is precomputed into one row per chunk. The cost is that the embedding has to decide, up front, what the chunk is about. Anything the model did not weight heavily during encoding is hard to recover later.

A cross-encoder fixes this by reading the query and document together. It is much more accurate, but it runs one forward pass per query-document pair. You can only afford it on a shortlist of 20 to 100 candidates. Late interaction sits between the two. Documents are still encoded once, offline, but they keep their token-level detail, so the scoring step can be sharp without running the full model on every pair.

How MaxSim scores a document

Given a query with token vectors Q (shape nq x d) and a document with token vectors D (shape nd x d), both L2-normalised, the score is:

score(Q, D) = sum over query tokens q_i of  max over doc tokens d_j of  (q_i · d_j)

For each query token, find the single document token it matches best. Add up those best matches. A query token like "ERR_CONN_RESET" can latch onto the exact token in a long log file, even if the rest of that log has nothing to do with the query. A single pooled vector would have averaged that token away.

Here is the scoring function and a ranking loop. It uses NumPy only:

import numpy as np


def maxsim(query_vecs: np.ndarray, doc_vecs: np.ndarray) -> float:
    """Late-interaction score. Both inputs must be L2-normalised rows."""
    sims = query_vecs @ doc_vecs.T          # (nq, nd) token-to-token similarities
    return float(sims.max(axis=1).sum())    # best doc token per query token, summed


def rank_docs(query_vecs: np.ndarray, docs: list[np.ndarray], k: int = 3):
    scores = np.array([maxsim(query_vecs, d) for d in docs])
    top = np.argsort(scores)[::-1][:k]
    return [(int(i), float(scores[i])) for i in top]


def normalise(x: np.ndarray) -> np.ndarray:
    return x / np.linalg.norm(x, axis=-1, keepdims=True)


if __name__ == "__main__":
    rng = np.random.default_rng(0)
    dim = 128

    # Stand-ins for token embeddings. In production these come from a
    # ColBERT-style encoder, one row per token.
    docs = [normalise(rng.standard_normal((n, dim))) for n in (180, 220, 150)]
    query = normalise(rng.standard_normal((12, dim)))

    print(rank_docs(query, docs, k=2))

Two details matter in practice. The vectors must be normalised, or the dot products stop being cosines and scores from long and short documents stop being comparable. And max runs over document tokens, so a long document has more chances to produce a high match. This is one reason ColBERT-style models are trained with in-batch negatives and why you should evaluate on your own corpus rather than trust a leaderboard.

Making it practical: PLAID and compression

Scoring every document with the full MaxSim is too slow at scale. PLAID, the retrieval engine introduced with ColBERTv2, first clusters token embeddings into centroids. It uses centroid lookups to find a small set of candidate documents, then runs exact MaxSim only on those candidates. ColBERTv2 also compresses each token vector with residual quantization, which cut the storage per token sharply compared with the original ColBERT.

For the visual case, ColPali applies the same scheme to page images. Each page is split into image patches, each patch gets an embedding, and a query is matched against those patches with MaxSim. You skip OCR and layout parsing entirely. That is useful for scanned PDFs, slides, and tables where text extraction loses the structure. The trade is compute and storage, which grow with the number of patches per page.

Storage: the cost you have to plan for

The per-token index is the main price. Here is a back-of-envelope comparison for one million chunks, each with about 200 tokens, using 128-dimension vectors stored as float16:

Approach	Vectors per chunk	Bytes per vector	Approx. index size (1M chunks)
Single dense vector, 1024-dim float32	1	4,096	~4 GB
Single dense vector, 384-dim float16	1	768	~0.8 GB
ColBERT-style, 128-dim float16, uncompressed	~200	256	~51 GB
ColBERT-style with residual compression	~200	roughly 20 to 32	~4 to 6 GB

These numbers are rough. The compressed row depends on the codec and settings, so measure it for your model before you commit. The pattern holds, though. Uncompressed late interaction is far larger than a single-vector index, and compression is not optional at scale.

Choosing between the options

Option	Recall on exact-detail queries	Query latency	Index size	Best use
Bi-encoder (single vector)	Moderate	Very low	Small	First-stage retrieval over large corpora
Cross-encoder reranker	High	High per pair	None extra	Re-ordering a shortlist of 20 to 100
ColBERT late interaction	High	Moderate	Large (token-level)	Precision-sensitive text search, or as a reranker
ColPali (page images)	High on layouts and tables	Moderate to high	Large (patches per page)	Scanned PDFs, slides, visual documents

Use the checklist below to decide:

• Do your queries depend on rare terms, identifiers, or specific phrases inside long chunks? If yes, late interaction is worth testing.
• Is a cross-encoder reranker already fixing most of the misses? If yes, the reranker may be cheaper than a new index.
• Do you have a clear recall gap on a golden set, measured per query type? Without that, you cannot tell whether the extra index earns its cost.
• Is your corpus under a few hundred thousand chunks? Storage is manageable. Above a few million, plan for compression from day one.
• Are your documents visual, with tables, charts, or scans where OCR breaks? That is the case ColPali was built for.

Common pitfalls

• Skipping normalisation. Unnormalised token vectors make scores drift with vector length. Normalise at encode time and again in your tests.
• Comparing scores across different query lengths. MaxSim is a sum, so queries with more tokens score higher in absolute terms. Compare within one query, or normalise by query token count if you need a cross-query threshold.
• Evaluating on a leaderboard instead of your data. Public benchmarks reward different things than your users do. Build a golden set of real queries with the relevant chunk labelled, then measure recall@10 and nDCG on it.
• Forgetting the index build cost. Encoding every chunk into many vectors takes longer and uses more GPU time than single-vector encoding. Budget for it in your re-index plan.
• Mixing encoder versions. A new model version produces a different token space. A zero-downtime migration like the one in our embedding migration guide applies here too, and the index must be rebuilt, not patched.
• Treating late interaction as a drop-in replacement. It usually works best as a reranker over a cheap first stage, or as the precision stage of a hybrid pipeline with BM25. Test both placements.

Next steps

1. Pick 50 to 100 real user queries and label the chunk that answers each one.
2. Measure recall@10 for your current retriever, broken down by query type.
3. Add a cross-encoder reranker on the top 50 and measure again. This is the cheapest lever.
4. If exact-detail queries still miss, try a ColBERT-style model as the reranker over the same candidates before you build a full token-level index.
5. Only then decide on a full late-interaction index, and plan compression before you start.

Check the official model cards and the ColBERT and ColPali papers for current benchmark numbers and recommended settings, since these models change quickly.

Want to build retrieval systems like this with hands-on feedback? The Joinloop AI Engineering cohort covers RAG, evals, and production retrieval end to end, with projects you can measure against real data.