Matryoshka Embeddings and Binary Quantization: Shrinking Your Vector Index Up to 128x

Most RAG systems store every embedding as a 1536- or 3072-dimension float32 vector. At a few million chunks, the index alone can run to tens of gigabytes of RAM, and every query scans it. The fix most teams reach for is a bigger instance. There is a cheaper fix: store less per vector, and use the full-precision data only where it matters.

Two techniques make this work together. Matryoshka truncation lets you keep only the first few hundred dimensions of an embedding. Binary quantization reduces each remaining dimension to one bit. Stacked, they turn a 4 KB vector into 32 bytes. The catch is accuracy, and the way to recover it is a two-stage search with a rescore step. This post explains why each piece works, shows runnable code, and gives you a checklist for measuring the loss on your own data.

The problem: float32 is expensive and mostly unnecessary

A 1024-dimension float32 vector takes 4,096 bytes. One million of them is about 4 GB before you add any index overhead. HNSW graphs add more on top, and the whole structure usually needs to sit in RAM for low-latency queries.

Most of that precision is wasted on ranking. A retrieval step only needs the right documents to land in the top 10 or top 50. It does not need the 7th decimal place of each component to be correct. The question is how much you can compress before the ranking breaks.

Matryoshka embeddings: the first dimensions carry the most signal

Matryoshka Representation Learning (MRL) trains an embedding model so that each prefix of the vector is useful on its own. The first 64 dimensions give a coarse representation, the first 256 a better one, and the full vector the best one. The model is trained with a loss applied at several truncation points at once, which pushes the important information toward the front.

This matters because you can truncate at query time without re-embedding anything. Check whether your model supports it. Several current embedding models document flexible output dimensions, and the model card or API docs will say which sizes are supported. If your model was not trained this way, truncating it usually hurts recall badly, so do not assume it will work.

The truncation step is simple:

import numpy as np

def truncate(vec: np.ndarray, dims: int) -> np.ndarray:
    # Keep the leading dims, then re-normalise so cosine similarity still works.
    prefix = vec[..., :dims]
    return prefix / np.linalg.norm(prefix, axis=-1, keepdims=True)

The re-normalisation step is easy to forget. Without it, dot products stop being cosine scores and your thresholds drift.

Binary quantization: one bit per dimension

Binary quantization keeps only the sign of each component. A positive value becomes 1, anything else becomes 0. Each vector is then packed into bits, and similarity is measured by Hamming distance: the number of bits that differ. The Hamming distance is computed with XOR and a popcount, which CPUs do very fast.

Put the two together and a 1024-dimension float32 vector (4,096 bytes) becomes 256 bits, or 32 bytes. That is a 128x reduction. In practice, most vector databases report something in the 32x range once you include index overhead and metadata, so measure your own system rather than trusting the headline number.

Binary codes are very cheap to scan, but they are lossy. Two vectors that differ in a small component can end up with identical bit patterns. That is why you do not use binary codes for the final answer.

The fix: a shortlist, then a rescore

The standard pattern has two stages:

1. Shortlist. Scan the binary index and take the top few hundred candidates by Hamming distance. This is cheap and fits in RAM.
2. Rescore. Load the full-precision vectors for just those candidates and rank them by exact cosine similarity. Return the top k.

The shortlist only needs to contain the true top results somewhere in its set. The rescore fixes the order. Because you read full vectors for a few hundred items instead of millions, the disk and memory cost stays small.

Here is a runnable version you can adapt. It uses synthetic embeddings so you can run it with no API keys:

import numpy as np

rng = np.random.default_rng(0)
N, D = 20_000, 1024          # corpus size, full embedding dimension
K = 256                      # Matryoshka prefix we keep
TOP_K, CANDIDATES = 10, 200  # final results, shortlist size

# Synthetic clustered embeddings, so neighbours are meaningful.
centers = rng.normal(size=(50, D))
corpus = centers[rng.integers(0, 50, N)] + 0.6 * rng.normal(size=(N, D))
corpus /= np.linalg.norm(corpus, axis=1, keepdims=True)
queries = corpus[rng.choice(N, 100, replace=False)] + 0.05 * rng.normal(size=(100, D))
queries /= np.linalg.norm(queries, axis=1, keepdims=True)

def to_binary(x):
    # Sign bit per dimension, packed 8 dims per byte.
    return np.packbits(x > 0, axis=-1)

def hamming_scores(q_bits, doc_bits):
    # Fewer differing bits means more similar, so negate for argsort.
    xor = np.bitwise_xor(doc_bits, q_bits)
    return -np.unpackbits(xor, axis=-1).sum(axis=-1)

# Build the compressed index once, at ingest time.
prefix = corpus[:, :K] / np.linalg.norm(corpus[:, :K], axis=1, keepdims=True)
prefix_bits = to_binary(prefix)

full_bytes = corpus.astype(np.float32).nbytes
print(f"full float32: {full_bytes/1e6:.1f} MB, binary index: {prefix_bits.nbytes/1e6:.2f} MB")

recalls = []
for q in queries:
    truth = np.argsort(-(corpus @ q))[:TOP_K]  # exact baseline

    q_prefix = q[:K] / np.linalg.norm(q[:K])
    shortlist = np.argsort(-hamming_scores(to_binary(q_prefix), prefix_bits))[:CANDIDATES]

    rescored = shortlist[np.argsort(-(corpus[shortlist] @ q))][:TOP_K]
    recalls.append(len(set(rescored) & set(truth)) / TOP_K)

print(f"recall@{TOP_K} after rescore: {np.mean(recalls):.3f}")

On this synthetic corpus the run prints about 0.75 recall@10 with a 200-candidate shortlist. Your number will differ, and that is the point. Recall depends heavily on your embedding model, your data, and how close your queries sit to the corpus distribution. The synthetic set shows the mechanics, not the figure you should expect.

Note that the rescore step here reads the full vectors from the in-memory corpus array. In production they would live on disk or in a separate store, and you would fetch only the shortlisted rows.

Measuring the tradeoff honestly

Do not pick a configuration from a blog post. Build a small evaluation set from your own traffic:

• Collect 200 to 500 real queries with known relevant documents. Production logs or support tickets work well.
• Compute the exact baseline with full float32 search.
• Sweep three knobs: Matryoshka dimension (for example 128, 256, 512), shortlist size (50, 100, 200, 500), and quantization (float32, int8, binary).
• Report recall@k against the baseline, and report latency and index size next to it.

Plot the results. You are looking for the knee of the curve, where a further size cut costs noticeably more recall than it saves.

Choosing a configuration

Option	Index size vs float32	Typical recall risk	When it fits
float32, full dims	1x	None (baseline)	Small corpora, or when you have not measured yet
Matryoshka truncation only	Proportional to dims kept	Low to moderate, model dependent	Model supports MRL and you need a simple win
int8 scalar quantization	About 4x smaller	Low	Default safe compression step
Binary + rescore	Roughly 32x or more smaller	Moderate, recovered by rescore	Large corpora where RAM is the bottleneck
Binary, no rescore	Same as above	High	Almost never for final ranking

The right answer is often two steps, not one. Start with int8 because the risk is low, and only go to binary with a rescore when index size is the real constraint.

Common pitfalls

• Truncating a model that was not trained for it. Dropping dimensions from a standard embedding model can collapse the geometry. Check the model docs first.
• Forgetting to re-normalise. Truncated vectors are no longer unit length. Cosine scores and thresholds will be wrong without the step.
• Using binary codes for the final ranking. Bits lose too much order information. Always rescore with full precision.
• Too small a shortlist. If the true top results never make the shortlist, no rescore can recover them. Measure recall at the shortlist stage, not only at the end.
• Mixing query and document settings. The query and every stored vector must use the same model, dimension, and quantization. A silent mismatch looks like a quality regression, and it is hard to debug.
• Trusting one benchmark. Published numbers come from specific datasets and models. Your corpus is different. Run the sweep.

Next steps

1. Check whether your embedding model documents flexible dimensions. If it does, measure recall at 256 and 512 dimensions against your current setting.
2. Add int8 quantization in your vector store if it supports it. It is the lowest-risk saving.
3. If RAM is still the constraint, prototype binary plus rescore on a copy of your index, using the evaluation set above.
4. Put recall, latency, and index size in the same dashboard so changes show up before users notice them.

Compression is one of the few levers in RAG that cuts cost and keeps quality, as long as you measure it. If you want to build production retrieval systems with proper evals, the Joinloop AI Engineering cohort works through this kind of tradeoff on real projects, with code reviews from practitioners who run these systems in production.