โ† Back to Blog
RAGAI EngineeringTools

Filtered Vector Search: Why Metadata Filters Quietly Break RAG Recall (and How to Fix It)

Add a WHERE clause to an approximate vector search and your top-10 can shrink to 3 results without a single error. Learn pre- vs post-filtering, pgvector iterative scans, partial indexes, and how to measure filtered recall.

By Joinloopยท17 November 2026ยท9 min read
ShareLinkedInXReddit
๐Ÿงฒ

Your RAG demo worked on the whole corpus. Then product asked for the obvious things: only this customer's documents, only the last 90 days, only doc_type = 'contract'. You added a WHERE clause, and nothing broke. No errors, no timeouts.

But a query that asked for 10 chunks now returns 3. Sometimes 0. The LLM answers from thin context, and quality drops for exactly the users with the most specific questions.

This is the filtered vector search problem. It is one of the most common silent failures in production RAG, and it comes from a mismatch between how approximate nearest neighbour (ANN) indexes work and how SQL filters work.

Why filters and ANN indexes fight each other

An HNSW index is a graph. A search walks the graph towards the query vector and keeps a candidate list of the closest nodes it has seen. The size of that list is a tuning knob (ef_search in most implementations). The index knows nothing about your tenant_id column.

So there are three places a filter can be applied:

Strategy How it works Fails when
Post-filtering Run ANN, get N candidates, then drop rows that fail the filter Filter is selective: most candidates are thrown away, you get fewer than k results
Pre-filtering Apply the filter first, then do an exact (brute-force) distance scan over survivors Filter is loose: you brute-force millions of vectors per query
In-index filtering Check the filter while walking the graph, keep traversing until k matches are found Very selective filters can disconnect the graph or force long traversals; depends on engine support

The arithmetic for post-filtering is brutal. The pgvector README spells it out: with HNSW, filtering is applied after the index scan, so if a condition matches 10% of rows and hnsw.ef_search is at its default of 40, only about 4 rows match on average. You asked for LIMIT 10. You get 4.

Now make the filter a tenant that owns 0.5% of the table. Expected matches from 40 candidates: 0.2. Most queries return nothing, and the ones that do return something return whatever happened to be near the query globally, not the best match inside the tenant.

The key variable is selectivity: the fraction of rows that pass the filter. Every decision in this post follows from it.

Fix 1: Let the planner go exact for very selective filters

If a filter leaves a few thousand rows, you do not need an ANN index at all. Exact distance over 5,000 vectors takes milliseconds and has perfect recall.

In Postgres, give the planner a way to find those rows cheaply:

CREATE INDEX ON chunks (tenant_id);
-- or for combined filters
CREATE INDEX ON chunks (tenant_id, doc_type);

With a B-tree on the filter column and up-to-date statistics, the planner can choose "index scan on tenant_id, then sort by distance" over the HNSW index. Check with EXPLAIN ANALYZE rather than assuming. If the plan shows the HNSW index being used for a tiny tenant, your stats are stale or the cost estimates are off.

Fix 2: Iterative index scans (pgvector 0.8.0+)

For the middle ground, where a filter matches somewhere between 1% and 30% of rows, pgvector 0.8.0 added iterative index scans. Instead of scanning once and giving up, the index keeps scanning further until enough rows pass the filter or a limit is hit.

-- Keep exact distance ordering
SET hnsw.iterative_scan = strict_order;

-- Or: better recall, results may be slightly out of order
SET hnsw.iterative_scan = relaxed_order;

-- Safety cap on how much of the index one query may visit (default 20000)
SET hnsw.max_scan_tuples = 20000;

SELECT id, content, embedding <=> $1 AS distance
FROM chunks
WHERE tenant_id = $2 AND created_at > now() - interval '90 days'
ORDER BY embedding <=> $1
LIMIT 10;

Two things to know:

  • relaxed_order can return rows slightly out of distance order. If order matters downstream (it usually does not if you rerank), the pgvector docs recommend wrapping the query in a materialized CTE and re-sorting by distance.
  • max_scan_tuples is your latency guard. A filter that matches almost nothing will scan up to that cap and still come back short. Iterative scans reduce the problem; they do not remove the need for Fix 1 or Fix 3 at the extremes.

Set these per transaction with SET LOCAL so they do not leak across pooled connections.

Fix 3: Partial indexes and partitioning

When the filter column has a small, known set of values, build the filter into the index itself:

-- A few distinct values: one partial HNSW index per value
CREATE INDEX chunks_contract_hnsw ON chunks
USING hnsw (embedding vector_cosine_ops)
WHERE (doc_type = 'contract');

Queries with WHERE doc_type = 'contract' now search a graph that contains only contracts. No filtering loss at all.

When there are many values (hundreds of tenants), partial indexes become unmanageable. Use partitioning instead:

CREATE TABLE chunks (
  id bigserial,
  tenant_id int NOT NULL,
  content text,
  embedding vector(1024)
) PARTITION BY LIST (tenant_id);

Each partition gets its own HNSW index, and partition pruning turns a filtered search into an unfiltered search over a smaller table. This also pairs well with row-level security for multi-tenant isolation. The cost is operational: more indexes to build, more memory, and partition management for new tenants. For thousands of small tenants, hash-partition into a fixed number of buckets and rely on Fix 1 inside each bucket.

What about dedicated vector databases?

Most dedicated engines do in-index filtering, which is the main thing you are paying for:

  • Qdrant builds payload indexes and adds extra graph links so the HNSW graph stays connected under filters; its query planner falls back to exact search for very selective filters.
  • Weaviate uses a pre-computed allow-list during traversal and offers an ACORN-based filter strategy for low-correlation filters.
  • Pinecone and Milvus apply metadata filters during the search rather than after it.

Implementation details change between versions, so check the current docs for your engine. The important point is behavioural: you still need to index the fields you filter on, and you still need to test recall at your real selectivities. "Supports filtering" is not the same as "holds recall at 0.1% selectivity".

Measure it: a filtered recall harness

You cannot tune what you do not measure. The ground truth for filtered search is cheap to compute: run the same filtered query with index scans disabled, which forces an exact scan, and compare IDs.

import psycopg
from pgvector.psycopg import register_vector

K = 10

QUERY = """
SELECT id FROM chunks
WHERE tenant_id = %s
ORDER BY embedding <=> %s
LIMIT %s
"""

def top_ids(conn, tenant_id, vec, *, exact=False, iterative=None):
    with conn.transaction():
        cur = conn.cursor()
        if exact:
            # Force a sequential scan + sort: perfect recall, slow
            cur.execute("SET LOCAL enable_indexscan = off")
            cur.execute("SET LOCAL enable_bitmapscan = off")
        elif iterative:
            cur.execute(f"SET LOCAL hnsw.iterative_scan = {iterative}")
        cur.execute(QUERY, (tenant_id, vec, K))
        return [row[0] for row in cur.fetchall()]

def evaluate(conn, samples, iterative=None):
    """samples: list of (tenant_id, query_vector) drawn from real traffic."""
    recalls, short = [], 0
    for tenant_id, vec in samples:
        truth = set(top_ids(conn, tenant_id, vec, exact=True))
        got = top_ids(conn, tenant_id, vec, iterative=iterative)
        if len(got) < min(K, len(truth)):
            short += 1
        recalls.append(len(truth & set(got)) / max(len(truth), 1))
    return sum(recalls) / len(recalls), short / len(samples)

with psycopg.connect("postgresql://localhost/rag") as conn:
    register_vector(conn)
    samples = load_samples()  # your own: (tenant_id, numpy vector) pairs
    for mode in (None, "strict_order", "relaxed_order"):
        recall, short_rate = evaluate(conn, samples, iterative=mode)
        print(f"{mode or 'default':>14}  recall@{K}={recall:.3f}  short={short_rate:.1%}")

iterative is interpolated from a fixed list, not user input; never build SET statements from request data.

Two metrics matter:

  • Filtered recall@k: overlap with the exact filtered result.
  • Short-result rate: the share of queries that return fewer than k rows when k matching rows exist. This is the metric that catches the silent failure, and it is worth alerting on in production.

Stratify your samples by selectivity. Averages hide the problem: big tenants look fine and drag the mean up while small tenants get empty results. Report recall for the smallest decile of tenants separately.

Choosing a strategy by selectivity

Filter matches Recommended approach (Postgres/pgvector)
< ~1% of rows, or under ~10k rows B-tree on filter column, exact distance scan
~1% to ~30% HNSW + iterative scan (relaxed_order if you rerank), raise max_scan_tuples if short
> ~30% Plain HNSW; modestly raise ef_search
Few fixed values (doc type, language) Partial HNSW index per value
Many values with strong isolation (tenant) Partition by the filter column

These thresholds are starting points, not laws. Your crossover depends on vector dimensions, hardware, and how correlated the filter is with the query. Run the harness.

Common pitfalls

  • Over-fetch and filter in application code. Fetching LIMIT 200 and filtering in Python is post-filtering with extra network cost. It still fails at low selectivity and breaks pagination.
  • Ignoring filter-query correlation. If the query is about invoices and the filter is doc_type = 'invoice', post-filtering looks great. If the filter is unrelated to the query (a tenant ID), it collapses. Test with realistic pairs, not random ones.
  • LLM-generated filters with no fallback. Self-query retrievers that turn "contracts from last year" into a metadata filter are useful, but a hallucinated or over-strict filter returns zero rows. Validate filter values against known enums, and retry with a relaxed filter when results are short.
  • Unindexed filter fields. In both Postgres and dedicated vector databases, filtering on an unindexed field degrades to a scan or to post-filtering.
  • Session settings on pooled connections. SET without LOCAL leaks iterative_scan or ef_search to other requests through PgBouncer.
  • Evaluating only unfiltered recall. Your golden set probably has no filters. Production queries almost always do.

Next steps

  1. Log result counts for every retrieval call and chart the short-result rate. This takes an hour and tells you whether you have the problem.
  2. Compute the selectivity distribution of your real filters (per tenant, per doc type, per date window).
  3. Run the recall harness on the smallest and largest segments.
  4. Apply the fix that matches each segment: exact scan, iterative scan, partial index, or partition.
  5. Add filtered queries to your retrieval golden set so regressions show up in CI.

Filtered retrieval, hybrid search, and retrieval evals are the kind of production detail we work through hands-on in the Joinloop AI engineering cohort, if you want to build these systems with feedback from practitioners.

ShareLinkedInXReddit

Want to build this yourself?

Join a live cohort and go from reading about AI to shipping it, with practitioners who do it daily.