Monitoring RAG in Production: Index Freshness, Retrieval Drift, and Silent Quality Decay

A RAG system rarely fails loudly. There is no stack trace when the retriever starts returning last year's pricing page, or when a new document template makes your chunker emit fragments with no context. Latency is fine, error rate is zero, and the answers are confidently, plausibly wrong. Users notice first. You notice when someone forwards a screenshot.

Unlike a normal service, a RAG pipeline depends on three things that change without you deploying anything: the corpus (documents are added, edited, deleted), the queries (users' vocabulary and intent shift), and the upstream models (embedding and LLM providers update or deprecate versions). Classic uptime monitoring watches none of them.

This post lays out four signals that catch most silent decay, with code you can adapt. It assumes you already have an offline golden set for retrieval; if not, start with our guide to evaluating retrieval with golden sets. Offline evals tell you whether the system was good at a point in time. Monitoring tells you whether it still is.

The four failure modes

Failure mode	What changes	Symptom users see	Signal to watch
Stale index	Source docs updated, index not	Outdated facts stated confidently	Index lag (source updated_at vs indexed time)
Retrieval drift	Query mix or corpus shifts	Relevant chunks rank lower	Top-k similarity score distribution
Coverage gap	Users ask about things you never indexed	"I don't know" or hallucinated answers	Empty/low-confidence retrieval rate
Pipeline regression	Parser, chunker, or embedding version changes	Fragmented or irrelevant context	Chunk stats, canary query recall

Each has a cheap detector. None needs an LLM judge.

Signal 1: Index freshness

The most common cause of "the bot is wrong" is that the bot is right about a document that no longer exists. Measure index lag: for each indexed chunk, how long since its source changed without the index catching up?

Store source_updated_at and indexed_at as metadata on every chunk at write time. Then run a periodic job that compares against the source of truth.

# freshness_check.py
from datetime import datetime, timezone, timedelta

def index_lag_report(source_docs, index_meta, sla=timedelta(hours=6)):
    """
    source_docs: {doc_id: updated_at (datetime, tz-aware)} from the source system
    index_meta:  {doc_id: indexed_at (datetime, tz-aware)} from your vector store
    """
    now = datetime.now(timezone.utc)
    stale, missing, orphaned = [], [], []

    for doc_id, updated_at in source_docs.items():
        indexed_at = index_meta.get(doc_id)
        if indexed_at is None:
            missing.append(doc_id)            # never indexed
        elif indexed_at < updated_at and now - updated_at > sla:
            stale.append((doc_id, now - updated_at))  # changed, not re-indexed

    for doc_id in index_meta:
        if doc_id not in source_docs:
            orphaned.append(doc_id)           # deleted at source, still searchable

    return {
        "stale": stale,
        "missing": missing,
        "orphaned": orphaned,
        "stale_pct": len(stale) / max(len(source_docs), 1),
    }

Pay attention to orphaned documents. Deletions are the case incremental pipelines forget: the source removes a policy, the index keeps it, and the bot cites it forever. Alert on any orphan count above zero for access-controlled or compliance-sensitive content.

Choose an SLA per content type. Pricing and policy pages may need minutes; a handbook may tolerate a day. A single global number is almost always wrong.

Signal 2: Retrieval score drift

Every vector query returns similarity scores. The distribution of the top-1 and top-k scores over time is a surprisingly good health indicator, because it moves when queries, corpus, or embeddings change, even when you cannot label anything.

Log, per query: top-1 score, mean of top-k, and the gap between top-1 and top-k. Then compare a recent window to a baseline window.

# score_drift.py
import numpy as np
from scipy.stats import ks_2samp

def score_drift(baseline_scores, recent_scores, alpha=0.01):
    """Compare top-1 similarity score distributions between two windows."""
    stat, p = ks_2samp(baseline_scores, recent_scores)
    return {
        "ks_stat": float(stat),
        "p_value": float(p),
        "baseline_p50": float(np.percentile(baseline_scores, 50)),
        "recent_p50": float(np.percentile(recent_scores, 50)),
        "baseline_p10": float(np.percentile(baseline_scores, 10)),
        "recent_p10": float(np.percentile(recent_scores, 10)),
        "drifted": bool(p < alpha and np.median(recent_scores) < np.median(baseline_scores)),
    }

Two cautions. First, with large traffic, a KS test flags statistically significant but practically tiny shifts, so also set an effect-size threshold (for example, median down by more than 0.03, tuned to your embedding model). Second, scores are only comparable within one embedding model and metric. After any embedding change, reset the baseline. A downward shift in the 10th percentile is usually more informative than the median: it shows the tail of queries that are increasingly poorly served.

A drop does not tell you why. It tells you where to look: new query topics (coverage), a changed corpus (freshness or parsing), or a model swap.

Signal 3: Empty and low-confidence retrieval rate

Define a "weak retrieval" as a query whose top-1 score falls below a threshold you calibrated on your golden set. Track the weak-retrieval rate per day and, more usefully, cluster the weak queries to see which topics are missing.

# coverage_gaps.py
from collections import Counter
import numpy as np
from sklearn.cluster import KMeans

def cluster_weak_queries(queries, embeddings, k=8):
    """Group low-confidence queries so you can see which topics lack content."""
    km = KMeans(n_clusters=k, n_init="auto", random_state=0).fit(np.asarray(embeddings))
    groups = {}
    for q, label in zip(queries, km.labels_):
        groups.setdefault(int(label), []).append(q)
    # Largest clusters first; show a few examples from each for a human to name
    return sorted(groups.values(), key=len, reverse=True), Counter(km.labels_)

Review the top clusters weekly. This turns monitoring into a content roadmap: "47 weak queries about refunds for annual plans" is a ticket for the docs team, not a prompt-engineering problem. It also pairs well with calibrated abstention: if retrieval is weak, the system should say so rather than improvise.

Signal 4: Canary queries and chunk statistics

Drift detectors are indirect. Add two direct checks that run on every ingestion and on a schedule.

Canary queries. Pick 30–100 questions from your golden set with known correct chunks or documents. Run them against the live index on a schedule and alert if recall@k drops below a floor. This catches silent regressions from re-indexing, embedding version bumps, or a parser update, using the same metrics you used at launch.

Chunk statistics. After each ingestion batch, compute simple distributions and compare to the previous batch:

def chunk_stats(chunks):
    lengths = [len(c["text"].split()) for c in chunks]
    return {
        "count": len(chunks),
        "p05_words": int(np.percentile(lengths, 5)),
        "p50_words": int(np.percentile(lengths, 50)),
        "p95_words": int(np.percentile(lengths, 95)),
        "tiny_pct": sum(l < 20 for l in lengths) / len(lengths),
        "duplicate_pct": 1 - len({c["text"] for c in chunks}) / len(chunks),
    }

A sudden jump in tiny chunks or duplicates usually means a parsing change upstream (a new PDF template, a headers-and-footers regression). It is far cheaper to catch at ingestion than after users complain. See our notes on document parsing for RAG for common parser failure patterns.

What to put on the dashboard

Panel	Metric	Alert when
Freshness	Stale %, orphan count, max index lag	Orphans > 0 on sensitive content; lag > SLA
Score health	Top-1 score p10 and p50, 7-day vs 30-day	p10 drops beyond your effect-size threshold
Coverage	Weak-retrieval rate, top weak clusters	Rate up sharply week over week
Canary	recall@k on fixed queries	Below the floor you set at launch
Ingestion	Chunk count, tiny %, duplicate %	Deviates sharply from previous batch
User signal	Thumbs-down rate, re-ask rate	Rising while other panels stay green

The last row matters. Retrieval metrics are proxies; user feedback is ground truth. When user signals worsen and the other panels are green, your proxies are missing something, so sample those traces and add the failure to your golden set. This is the loop described in mining production traces into eval data.

Common pitfalls

• Comparing scores across embedding models. A model upgrade shifts the entire score distribution. Reset baselines and thresholds when you re-embed.
• Alerting on the mean. Averages hide the tail. Watch percentiles, especially p10.
• One global freshness SLA. Different content decays at different rates; set SLAs per source.
• Forgetting deletions. Orphaned chunks are the quietest and most damaging staleness bug.
• Logging queries without privacy review. Query logs often contain personal data. Redact or hash before storing, and check retention rules with your legal team.
• Treating drift as a bug in the model. Most "drift" is a corpus or ingestion change. Check what changed upstream before touching prompts.
• No owner. An alert nobody owns is noise. Assign each panel to a person or team.

A rollout plan you can start tomorrow

1. Add source_updated_at, indexed_at, and embedding_model metadata to every chunk if you do not already.
2. Log top-1 score, top-k scores, and result doc IDs for every query.
3. Schedule the freshness job and the canary-query job (daily is a fine start).
4. Baseline the score distribution for two weeks of normal traffic, then add drift alerts.
5. Review weak-query clusters weekly and feed them to whoever owns content.

Tooling varies: many teams build this on their existing observability stack, and LLM tracing platforms offer retrieval views. Check your vendor's docs for what retrieval metadata they capture before building from scratch.

Next steps

Monitoring turns RAG from a launch artifact into a system you can keep healthy. If you want to build and operate pipelines like this with live feedback from practitioners, our hands-on AI Engineering cohort covers RAG evaluation, observability, and production operations. If you are a product manager deciding which quality signals your team should own, the PM cohort walks through how to define them.