RAG Citation Verification: Claim-Level Groundedness Checks That Catch Fabricated Answers

Your RAG system returns an answer with neat [1], [2] citations. Users trust it more because of them. But the citations are just text the model generated. Nothing guarantees that source [2] actually says what the sentence beside it claims.

This is the gap between having citations and being grounded. Published research on RAG citation behaviour suggests that models sometimes answer from parametric memory and then attach a plausible-looking source afterwards ("post-rationalised" citations). Treat any specific percentage you read online with caution, but the failure mode is real and easy to reproduce on your own data.

This post shows how to close that gap: decompose answers into claims, check each claim against the cited span, and decide what to do when a claim fails.

Three words people mix up

Term	Question it answers	Granularity
Faithfulness	Is the whole answer consistent with the retrieved context?	Answer
Groundedness	Is each statement supported by a specific source?	Sentence / claim
Factuality	Is the statement true in the real world?	Claim vs. ground truth

A faithful answer can still be wrong if your corpus is wrong or stale. A factual answer can still be ungrounded if the model got it from memory. For RAG, groundedness is the property you can actually enforce, because you control the sources.

Why answer-level scores hide the problem

If you score an answer with a single "faithfulness: 0.9" number, an answer with nine supported claims and one fabricated claim looks excellent. But the fabricated claim is the one that ends up in a customer email or a compliance report.

Measure at the claim level and gate on the minimum, not the average. One unsupported claim should be visible.

The pipeline

question → retrieve → generate (structured, with citations)
        → split into claims → for each claim: fetch cited span
        → entailment check → pass / repair / abstain

Four design decisions matter:

1. Force structured citations. Make the model emit claim-to-chunk mappings, not free-text brackets.
2. Check against the cited span, not the whole corpus. Otherwise a claim "passes" because some unrelated chunk happens to support it, and your citations stay wrong.
3. Use entailment, not similarity. Embedding similarity says "same topic". Entailment says "this text supports that statement". "Revenue grew 12%" and "Revenue fell 12%" are near-identical in embedding space.
4. Decide the failure policy up front. Repair, drop the claim, or abstain.

Step 1: Structured output with per-claim citations

from pydantic import BaseModel, Field

class Claim(BaseModel):
    text: str = Field(description="One atomic factual statement")
    source_ids: list[str] = Field(
        description="IDs of retrieved chunks that directly support this claim"
    )

class GroundedAnswer(BaseModel):
    claims: list[Claim]
    abstain: bool = Field(
        default=False,
        description="True if the context does not contain enough information",
    )

Prompt the model with chunk IDs and an explicit instruction: each claim must be a single fact; cite only chunks that directly state it; set abstain=true rather than guess. Use your provider's structured-output feature to enforce the schema (see the official docs for the exact parameter in your SDK).

Atomic claims matter. "The plan costs $20/month and includes SSO" is two claims. If SSO is unsupported, a combined sentence either fails entirely (losing the true half) or passes wrongly.

Step 2: Check each claim against its cited span

You have two practical options for the verifier: a small NLI (natural language inference) model, or an LLM-as-judge with a tight rubric. Both take (premise=span, hypothesis=claim) and return supported / contradicted / not enough information.

import json
from typing import Literal
from pydantic import BaseModel

class Verdict(BaseModel):
    label: Literal["supported", "contradicted", "unsupported"]
    evidence_quote: str  # exact substring of the premise, or ""

JUDGE_PROMPT = """You are verifying a claim against a source passage.

Source passage:
\"\"\"{premise}\"\"\"

Claim:
\"\"\"{claim}\"\"\"

Rules:
- "supported" only if the passage explicitly states or directly implies the claim.
- "contradicted" if the passage states the opposite.
- "unsupported" if the passage is related but does not establish the claim.
- evidence_quote must be copied verbatim from the passage. Empty if unsupported.
Return JSON: {{"label": ..., "evidence_quote": ...}}"""

def verify_claim(llm, claim: str, chunks: dict[str, str], source_ids: list[str]) -> Verdict:
    premise = "\n---\n".join(chunks[s] for s in source_ids if s in chunks)
    if not premise:
        return Verdict(label="unsupported", evidence_quote="")
    raw = llm(JUDGE_PROMPT.format(premise=premise, claim=claim))
    verdict = Verdict(**json.loads(raw))
    # Hard check: the quote must really exist in the source.
    if verdict.label == "supported" and verdict.evidence_quote not in premise:
        return Verdict(label="unsupported", evidence_quote="")
    return verdict

The evidence_quote in premise check is cheap and powerful. A judge that says "supported" but can't quote the source verbatim is usually guessing. Note the early return as well: a claim citing a non-existent chunk ID is automatically unsupported.

Step 3: Run it concurrently and gate on the worst claim

import asyncio

async def verify_answer(llm, answer: GroundedAnswer, chunks: dict[str, str]):
    # verify_claim is blocking, so run each call in a worker thread
    results = await asyncio.gather(*[
        asyncio.to_thread(verify_claim, llm, c.text, chunks, c.source_ids)
        for c in answer.claims
    ])
    failures = [
        (c, v) for c, v in zip(answer.claims, results) if v.label != "supported"
    ]
    return {
        "pass": not failures,
        "support_rate": 1 - len(failures) / max(len(answer.claims), 1),
        "failures": failures,
    }

Claims are independent, so latency is roughly one judge call, not N. Use a small, fast model for the judge; it is a classification task with a quote check as a backstop.

Step 4: Choose a failure policy

Policy	How it works	Best for	Cost
Drop	Remove failed claims, render the rest	Low-stakes summaries, search snippets	Free, but can leave gaps
Repair	Re-prompt with only supported claims + the failures flagged	Chat assistants	One extra LLM call
Abstain	Return "I can't confirm this from the documents"	Legal, medical, finance, policy	Lower coverage, highest trust
Flag	Show answer with unverified claims highlighted	Internal analyst tools	UI work

A reasonable default: repair once, then abstain. Never loop indefinitely; cap retries at one.

Where this fits: online vs. offline

You rarely want to run full claim verification on every request.

Mode	When	What you run
Offline eval	Every prompt/retriever change in CI	Full claim-level check on a golden set
Online sampling	A few percent of production traffic	Full check, results logged to dashboards
Online gating	High-stakes routes only	Full check blocking the response
Cheap online triage	All traffic	Citation-exists check, quote-in-source check, abstain-rate tracking

The cheap triage layer costs almost nothing: confirm every cited ID exists in the retrieved set, and confirm numbers and named entities in each claim appear in the cited span. This catches a surprising share of fabrications before you spend a single judge call.

Calibrate your judge before you trust it

An LLM judge is itself a model that makes mistakes. Before using it as a gate:

1. Hand-label 100 to 200 (claim, span) pairs from your own domain, including hard negatives such as near-miss numbers and negations.
2. Measure precision and recall of the "supported" label separately. For a gate, you usually care most about not passing unsupported claims.
3. Re-check whenever you swap the judge model or edit the rubric.

If this sounds like your LLM-as-judge setup, the calibration approach is the same one you would use for any other rubric-based eval.

Common pitfalls

• Verifying against the whole retrieved set. A claim passes because any chunk supports it, hiding wrong citations. Check against the cited span.
• Non-atomic claims. Compound sentences let unsupported halves ride on supported ones.
• Similarity as a proxy for support. Negations, numbers, and dates are where embeddings fail.
• Ignoring retrieval failures. If the answer isn't in the retrieved chunks, no verifier will rescue it. Track retrieval recall separately, and treat a high abstain rate as a retrieval signal.
• Judging in the same model family and prompt as generation. Correlated errors. Use a different prompt, and ideally a different model, for verification.
• Rewarding abstention too little. If your UI or metrics punish "I don't know", the model learns to guess. Report abstain rate as a first-class metric.
• Skipping chunk boundaries. A fact split across two chunks can look unsupported in either. Cite multiple IDs per claim and verify against their union.
• Trusting log-probs. Low token probability is not a reliable hallucination signal in modern instruction-tuned models. Verify against sources instead.

Checklist

• Answers are generated as structured claims with chunk IDs
• Claims are atomic, one fact each
• Verification is entailment-based, against the cited span only
• Judge must return a verbatim evidence quote, enforced in code
• Gate on unsupported claims, not on an averaged score
• Failure policy defined: repair once, then abstain
• Judge calibrated on hand-labelled domain data
• Support rate, abstain rate and failure examples are logged and reviewed weekly

Next steps

1. Take 50 real production answers and run the verifier offline. Read every failure by hand; you will learn more from that hour than from any benchmark.
2. Add the cheap triage checks (ID exists, entities and numbers appear in span) to every request.
3. Put the full claim-level check into CI against your golden set so retriever and prompt changes can't silently reduce groundedness.

Tool and API details change quickly, so check your provider's documentation for current structured-output and batching options.

---

Want to build and evaluate production RAG systems with feedback from practitioners? Take a look at our AI Engineering cohort.