Faithful Long-Document Summarization: Map-Reduce, Rolling Summaries and Claim Checks

Most teams first hit long-document summarization with a single prompt: paste the contract, ask for a summary, ship it. That works until the document is 300 pages, or until someone asks why the summary said the liability cap was $2M when the contract says $200K. The model did not refuse. It produced something plausible, and nothing in the pipeline checked it.

This post covers the three patterns you will actually use for long documents, how to choose between them, and a verification step that catches the errors a summary cannot see in itself. The code uses the Anthropic Python SDK. Check the current model IDs and parameter names against the official docs before you deploy.

The problem has two parts

A long document creates two different failures:

Coverage loss. The document does not fit, or it fits but the model attends unevenly. Details in the middle get dropped. This is the "lost in the middle" effect, and it gets worse as the input grows.
Faithfulness loss. The model writes a fluent sentence that is not supported by the text. Numbers, dates, names, and negations are the usual casualties. A "not" dropped from "shall not exceed" is the worst case.

Map-reduce and rolling summaries mostly address coverage. Claim checks address faithfulness. You usually need both.

Pattern 1: Map-reduce

Split the document into chunks, process each chunk independently (the map step), then combine the results (the reduce step). Every chunk gets the model's full attention, and the reduce step only sees extracted notes, not the raw text.

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-sonnet-5-5"  # confirm the current ID in the docs


def llm(system: str, user: str, max_tokens: int = 1024) -> str:
    resp = client.messages.create(
        model=MODEL,
        max_tokens=max_tokens,
        system=system,
        messages=[{"role": "user", "content": user}],
    )
    return resp.content[0].text


def chunk_text(text: str, max_chars: int = 12_000, overlap: int = 800) -> list[str]:
    """Split on paragraph breaks where possible, with a small overlap so a
    sentence cut at a boundary still appears whole in one chunk."""
    chunks, start = [], 0
    while start < len(text):
        end = min(start + max_chars, len(text))
        if end < len(text):
            cut = text.rfind("\n\n", start + max_chars // 2, end)
            if cut != -1:
                end = cut
        chunks.append(text[start:end])
        if end >= len(text):
            break
        start = end - overlap
    return chunks


MAP_SYSTEM = """You extract facts from one section of a longer document.
Return bullet points only. Each bullet is a claim stated in the section,
followed by a verbatim quote of at most 25 words in double quotes that supports it.
If the section has nothing relevant to the question, return exactly: NONE"""

REDUCE_SYSTEM = """You combine extracted notes into one answer.
Use only the notes. Keep the quote attached to each claim you use.
If notes conflict, say so explicitly and show both. If the notes do not
answer the question, say that instead of guessing."""


def map_reduce(text: str, question: str) -> str:
    chunks = chunk_text(text)
    notes = [
        llm(MAP_SYSTEM, f"Question: {question}\n\nSection:\n{c}")
        for c in chunks
    ]
    relevant = [n for n in notes if n.strip() != "NONE"]
    joined = "\n\n---\n\n".join(relevant) or "(no relevant notes found)"
    return llm(REDUCE_SYSTEM, f"Question: {question}\n\nNotes:\n{joined}", max_tokens=2048)

Why it works: the map calls are independent, so you can run them in parallel and cap concurrency to respect rate limits. Asking for verbatim quotes at the map step gives you something to check later. The reduce step cannot introduce a fact that no map call produced, because it never sees the source.

Where it breaks: if a fact is split across two chunks, neither map call sees the whole thing. The overlap helps for sentences, not for an argument that spans pages. A second failure: the reduce prompt can still be too large if you have hundreds of chunks. Add a second reduce layer (reduce groups of 20 notes, then reduce those) when the joined notes exceed your budget.

Pattern 2: Rolling summary

Process chunks in order and carry a running state forward. Each step updates the notes given the new section.

ROLL_SYSTEM = """You maintain running notes on a document being read in order.
Given the current notes and a new section, return the updated notes.
Rules: keep every number, date, name, and negation from the notes. Remove
only what the new section makes obsolete. Stay under 800 words."""


def rolling_summary(text: str, question: str) -> str:
    notes = ""
    for chunk in chunk_text(text):
        notes = llm(
            ROLL_SYSTEM,
            f"Question we are answering: {question}\n\n"
            f"Current notes:\n{notes or '(empty)'}\n\n"
            f"New section:\n{chunk}",
            max_tokens=1200,
        )
    return notes

Why it works: the final state reflects the whole document in order, which suits narratives, meeting transcripts, and anything where later sections revise earlier ones. It also produces a single coherent output without a reduce step.

Where it breaks: it is sequential, so it is slow. Errors compound: a dropped detail in step 3 is gone by step 30, and the model rewrites its own notes each turn, so small drifts accumulate. The "keep every number, date, name, and negation" rule is there to slow that drift. It does not stop it.

Choosing between them

Consideration	Map-reduce	Rolling summary
Latency	Low (parallel map)	High (sequential)
Cost	Predictable, one call per chunk plus reduce	One call per chunk, but the state grows
Cross-section reasoning	Weak; depends on reduce	Moderate; state carries context
Error drift	Bounded; each chunk is independent	Compounds over the document
Traceability	High; quotes map to chunks	Low; state is rewritten
Best for	Question answering, extraction, audits	Narratives, transcripts, ordered events

For contracts, research papers, and compliance documents, default to map-reduce with quotes. You want traceability more than narrative flow. Use rolling summaries when the order of events is the point.

The step most teams skip: claim checking

A summary can be fluent and wrong, and the model that wrote it will not reliably notice. Verify it with a separate call that sees only the claims and the source:

CLAIM_SPLIT = "Split the text into atomic factual claims. One claim per line. No numbering, no commentary."

VERIFY_SYSTEM = """You check claims against a source document.
For each claim, output one line in this exact format:
<LABEL> | <claim>
LABEL is SUPPORTED if the source states it, CONTRADICTED if the source
states something incompatible, and UNSUPPORTED if the source does not
address it. Judge only from the source. Do not use outside knowledge."""


def check_claims(source: str, answer: str) -> list[str]:
    claims = llm(CLAIM_SPLIT, answer)
    report = llm(
        VERIFY_SYSTEM,
        f"Source:\n{source}\n\nClaims:\n{claims}",
        max_tokens=2048,
    )
    return [line for line in report.splitlines() if line.strip()]


def flagged(report: list[str]) -> list[str]:
    return [line for line in report if not line.startswith("SUPPORTED")]

Three details matter here:

The verifier gets the source, not the summary. Judging the summary against itself proves nothing.
Atomic claims. "Revenue grew 12% and churn fell" is two claims. A verifier that gets one compound claim will often mark it supported because half of it is.
Fail closed. Anything not labelled SUPPORTED goes to a human or gets blocked, not silently shipped.

If the source does not fit in one call, retrieve the relevant chunks per claim (the same retrieval you would use for RAG) and verify each claim against those. This is cheaper and usually more accurate than passing the whole document.

Verification is also not perfect. It catches inventions and number swaps well. It is weaker at subtle inference, such as a summary that is technically supported but misleading. For high-stakes output, sample a percentage of summaries for human review and track the flag rate over time. A rising flag rate after a model or prompt change is the early warning you want.

Common pitfalls

Chunking by character count alone. You will cut tables and lists in half. Split on headings or paragraph breaks where you can, and keep tables whole.
Asking for "a summary" with no question. The map step has no basis for deciding what is relevant. Always pass the question or the output schema down to every stage.
No quotes. Without verbatim quotes you have no cheap way to check the map output, and the reduce step has nothing to point back to.
Trusting the reduce step. It is the most fluent stage and the least checked. Run the claim check on its output, not on the notes.
Comparing pipelines on one document. Test on 20 to 30 documents you have labelled with the facts that matter. Measure recall of key facts and the flag rate. A single demo doc tells you nothing about either.

Next steps

Build a small eval set: 20 documents, each with 5 to 10 facts a correct summary must contain, and 2 to 3 traps (a negation, a changed number, a date that shifts).
Implement map-reduce first, since it gives you quotes and parallelism.
Add check_claims and report the flag rate per document.
Only switch to rolling summaries if your eval shows that cross-section order matters for the task.
Read the current structured-output and context-window guidance in the official Anthropic docs before tuning chunk sizes, since limits and recommendations change.

If you want to build this kind of pipeline with a team and get feedback on your evals, the Joinloop AI cohort runs hands-on sessions on exactly these production LLM patterns.