Long Context vs RAG: A Decision Framework With a Cost and Latency Calculator

Every few months a model ships with a bigger context window and someone declares RAG dead. Then the first invoice arrives. Or the first p95 latency chart. Or a user asks a question whose answer sits in the middle of a 400-page contract and the model confidently misses it.

The real question isn't "long context or RAG?" It's "for this corpus, this query pattern, and this budget, which one fails less often per dollar?" This post gives you a way to answer that in an afternoon instead of by opinion.

The problem: three variables people conflate

Teams argue about long context vs RAG as if it's one decision. It's actually three:

Capacity – does the data fit in the window at all?
Cost and latency – what do you pay on every request?
Accuracy – can the model reliably find and use the right piece?

A 1M-token window solves capacity. It does nothing for cost (you pay per input token, every call) and only partially solves accuracy (models degrade on retrieval-style tasks as the prompt grows, and multi-fact reasoning degrades faster than single-needle lookup). Check your provider's current docs for window sizes and pricing, because both change often.

How each approach actually behaves

Long context ("stuff it all in")

Zero retrieval infrastructure. No chunking, no embeddings, no index.
Model sees everything, so cross-document reasoning ("compare clause 4 in contract A with clause 9 in contract B") works naturally.
You pay for the full corpus on every call unless you use prompt caching.
Latency scales with input size: time-to-first-token on a few hundred thousand tokens is noticeably worse than on 5k.

RAG (retrieve then generate)

Pays for ~2–8k tokens per call regardless of corpus size.
Scales to corpora of any size, with incremental updates.
Adds failure modes: bad chunking, missed recall, stale index. If the retriever misses it, the model can't use it.
Needs evaluation of the retriever separately from the generator.

Hybrid ("retrieve wide, read long")

Retrieve generously (top 30–50 chunks or whole sections), then feed a large but bounded context.
Trades some cost for much better recall than tight top-5 RAG.

The decision framework

Answer these in order. Stop at the first decisive answer.

Does the corpus fit in the window with headroom? If it's larger than roughly half the window, go RAG or hybrid. Don't plan to run at the edge.
How often is the same corpus queried? Many queries over one stable corpus makes caching attractive, which makes long context cheap. One-off questions over changing documents favour RAG.
Is the answer local or global? "What's the refund window?" is local, so use RAG. "Summarise themes across all 200 tickets" is global, so use long context or map-reduce.
How fresh must data be? Minutes-fresh data usually needs an index you can update incrementally, so go RAG.
Do you need citations and access control per user? Per-user filtering is natural in a retriever (metadata filters, row-level security) and awkward when you stuff a shared prompt.

A cost and latency calculator

Before debating, compute. This script compares per-query cost for three strategies. Prices are inputs, so plug in your provider's current numbers rather than trusting the placeholders.

from dataclasses import dataclass

@dataclass
class Pricing:
    input_per_mtok: float          # $ per 1M uncached input tokens
    cached_input_per_mtok: float   # $ per 1M cached input tokens (read)
    output_per_mtok: float         # $ per 1M output tokens

def long_context_cost(corpus_tokens, question_tokens, output_tokens,
                      p: Pricing, cache_hit_rate: float) -> float:
    """Whole corpus in the prompt; cache_hit_rate is the share of calls
    where the corpus prefix is served from the prompt cache."""
    total_in = corpus_tokens + question_tokens
    cached = corpus_tokens * cache_hit_rate
    uncached = total_in - cached
    return (uncached * p.input_per_mtok
            + cached * p.cached_input_per_mtok
            + output_tokens * p.output_per_mtok) / 1_000_000

def rag_cost(retrieved_tokens, question_tokens, output_tokens, p: Pricing,
             embed_cost_per_query=0.000002, rerank_cost_per_query=0.001) -> float:
    llm = ((retrieved_tokens + question_tokens) * p.input_per_mtok
           + output_tokens * p.output_per_mtok) / 1_000_000
    return llm + embed_cost_per_query + rerank_cost_per_query

if __name__ == "__main__":
    # Placeholder prices - replace with your provider's current pricing.
    p = Pricing(input_per_mtok=3.0, cached_input_per_mtok=0.30, output_per_mtok=15.0)

    for corpus in (50_000, 200_000, 800_000):
        lc_cold = long_context_cost(corpus, 200, 500, p, cache_hit_rate=0.0)
        lc_warm = long_context_cost(corpus, 200, 500, p, cache_hit_rate=0.9)
        rag = rag_cost(6_000, 200, 500, p)
        print(f"corpus={corpus:>8,} tok | long-ctx cold ${lc_cold:.3f} "
              f"| long-ctx cached ${lc_warm:.3f} | RAG ${rag:.4f}")

Run it with your real numbers and your real query volume. Typical takeaways, which you should verify against your own pricing:

At small corpora (tens of thousands of tokens) with caching, long context is competitive and far simpler.
At hundreds of thousands of tokens, uncached long context is one to two orders of magnitude more expensive per query than RAG.
Caching narrows the gap but has a time-to-live. Spiky or low-volume traffic won't keep the cache warm, so model your actual hit rate, not the best case.

The comparison table

Dimension	Long context	RAG	Hybrid
Setup effort	Minimal	Chunking, embeddings, index, eval	Both
Per-query cost	High, unless cached	Low and flat	Medium
Latency	Grows with corpus	Retrieval adds ~100–500 ms, generation is fast	Medium
Cross-document reasoning	Strong	Weak with tight top-k	Good
Needle lookup recall	Good but degrades with size	Depends on retriever	Best of both
Freshness	Rebuild the prompt (and lose cache)	Incremental updates	Incremental
Per-user access control	Awkward	Natural (metadata filters)	Natural
Debuggability	"Why didn't it see X?" is hard	You can inspect what was retrieved	Inspectable
Corpus ceiling	The window	Effectively none	Effectively none

Measure accuracy instead of guessing

Cost is easy to compute. Accuracy needs a golden set. Build 50–100 real questions with known answers, tag each as local (one fact) or global (needs many places), and run both pipelines:

import json

def evaluate(pipeline, golden_path="golden.jsonl", judge=None):
    results = {"local": [], "global": []}
    for line in open(golden_path):
        row = json.loads(line)  # {"q":..., "answer":..., "kind": "local"|"global"}
        pred = pipeline(row["q"])
        ok = judge(row["q"], row["answer"], pred)   # LLM judge or exact match
        results[row["kind"]].append(ok)
    return {k: sum(v) / max(len(v), 1) for k, v in results.items()}

# Compare three pipelines on the same set:
# evaluate(long_context_pipeline), evaluate(rag_top5), evaluate(hybrid_top40)

Break results out by question kind. A common finding: RAG wins or ties on local questions at a fraction of the cost, while long context or hybrid wins on global ones. That tells you to route by query type rather than pick one globally. If you use an LLM judge, calibrate it first against human labels.

A hybrid pattern that works

For many production systems the best answer is neither extreme:

Retrieve generously: 30–50 chunks with a hybrid (BM25 + vector) search.
Rerank and keep a bounded budget, say 20–30k tokens.
Expand to parent sections so the model sees surrounding context.
Place the most relevant material at the start and end of the prompt, not buried in the middle.
Ask for citations pointing to chunk IDs, and verify them.

You get most of long context's recall with a cost ceiling you control.

Common pitfalls

Benchmarking on needle-in-a-haystack only. Single-fact lookup is the easy case. Test multi-hop and aggregation questions, since those degrade first.
Ignoring cache economics. A 90% cache hit rate in a demo becomes 20% with real, spiky traffic. Model the TTL.
Treating RAG as one number. If RAG underperforms, find out whether retrieval missed (recall problem) or the model ignored retrieved text (generation problem). They need different fixes.
Stuffing everything "just in case". More irrelevant tokens means more distraction, more cost, and slower responses. Relevance beats volume.
Forgetting per-user permissions. A shared giant prompt can leak documents a user shouldn't see. Retrieval with filters is safer by construction.
Locking in too early. Window sizes and prices keep moving. Keep your pipeline behind an interface so you can switch strategies when the numbers change.

Checklist

Corpus size in tokens, and its growth rate
Query volume and cache-hit estimate, measured not assumed
Share of local vs global questions
Freshness requirement
Access-control requirement
Golden set of 50+ questions, split by kind
Cost per successful answer (not just cost per query) for each strategy
Routing rule if the strategies win on different query types

Next steps

Start with the calculator and your real traffic numbers, then run the golden-set comparison. If the winners differ by query type, add a small router. If you're building the retrieval side, look at hybrid search, reranking, and retrieval evaluation, since those decide whether RAG or the hybrid approach holds up.

We work through exactly these trade-offs, with hands-on labs on real corpora, in the Joinloop engineering cohort. If you build AI systems for a living, take a look at the Joinloop cohort. If you're a PM who needs to make the cost-versus-quality call for your team, see the PM cohort.