Prompt Caching in Production: Cut LLM Cost and Latency Without Touching Your Model

Your agent sends the same 8,000-token system prompt, tool schema, and retrieved context on every turn of a conversation. The model re-reads all of it, every time, and you pay full input price for every token on every call. On a 20-turn conversation that is 160,000 input tokens billed for content that never changed after turn one.

Prompt caching fixes this. It is not semantic caching (matching similar user queries to stored responses) and it is not your own key-value store. It is a provider feature that lets the model reuse the computed internal state of a prompt prefix across calls, so you pay a fraction of the price and wait far less for the first token.

This post covers how it works on the major providers, how to structure a prompt so you actually hit the cache, what it costs, and the failure modes that cost teams money without them noticing.

The problem in numbers

A typical production agent call looks like this:

Segment	Tokens	Changes between calls?
System prompt + instructions	1,500	No
Tool / function schemas	2,000	No
Retrieved documents (RAG context)	4,000	Rarely
Conversation history	3,000	Grows each turn
New user message	200	Yes

Roughly 7,500 of those ~10,700 input tokens are identical to the previous call. Without caching you pay for all 10,700 at full rate and the model processes all of them before emitting a token. With caching, the stable prefix is served from cache at a large discount and the model skips re-processing it, so time-to-first-token drops.

How provider-native caching works

Transformer inference over a prompt produces a set of intermediate tensors (the attention key/value cache). For a fixed prefix, those tensors are deterministic. Providers persist them keyed by an exact hash of the token prefix. The next request whose prefix hashes identically reuses the stored state instead of recomputing it.

Three consequences follow directly from that mechanism:

It is prefix-only and exact-match. The cache hit covers the longest common token prefix between your request and a cached one. One different token anywhere breaks the cache from that point forward.
Order matters absolutely. Put stable content first, volatile content last. A timestamp near the top of your system prompt invalidates everything after it.
There is a minimum size. Providers set a floor (commonly ~1,000+ tokens) below which caching does nothing.

Anthropic (Claude)

You explicitly mark cache breakpoints with cache_control on a content block. Everything from the start of the prompt up to and including that block becomes cacheable. You get up to four breakpoints.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5",  # check the model catalog for current IDs
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_STATIC_INSTRUCTIONS,   # 1,500 tokens, never changes
            "cache_control": {"type": "ephemeral"},
        },
        {
            "type": "text",
            "text": retrieved_context,          # stable within a session
            "cache_control": {"type": "ephemeral"},
        },
    ],
    messages=conversation,                       # volatile, no cache_control
)

print(response.usage)
# cache_creation_input_tokens: 3500  (first call, written to cache)
# cache_read_input_tokens: 0
# on the next call within the TTL:
# cache_creation_input_tokens: 0
# cache_read_input_tokens: 3500      (served from cache, ~0.1x price)

Default cache lifetime is short (on the order of a few minutes), refreshed on every hit. A longer TTL option exists at a higher write cost. Always read response.usage to confirm you are getting cache_read_input_tokens — if you see repeated cache_creation, your prefix is changing.

OpenAI

Caching is automatic for prompts above the size threshold. There are no breakpoints to set — the API caches prefixes of qualifying requests and reports usage.prompt_tokens_details.cached_tokens on the response. Cached input is billed at a reduced rate. Your job is purely to keep the prefix byte-stable and route repeat traffic so it lands on infrastructure that still holds the cache (the prompt_cache_key parameter helps steer this).

from openai import OpenAI

client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-4.1",  # verify current model IDs in the docs
    messages=[
        {"role": "system", "content": LONG_STATIC_INSTRUCTIONS},
        *conversation,
    ],
    prompt_cache_key="agent-v3-tenant-42",
)

print(resp.usage.prompt_tokens_details.cached_tokens)

Google Gemini

Two modes. Implicit caching works like OpenAI's — automatic, best-effort, discounted reads. Explicit caching via the CachedContent API gives you a handle to a blob of content with a TTL you control; you pass the handle instead of re-sending the content, and you pay a storage fee per hour. Explicit caching is the right call when you have a large fixed corpus (a codebase, a policy manual, a long video) hit by many requests.

Structuring prompts for cache hits

The single highest-leverage change: sort your prompt by volatility, ascending.

[ system role and rules        ]  <- never changes
[ tool / function definitions   ]  <- changes on deploy
[ few-shot examples             ]  <- changes on deploy
[ retrieved documents           ]  <- changes per session
[ cache breakpoint here         ]
[ conversation history          ]  <- changes per turn
[ current user message          ]  <- always new

Concrete rules that follow from this:

Never interpolate dynamic values into the prefix. No Current time: {now}, no Request ID: {uuid}, no User: {name} in the system prompt. Move per-request facts into the final user message or a trailing block.
Freeze serialization. json.dumps(tools) must produce identical bytes every call — same key order, same whitespace. Sort keys explicitly. A dict that iterates in a different order breaks the cache silently.
Version your prefix deliberately. When you do change the system prompt, you pay one cache-write across your fleet and then ride the new cache. Expected. Just do not do it accidentally on every request.
Keep RAG context stable within a session. If you re-rank or re-retrieve on every turn, the document block changes every turn and nothing after the system prompt caches. Retrieve once per session, or cache the retrieval, or accept a shorter cacheable prefix.
Put the breakpoint after the largest stable region, not in the middle of it.

What it costs

Approximate economics (check current pricing pages — these move):

Operation	Price relative to base input
Cache read	~0.1x (Anthropic), ~0.25–0.5x (OpenAI/Gemini implicit)
Cache write (short TTL)	~1.25x
Cache write (extended TTL)	~2x
Explicit cache storage (Gemini)	hourly fee per 1M tokens held

The write premium means caching is a loss if a prefix is used only once. It pays off from roughly the second hit within the TTL. For a chat session with 5+ turns, or a batch job reusing one context across hundreds of items, the savings are large — a 10,000-token static prefix hit 10 times goes from 100,000 billed input tokens to about 11,250.

Common pitfalls

Silent cache misses. Nothing errors when your cache never hits. You just pay full price and stay slow. Log cached_tokens / cache_read_input_tokens as a metric and alert on a low hit rate.
A dynamic token in the prefix. The classic: a logging middleware injects a trace ID into the system prompt, or a template renders the current date. Prefix hit rate goes to zero. Diff two consecutive raw request bodies byte-for-byte when debugging.
TTL expiry between turns. If a user takes 10 minutes to reply and the TTL is 5, the next turn is a cache write, not a read. For slow conversational flows, consider the extended-TTL option or a keep-warm ping.
Non-deterministic tool schema ordering. Building the tools array from a set or an unordered map. Sort it.
Load balancing away from the warm cache. On automatic-caching providers, requests routed to a different backend may miss. Use the provider's cache-key / routing hint to keep a tenant's traffic sticky.
Caching content that should not persist. The cache holds your prompt prefix on provider infrastructure for the TTL. Understand the provider's retention and isolation terms before caching sensitive context; keep genuinely sensitive per-user data out of the cached prefix.
Over-splitting breakpoints. On Anthropic you have four. Spending them on tiny regions instead of the few large stable boundaries wastes them.
Assuming it helps output cost. Caching discounts input tokens only. Output is billed normally. If your calls are output-heavy, the win is smaller.

Rollout checklist

Next steps

Start with the highest-traffic agent you run. Instrument the cache token counts first so you have a before number, then reorder the prompt and set breakpoints. Multi-turn chat and large-context batch jobs give the fastest payback. Pair this with request-level tracing so a dropped hit rate shows up as an alert, not a surprise on the invoice.

Prompt caching is one of a handful of changes that cut both cost and latency at once with no quality tradeoff — the others being model routing, semantic caching for repeated queries, and tightening context. They compose well; a routed, prefix-cached, context-trimmed pipeline can run at a fraction of a naive implementation's cost.

Working through these tradeoffs on a real system is most of what our engineering cohort covers — production LLM systems, evals, cost and latency, and the operational glue around them. If you are building with LLMs and want to go deeper with a group doing the same, take a look at the Joinloop engineering cohort.