Record-and-Replay Testing for LLM Apps: Deterministic CI Without Mocking Everything

Every team shipping an LLM feature ends up in the same bind with tests. You have three options, and all of them hurt:

Call the real API in CI. Tests are slow, cost money on every push, and fail randomly because the model's output varies or the provider has a bad minute.
Hand-written mocks. Fast and free, but you're testing your imagination of the model. The mock returns clean JSON; production returns a code fence, a refusal, or a tool call with a missing argument.
No tests for the LLM path. Common, and how regressions in prompts, parsers and tool wiring reach users.

Record-and-replay (the "cassette" pattern, popularized by VCR in Ruby and vcrpy in Python) is a fourth option. You run the test once against the real model and save the exact HTTP request/response pair. Every later run replays the saved response instead of hitting the network. You get real model output, deterministic and free.

This post shows how to build it for an LLM app, how to key cassettes correctly, and the failure modes that make naive setups useless.

What replay tests actually verify (and what they don't)

Be precise about the claim. A replayed test verifies everything in your code that surrounds the model call:

Prompt construction: does the request body contain what you expect?
Response parsing, schema validation, retry and fallback logic
Tool-call dispatch: given this model response, does your agent call the right function with the right arguments?
Multi-step agent flows: the full sequence of calls, in order

It does not verify that the model still behaves well. If you change the prompt, the recording is stale by definition, and replay will either fail loudly (good) or return an answer to a different question (bad). Model quality is the job of evals; replay tests are the job of regression protection for your code. Keep the two separate. See our guide on evals for production agents for the quality side.

The core design: key on the request, not on the test name

The mistake most first implementations make is keying recordings by test name. Then you edit the prompt, the test still passes against the old recording, and you've learned nothing.

Key the cassette on a hash of the normalized request: model, messages, tools, temperature, and any other parameter that affects output. Change the prompt and the hash changes, the cassette misses, and the test tells you to re-record.

Normalize before hashing. Strip things that vary without meaning: request IDs, timestamps injected into the system prompt, API keys, trace headers.

A minimal implementation in Python

This wraps any function that takes a request dict and returns a response dict. It works with the OpenAI or Anthropic SDKs by wrapping your thin client layer (you should have one; see below).

import hashlib
import json
import os
from pathlib import Path

CASSETTE_DIR = Path("tests/cassettes")
MODE = os.environ.get("LLM_CASSETTE_MODE", "replay")  # replay | record | auto

VOLATILE_KEYS = {"user", "metadata", "request_id"}


def normalize(request: dict) -> dict:
    """Drop fields that change between runs but don't affect output."""
    return {k: v for k, v in request.items() if k not in VOLATILE_KEYS}


def cassette_key(request: dict) -> str:
    blob = json.dumps(normalize(request), sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]


class CassetteMiss(RuntimeError):
    pass


def with_cassette(real_call):
    """Wrap a function: request dict -> response dict."""

    def wrapped(request: dict) -> dict:
        path = CASSETTE_DIR / f"{cassette_key(request)}.json"

        if MODE in ("replay", "auto") and path.exists():
            return json.loads(path.read_text())["response"]

        if MODE == "replay":
            raise CassetteMiss(
                f"No recording for request {path.name}. "
                "Re-record with LLM_CASSETTE_MODE=record."
            )

        response = real_call(request)  # record or auto
        CASSETTE_DIR.mkdir(parents=True, exist_ok=True)
        path.write_text(
            json.dumps(
                {"request": normalize(request), "response": response},
                indent=2,
                ensure_ascii=False,
            )
        )
        return response

    return wrapped

Using it with the OpenAI Python SDK (check the official docs for the current method names and response shape for your SDK version):

from openai import OpenAI

client = OpenAI()


def _call(request: dict) -> dict:
    resp = client.chat.completions.create(**request)
    return resp.model_dump()


call_llm = with_cassette(_call)

Everything in your app goes through call_llm. That single seam is the most important architectural decision here: if your code scatters client.chat.completions.create across twenty files, you can't record anything cleanly.

Now a test:

def test_ticket_triage_routes_billing_to_finance():
    ticket = "I was charged twice for my March invoice."
    result = triage_ticket(ticket)   # internally builds a request, calls call_llm

    assert result.category == "billing"
    assert result.assigned_team == "finance"
    assert result.needs_human is False

First run, locally:

LLM_CASSETTE_MODE=record pytest tests/test_triage.py

Commit the generated tests/cassettes/*.json. CI runs in the default replay mode, with no API key at all. If a developer changes the prompt, CI fails with CassetteMiss, and the fix is to re-record and review the diff.

Recording tool calls and multi-step agents

Agents are where this pays off most. An agent loop makes N calls, and each depends on the previous response plus tool results. Because the key is a hash of the full request (which includes the conversation so far), each step gets its own cassette automatically. Replaying step 3 only works if steps 1 and 2 produced identical history, which is exactly the property you want: the test fails if your tool-result formatting changes.

Two requirements make this reliable:

Tools must be deterministic during replay. If a tool returns the current time or a random ID that gets fed back into the next request, the hash changes every run. Inject a fake clock and seeded ID generator in tests.
Stub the tools' external effects. Replay covers the LLM, not your database or search API. Use fixtures or fakes for those, or record them with the same mechanism.

Streaming responses

If you use streaming, record the sequence of chunks, not the assembled text, otherwise your stream-handling code (partial JSON, tool-call deltas, early cancellation) goes untested. Store the list of chunk dicts and replay them through the same iterator interface. Our streaming guide covers the edge cases worth asserting on.

Choosing an approach

Approach	Real model output	Deterministic	Cost per CI run	Catches prompt drift	Maintenance
Live API in CI	Yes	No	High	Sort of (flaky)	Low
Hand-written mocks	No	Yes	None	No	High, mocks rot
HTTP-level cassettes (`vcrpy`, etc.)	Yes	Yes	None	Yes, via request match	Medium
App-level cassettes (above)	Yes	Yes	None	Yes, via request hash	Medium
Evals on a golden set	Yes	Statistically	Moderate	Yes (quality)	Medium

HTTP-level libraries are quicker to adopt since they patch the transport, but they match on method, URI and sometimes body, and they're sensitive to SDK internals like added headers or changed retry behaviour. App-level wrapping, as above, is more stable across SDK upgrades because you control the request dict. Pick HTTP-level if you want zero code changes; pick app-level if you want control over normalization and are willing to have one seam.

A checklist for a healthy cassette suite

All model calls go through one function
Cassettes keyed on a normalized request hash, not test names
CI runs in strict replay mode with no API key present, so an accidental live call fails instead of billing you
Secrets and PII scrubbed before writing cassettes
Cassette files are committed and reviewed like snapshots
A separate scheduled job re-records a subset against the live model and flags diffs
Temperature and seed are fixed in test requests where the provider supports it

Common pitfalls

Committing secrets and PII. A cassette contains the full request, including whatever customer text you sent. Scrub headers and run a redaction pass on bodies before saving. Never record from production traffic without redacting; see our post on PII redaction in LLM pipelines.

Treating a green replay as evidence of quality. The recording captured one sample of a stochastic system. A test that passes forever on a lucky response says nothing about the other 99 samples. Use replay for code correctness, and run evals for quality.

Never re-recording. Providers deprecate and update models. If cassettes are two years old, you're testing against a model nobody serves. Schedule a periodic re-record (monthly is reasonable) and review the diff. Large semantic diffs are an early warning for your evals.

Non-deterministic prompts. If your system prompt embeds the date, a user ID, or a dict iterated in nondeterministic order, the hash changes on every run and every test is a miss. Sort keys, inject clocks, and move volatile values into the normalizer only if they genuinely don't affect the output.

Huge cassettes. Long-context requests produce multi-megabyte files. Store the request hash and response, and keep the full request only for small cases (or gzip them). Don't bloat your repo with 200 KB documents repeated per test.

Recording failures by accident. If you record during a provider outage, you'll save a 529 and replay it forever. Only persist successful responses, and add explicit failure-injection tests for retries instead; see our post on rate limits, retries and backpressure.

Next steps

Find every place your code calls an LLM SDK directly and funnel them through a single function.
Add the wrapper above, record your five most important flows, and commit the cassettes.
Turn on strict replay in CI with the API key removed from the environment.
Add a weekly job that re-records against the live model and posts the diff to your team channel.
Keep the quality question in your eval harness, not in these tests.

You'll get a CI pipeline that is fast, free, and fails for the right reasons. If you want to build this kind of testing and eval infrastructure hands-on, with real agent codebases and reviews, take a look at the Joinloop AI engineering cohort.