Shipping LLM Changes Safely: Shadow Mode, Canaries, and Online A/B Tests

Here's a familiar story. Someone tightens the system prompt, swaps to a cheaper model, or adds a retrieval step. The offline eval goes from 81% to 86%. The change ships to everyone on Tuesday. By Thursday, support tickets are up, the "regenerate" button gets clicked twice as often, and nobody can say which change caused it, because three other prompt edits went out the same week.

The offline eval didn't lie. It just answered a narrower question than the one you cared about. A golden dataset of 300 examples measures behaviour on the inputs you thought of. Production traffic contains the inputs you didn't think of: long-tail languages, pasted stack traces, users who type one word, and the edge cases that only show up at volume.

Traditional software solved this years ago with progressive delivery. LLM features need the same thing, plus a few changes, because LLM outputs are non-deterministic and "correct" is often a judgment call rather than a status code.

Why LLM changes need their own rollout discipline

Three properties make LLM changes riskier than ordinary code changes:

Small edits have wide effects. One sentence in a system prompt can change tone, length, refusal rate, and tool-call behaviour across every request. There's no equivalent of "this diff only touches the checkout page."
Failures are quiet. A bad deploy of normal code throws 500s. A bad prompt returns 200 OK with a slightly worse answer. Your error-rate dashboard stays green.
Your offline set drifts out of date. The golden set reflects last quarter's users. Meanwhile, the share of queries about the feature you launched last month has gone from zero to 15%.

So the question at each rollout step isn't "did it crash?" It's "is it at least as good on real traffic, at a cost and latency we accept?"

The rollout ladder

Think of it as four rungs. Each one exposes more users and gives you a stronger signal.

Stage	Users see new output?	Traffic	What it catches	Typical duration
Offline eval	No	Golden set	Known failure modes, regressions on curated cases	Minutes
Shadow	No	5–100% mirrored	Latency, cost, crash/parse rates, output drift on real inputs	1–3 days
Canary	Yes	1–5%	User-visible breakage, guardrail trips, complaint spikes	1–2 days
A/B test	Yes	50/50 (or 10/90)	Whether it's actually better on product metrics	1–3 weeks

Not every change needs every rung. Fixing a typo in a prompt can skip straight to canary. A model swap or a new retrieval pipeline should climb the whole ladder.

Rung 1: Shadow mode

In shadow mode, the new variant (the "challenger") runs on real requests in parallel with the current one (the "champion"). Users only ever see the champion's output. You log both outputs and compare them later.

The key rule is that the shadow path must never hurt the live path. It runs in the background, has its own timeout, and swallows its own exceptions.

import asyncio
import hashlib
import json
import logging
import time

from openai import AsyncOpenAI

client = AsyncOpenAI()
log = logging.getLogger("rollout")

VARIANTS = {
    "champion": {"model": "gpt-4.1", "prompt_version": "support-v12"},
    "challenger": {"model": "gpt-4.1-mini", "prompt_version": "support-v13"},
}
SHADOW_RATE = 0.20  # mirror 20% of traffic

PROMPTS = {
    "support-v12": "You are a support assistant. Answer concisely.",
    "support-v13": "You are a support assistant. Answer in under 120 words. Cite the doc you used.",
}


def bucket(user_id: str, salt: str) -> float:
    """Deterministic 0..1 bucket so a user always lands in the same group."""
    h = hashlib.sha256(f"{salt}:{user_id}".encode()).hexdigest()
    return int(h[:8], 16) / 0xFFFFFFFF


async def call_variant(name: str, messages: list[dict]) -> dict:
    cfg = VARIANTS[name]
    start = time.perf_counter()
    resp = await client.chat.completions.create(
        model=cfg["model"],
        messages=[{"role": "system", "content": PROMPTS[cfg["prompt_version"]]}, *messages],
    )
    return {
        "variant": name,
        **cfg,
        "output": resp.choices[0].message.content,
        "latency_ms": round((time.perf_counter() - start) * 1000),
        "input_tokens": resp.usage.prompt_tokens,
        "output_tokens": resp.usage.completion_tokens,
    }


async def run_shadow(request_id: str, messages: list[dict]) -> None:
    try:
        result = await asyncio.wait_for(call_variant("challenger", messages), timeout=30)
        log.info(json.dumps({"request_id": request_id, "shadow": True, **result}))
    except Exception as exc:  # the shadow path must never raise into the live path
        log.warning(json.dumps({"request_id": request_id, "shadow": True, "error": repr(exc)}))


_background: set[asyncio.Task] = set()


async def handle(request_id: str, user_id: str, messages: list[dict]) -> str:
    if bucket(user_id, salt="shadow-support-v13") < SHADOW_RATE:
        task = asyncio.create_task(run_shadow(request_id, messages))
        _background.add(task)  # keep a reference so the task isn't garbage-collected
        task.add_done_callback(_background.discard)

    live = await call_variant("champion", messages)
    log.info(json.dumps({"request_id": request_id, "shadow": False, **live}))
    return live["output"]

A few details matter here:

Deterministic bucketing (hash of user ID plus a salt) means the same user always gets the same treatment. Use a new salt for each experiment so you don't keep putting the same users in the treatment group every time.
Log the request_id on both sides so you can join champion and challenger outputs for pairwise comparison.
Shadow doubles your spend on the mirrored slice. At 20% shadow, your LLM bill goes up by roughly 20% times the challenger's cost ratio. Budget for it, or shadow a smaller slice.
Never shadow side effects. If the variant can call tools that send emails, write to a database, or charge cards, stub those tools out in the shadow path. Otherwise "shadow" means doing the action twice.

What to measure in shadow

Since users don't see the shadow output, you can't measure user behaviour. You can measure everything else:

Operational: p50/p95 latency, error and timeout rate, tokens and cost per request.
Structural: JSON parse failures, schema violations, tool-call validity, refusal rate, output length distribution.
Quality, pairwise: run an LLM judge on (input, champion output, challenger output) and ask which is better, or whether they're equivalent. Pairwise judgments are more stable than absolute 1–10 scores. Randomise which answer appears first, because judges have a position bias.

Pairwise win rate on real traffic is the single most useful shadow metric. If the challenger wins or ties on 92% of mirrored requests, look at the 8% losses before moving on. That slice usually shows you exactly which input types your golden set was missing. Add them to it.

Rung 2: Canary

A canary sends real users the challenger's output, but only a small share of them (1–5%). The aim isn't to prove the challenger is better. It's to catch anything that is obviously worse before it reaches more users.

Define kill criteria before you start, and automate them where you can:

from dataclasses import dataclass


@dataclass
class Guardrail:
    metric: str
    max_relative_increase: float  # e.g. 0.25 = challenger may be at most 25% worse
    min_samples: int


GUARDRAILS = [
    Guardrail("error_rate", 0.10, 500),
    Guardrail("p95_latency_ms", 0.25, 500),
    Guardrail("regenerate_rate", 0.15, 1000),
    Guardrail("thumbs_down_rate", 0.20, 300),
    Guardrail("safety_filter_trip_rate", 0.0, 1000),  # zero tolerance
]


def should_rollback(champion: dict, challenger: dict, n_challenger: int) -> list[str]:
    breaches = []
    for g in GUARDRAILS:
        if n_challenger < g.min_samples:
            continue  # not enough data yet; don't act on noise
        base, cand = champion[g.metric], challenger[g.metric]
        if base == 0:
            if cand > 0 and g.max_relative_increase == 0:
                breaches.append(f"{g.metric}: {cand:.4f} vs 0 baseline")
            continue
        if (cand - base) / base > g.max_relative_increase:
            breaches.append(f"{g.metric}: {cand:.4f} vs {base:.4f}")
    return breaches

Run this every 15–30 minutes during the canary. If anything breaches, flip the flag back to 0% and page whoever owns the change. The min_samples floor matters: at 1% traffic, the first hour of data is mostly noise, and auto-rollback on noise trains your team to ignore alerts.

Choose canary metrics that are behavioural proxies for quality: regenerate clicks, copy-to-clipboard, edits to the generated draft, abandonment mid-stream, escalation to a human. Thumbs-up and thumbs-down buttons are useful but sparse. Usually fewer than 2% of responses get any rating, and the people who rate skew negative.

Rung 3: The A/B test

The canary tells you the challenger isn't broken. The A/B test tells you whether it's better enough to justify the switch. This is where PMs should own the decision.

Write down three things before the test starts:

Primary metric: one metric you'll make the call on. For example, "task resolved without human escalation" for a support bot, or "suggestion accepted and kept after 24h" for a writing or coding assistant.
Guardrail metrics: metrics that must not get worse. Cost per resolved task, p95 latency, safety trips.
Minimum detectable effect and duration: how big a lift you care about, and how long you need to run to detect it.

That third point is where most LLM A/B tests go wrong. A quick sample-size check:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline = 0.62  # current resolution rate
target = 0.64    # smallest lift worth shipping (+2 pts)

effect = proportion_effectsize(target, baseline)
n_per_arm = NormalIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.8)
print(f"{n_per_arm:,.0f} conversations per arm")  # roughly 4,600 per arm

If your feature handles 1,500 conversations a day, a 50/50 split needs about 6 days. Round that up to two full weeks so weekday and weekend traffic are both covered. Detecting a 1-point lift instead of 2 takes about four times as many samples. That's longer than most teams expect. It's also why you shouldn't stop the test the moment the dashboard turns green. Checking repeatedly and stopping at the first significant result inflates false positives. If you need to check early, use a sequential testing method (most experimentation platforms offer one) instead of repeated fixed-horizon t-tests.

Randomise on the right unit

Assign variants per user (or per account for B2B), not per request. If one user sees both variants within a session, their experience is incoherent: tone changes mid-conversation, and a follow-up question goes to a model that never saw the earlier answer. Per-request randomisation also contaminates metrics like retention, which are measured per user anyway.

Who owns what

Decision	Owner	Input from
Which rungs a change must climb	Eng lead	PM, based on blast radius
Shadow pass criteria (win rate, cost, latency)	Eng	PM signs off on thresholds
Canary kill criteria	Eng + on-call	PM, support
A/B primary metric and MDE	PM	Data/analytics
Final ship / no-ship	PM	Eng flags operational risks
Adding shadow losses to the golden set	Eng	PM triages which matter

Pre-rollout checklist

Every variant is versioned: model ID, prompt version, retrieval config, tool set. Logged on every request.
Variant assignment is deterministic, per user, with a per-experiment salt.
Shadow path has its own timeout, swallows errors, and has side-effecting tools stubbed out.
Champion and challenger outputs can be joined on request_id.
Pairwise judge prompt randomises answer order and has been spot-checked against human labels.
Canary kill criteria are written down, with sample-size floors.
Rollback is a flag flip, not a deploy.
A/B primary metric, guardrails, MDE, and planned duration are documented before launch.
Only one experiment touches a given prompt surface at a time.

Common pitfalls

Shipping several changes at once. If a prompt edit, a model swap, and a new retriever ship together, a regression can't be attributed to any one of them. Either stack changes so each one climbs the ladder separately, or run a factorial test if you have the traffic for it.

Using the judge from your offline eval as the only shadow signal. If that judge was tuned on your golden set, it shares the same blind spots. Mix in structural checks and periodic human review of a random sample of disagreements.

Forgetting provider-side drift. If your champion points at a floating alias like gpt-4.1 or claude-sonnet-latest, the provider can change it under you. Pin dated model snapshots where your provider offers them, so the champion stays the same during an experiment. Check your provider's docs for which identifiers are pinned.

Comparing cost per request instead of cost per outcome. A cheaper model that needs 1.4 regenerations per task isn't cheaper. Divide spend by resolved tasks, accepted suggestions, or whatever your primary metric counts.

Novelty effects. Users click more on anything new for the first few days. If the lift is concentrated in days 1–3 and gone by day 7, it was novelty, not an improvement. Plot the effect over time, not just the aggregate.

Leaving the shadow running forever. Shadow traffic costs real money. Put an end date on every experiment flag and remove dead variants from the code once a decision is made.

Next steps

If you do nothing else this week: add model, prompt_version, and variant fields to every LLM log line, and move your current prompt behind a feature flag. Those two changes make every later rung possible. With them in place, your next model swap can go through shadow mode first. You'll probably find a category of inputs your offline eval never covered.

Rollouts, evals, and the PM-engineering handoff around them are core to how we teach production AI at Joinloop. Engineers can go deeper in the AI engineering cohort. If you're the PM who has to make the ship call, the PM cohort covers experiment design for AI features end to end.