A/B Testing LLM Features: Sample Sizes, CUPED, and Guardrail Metrics

You changed the prompt, swapped the model, or added a reranker. Offline evals say it is better. Now someone asks the only question that matters: does it move the product metric? Teams usually answer by running a two-week A/B test, seeing a +2% lift with a p-value under 0.05, and shipping. A month later the lift is gone, or the bill has doubled, or support tickets rose.

LLM features break several assumptions that classic A/B testing relies on. This post covers what changes, how to size the test, how to reduce variance with CUPED, which guardrails to set, and a checklist you can apply to your next experiment.

Why LLM experiments are noisier

A normal A/B test compares two deterministic treatments. Variance comes from users. With an LLM feature you add several more sources:

• Output variance. The same input can yield different outputs, even with temperature set low. Floating-point non-associativity, batching, and provider-side load balancing mean "deterministic" settings are not a guarantee. Measure it for your task rather than assuming.
• Judge variance. If you score quality with an LLM judge, the judge adds its own noise. See our post on LLM-as-judge calibration.
• Treatment drift. Providers update models behind an alias. Your treatment can change mid-experiment, and the control can too if both call the same endpoint.
• Heterogeneous effects. A change that helps short, easy queries can hurt long, ambiguous ones. The average hides it.
• Novelty effects. Users click on AI features more in the first days. Early engagement lifts often decay.

The practical consequence: you typically need more samples than for a deterministic feature, and you need to decide the metrics, stopping rule, and guardrails before looking at results. Published guidance on the exact multiplier varies widely, so measure your own variance instead of trusting a rule of thumb.

Step 1: Pick metrics in three tiers

Tier	Purpose	Examples	Decision rule
Primary	The one thing you want to move	Task completion, resolution without escalation, accepted suggestion rate	Must improve with significance
Guardrail	Things that must not get worse	p95 latency, cost per request, refusal rate, thumbs-down rate, escalation rate	Must not degrade beyond a pre-set margin
Diagnostic	Explain why	Judge-scored quality, retrieval hit rate, tool-call error rate	No decision power, used for debugging

Teams that optimise one outcome metric with no guardrails routinely ship "improvements" that are net negatives. A more verbose answer can raise engagement and also raise token cost and reading time. Write the guardrail thresholds in the experiment doc, for example "cost per resolved task may not rise more than 10%".

Step 2: Size the test with measured variance

Do not guess. Run a short A/A period (both arms on the current system) or replay logged traffic, compute the standard deviation of your primary metric, and plug it into a power calculation.

from math import ceil
from statistics import NormalDist

def sample_size_per_arm(sigma: float, mde: float,
                        alpha: float = 0.05, power: float = 0.8) -> int:
    """Two-sided, two-sample test for a difference in means.

    sigma: standard deviation of the per-user metric (measure it from A/A data)
    mde:   minimum detectable effect, in the metric's own units
    """
    z_a = NormalDist().inv_cdf(1 - alpha / 2)
    z_b = NormalDist().inv_cdf(power)
    return ceil(2 * ((z_a + z_b) * sigma / mde) ** 2)

# Example: mean task score 0.62, sigma 0.30, we want to detect +0.02
print(sample_size_per_arm(sigma=0.30, mde=0.02))   # ~3532 users per arm
# If LLM output noise inflates sigma to 0.40:
print(sample_size_per_arm(sigma=0.40, mde=0.02))   # ~6280 users per arm

Notice how sensitive this is: raising sigma from 0.30 to 0.40 nearly doubles the required sample, because the requirement scales with sigma squared. If your traffic cannot reach that number in a reasonable time, either accept a larger MDE or reduce variance (next step).

For binary metrics such as resolved or not, use the proportion version of the formula, or a library such as statsmodels.

Step 3: Cut variance with CUPED

CUPED (Controlled-experiment Using Pre-Experiment Data) subtracts the part of each user's metric that is predictable from their behaviour before the experiment. It does not bias the estimate, because pre-experiment data is independent of treatment assignment. Reported variance reductions vary by metric and by how correlated pre-period behaviour is with the outcome, so test it on your own data.

import numpy as np

def cuped_adjust(y: np.ndarray, x_pre: np.ndarray) -> np.ndarray:
    """y: metric during experiment, x_pre: same user's metric before it.

    Compute theta on the pooled data (both arms) so it is not tied to treatment.
    """
    theta = np.cov(y, x_pre, ddof=1)[0, 1] / np.var(x_pre, ddof=1)
    return y - theta * (x_pre - x_pre.mean())

def lift(y_ctrl, y_treat):
    from scipy import stats
    diff = y_treat.mean() - y_ctrl.mean()
    t = stats.ttest_ind(y_treat, y_ctrl, equal_var=False)
    return diff, t.pvalue

# y, x_pre, arm come from your warehouse: one row per user
y_adj = cuped_adjust(y, x_pre)
print("raw      ", lift(y[arm == 0], y[arm == 1]))
print("adjusted ", lift(y_adj[arm == 0], y_adj[arm == 1]))

Pick a covariate that is strongly correlated with the outcome: the same metric from the prior 2 to 4 weeks, prior sessions, or prior AI feature usage. New users have no history, so either exclude them from the CUPED analysis or report them as a separate segment.

Variance from the model itself is harder to remove in a live test. If you can afford it offline, average several samples per prompt when evaluating. In production, you mainly control it by lowering temperature where quality allows, pinning model versions, and increasing sample size.

Step 4: Randomise carefully

• Randomise by user, not by request. Otherwise a single user sees both variants and the conversation context leaks between arms.
• Pin the model version in both arms (dated snapshot, not "latest"). Log it per request so you can detect silent provider changes.
• Cache isolation. If you use semantic caching, keep separate caches per arm or the control can serve treatment answers.
• Log everything per request: arm, model version, prompt version, tokens, latency, tool calls, refusal flag. You cannot diagnose a surprising result without it.

Before exposing real users, validate the treatment with shadow mode and canary rollouts. The A/B test then answers a value question, not a safety one.

Step 5: Decide how you will stop

Checking a dashboard daily and stopping when p dips below 0.05 inflates false positives badly. Choose one of:

Approach	How it works	Use when
Fixed horizon	Pre-compute sample size, look once at the end	Simple, traffic is predictable
Sequential testing	Always-valid p-values or alpha-spending boundaries	Stakeholders will peek, or you want early stopping on harm
Bayesian with decision threshold	Posterior probability of lift above a threshold, with expected loss	Team is comfortable with the framework and pre-registers thresholds

Whatever you pick, run at least one full weekly cycle so weekday and weekend behaviour are both represented, and long enough that novelty effects fade. Compare week one with week two lift as a sanity check.

Step 6: Read results by segment, with caution

Break the primary and guardrail metrics down by query type, language, user tenure, and input length. Heterogeneity is where LLM changes often hide regressions. But slicing creates multiple comparisons, so treat segment findings as hypotheses for a follow-up test unless you pre-registered them and corrected for multiplicity.

A quick readout template:

Metric	Control	Treatment	Delta	95% CI	Verdict
Resolution rate (primary)	61.8%	63.1%	+1.3 pts	[+0.4, +2.2]	Win
Cost per request	$0.0041	$0.0049	+19%	[+15%, +23%]	Guardrail breach
p95 latency	3.1 s	3.4 s	+0.3 s	[+0.1, +0.5]	Within margin
Refusal rate	2.2%	2.4%	+0.2 pts	[-0.1, +0.5]	Inconclusive

This (illustrative) result is a win on the primary metric and a failure on a guardrail. The honest decision is "do not ship as is": look for a cheaper variant, or decide explicitly that the cost is worth it.

Common pitfalls

1. Treating offline eval wins as product wins. Offline scores measure the task you wrote tests for. Users do other things.
2. No A/A test. Without it you do not know your noise floor or whether assignment is broken.
3. Floating model aliases. The model changes under you and you attribute the shift to your prompt.
4. Judge-only metrics. If the only metric is an LLM judge score, you are testing what the judge likes. Anchor on a behavioural metric.
5. Ignoring the cost and latency arms. Both are real product metrics for AI features.
6. Peeking. Stopping on the first significant reading.
7. Underpowered tests read as "no effect". A flat result with a wide interval means unknown, not zero.
8. Contaminated arms. Shared caches, shared conversation memory, or users switching devices.

Pre-launch checklist

• Primary metric, guardrails, and diagnostics written down before launch
• Sigma measured from A/A or replay data; sample size computed
• MDE agreed with the PM and is worth the traffic required
• Randomisation by user; model and prompt versions pinned and logged
• Caches and memory isolated per arm
• Stopping rule chosen (fixed horizon or sequential)
• CUPED covariate selected, new-user handling decided
• Rollback plan and harm threshold for early stop

Next steps

Start with the cheapest step: run an A/A test on your current LLM feature this week and compute the variance of your primary metric. That single number tells you whether your planned experiments are even feasible. Then write the experiment doc with the three metric tiers before you build the variant.

Statistical libraries and experimentation platforms change, so check the official documentation of whatever tooling you use for exact function behaviour.

If you want to learn how to design evals, experiments, and production gates for AI features through hands-on reviews, explore the Joinloop PM cohort or, if you are building these systems yourself, the engineering cohort.