pass@k vs pass^k: Why Your Agent Eval Needs Consistency, Not Just Capability

Your agent passes 80% of the eval suite. You ship it. Two days later, a support ticket comes in: the same customer asked the same question twice and got two different refund policies. Both answers were "correct" in the sense that the agent could produce them, and your eval would have passed either run.

This is the gap between capability and reliability. A single eval run tells you the agent can solve a task. It does not tell you whether it solves it every time a real user shows up. For agents, which make many tool calls and sample at every step, that gap is large and it compounds.

The fix is to measure two different numbers and be deliberate about which one gates a release.

The two metrics

Run each task n times. Count the successes c. Then:

• pass@k is the probability that at least one of k attempts succeeds. It answers: "If I let the system retry, how often does it eventually get there?"
• pass^k (read "pass hat k") is the probability that all k attempts succeed. It answers: "If a user hits this task k times, how often does the agent get it right every time?"

pass@k comes from code generation research, where the Codex paper (Chen et al., 2021) popularised an unbiased estimator for it. pass^k was formalised for agents in the τ-bench paper (Yao et al., 2024), which evaluates tool-using agents against simulated users and domain policies. Their key finding is the one to internalise: consistency collapses as k grows. Models that look strong at pass^1 (which is just average reward) can fall dramatically at pass^8.

The two metrics move in opposite directions in terms of what they reward:

Metric	Question it answers	Rewards	Use it for
pass@1	How often does one attempt succeed?	Raw capability	Tracking model progress
pass@k	Can it succeed somewhere in k tries?	Diversity, retries	Systems with automatic retry or best-of-k selection
pass^k	Does it succeed every time in k tries?	Consistency	Release gates for user-facing agents

If your product shows one answer to one user and that answer moves money, changes an account, or sends an email, pass@k is flattering you. You care about pass^k.

Why agents make this worse than chatbots

A single LLM call has one sampling step. An agent has a loop: plan, call a tool, read the result, decide, call another tool, write the answer. Each step is a fresh chance to diverge. If each step is 95% reliable and a task takes ten steps, the naive independent-steps estimate of end-to-end reliability is 0.95^10 ≈ 60%. Real agents are not independent across steps, so the true number is messier, but the direction holds. Length multiplies variance.

There is a second source of noise that people forget: the user simulator. If your eval uses an LLM to play the customer, and that simulator is itself stochastic, variance across trials mixes agent inconsistency with simulator inconsistency. The fix is to pin the simulator (temperature 0, fixed seed where the provider supports it) or to log simulator turns and treat them as part of the test fixture.

Estimating both from a few trials

You do not need thousands of runs. You need n strictly greater than or equal to k, and enough tasks that the average is stable. A useful default is n = 8 trials per task, with k in {1, 2, 4, 8}.

The estimators are short:

from math import comb


def pass_at_k(n: int, c: int, k: int) -> float:
    """P(at least one of k attempts succeeds), given c successes in n trials."""
    if n - c < k:
        # Fewer failures than attempts: a success is guaranteed to appear.
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)


def pass_hat_k(n: int, c: int, k: int) -> float:
    """P(all k attempts succeed), given c successes in n trials."""
    if c < k:
        return 0.0
    return comb(c, k) / comb(n, k)


def summarize(results: dict[str, list[bool]], k: int) -> dict[str, float]:
    """results maps task_id to a list of pass/fail outcomes, one per trial."""
    if not results:
        raise ValueError("no results to summarize")

    at_k, hat_k = [], []
    for trials in results.values():
        n, c = len(trials), sum(trials)
        if n < k:
            raise ValueError(f"need at least k={k} trials per task, got {n}")
        at_k.append(pass_at_k(n, c, k))
        hat_k.append(pass_hat_k(n, c, k))

    return {
        f"pass@{k}": sum(at_k) / len(at_k),
        f"pass^{k}": sum(hat_k) / len(hat_k),
    }


if __name__ == "__main__":
    # One task, 8 trials, 6 successes.
    results = {"refund-duplicate-charge": [True] * 6 + [False] * 2}
    for k in (1, 2, 4, 8):
        print(summarize(results, k))

Run it and you get the numbers that matter for this one task. With 6 successes out of 8:

• pass@1 = 0.75 and pass^1 = 0.75. Identical at k = 1, by definition.
• pass@4 = 1.0. With only 2 failures, any 4 attempts must include a success.
• pass^4 ≈ 0.21. Only 15 of the 70 four-attempt combinations are all successes.
• pass^8 = 0.0. You never saw 8 successes in a row, so the estimate is zero.

Read that again. The same agent looks perfect under retry (pass@4 = 1.0) and mediocre under repetition (pass^4 ≈ 0.21). Which one you report decides whether you ship.

Wiring it into a trial runner

The estimator is the easy part. The harness needs to run each task several times with independent sessions, so state from one run does not leak into the next:

import asyncio
from collections.abc import Awaitable, Callable

TrialFn = Callable[[str, int], Awaitable[bool]]


async def run_trials(
    tasks: list[str],
    trial: TrialFn,
    n: int = 8,
    concurrency: int = 8,
) -> dict[str, list[bool]]:
    sem = asyncio.Semaphore(concurrency)

    async def one(task: str, seed: int) -> bool:
        async with sem:
            # Each call must build a fresh agent, tool sandbox and simulator.
            return await trial(task, seed)

    results: dict[str, list[bool]] = {t: [] for t in tasks}
    jobs = [(t, i) for t in tasks for i in range(n)]
    outcomes = await asyncio.gather(*(one(t, i) for t, i in jobs))
    for (task, _), ok in zip(jobs, outcomes):
        results[task].append(ok)
    return results

Two rules keep this honest:

1. Fresh state per trial. A shared database or a cached tool result makes the trials dependent, and pass^k assumes independence. Reset the sandbox, or use a per-trial copy of the fixture.
2. Grade the outcome, not the transcript. Use a deterministic check where you can: did the refund row exist, did the email go to the right address. The trajectory-level checks covered in Trajectory Evals are useful for diagnosing why a trial failed, but the pass/fail bit should come from state.

Turning it into a release gate

Once you have the numbers, pick the gate by risk, not by habit:

Surface	Gate	Example threshold
Read-only answers (FAQ, search)	pass@1 plus a quality score	Average pass@1 at or above your baseline
Write actions (refunds, bookings, edits)	pass^k at the k you expect users to hit	pass^4 at or above 0.9 before enabling by default
Retry-wrapped workflows	Both, with pass@k for the retry budget	pass@2 high, pass^2 tracked as a regression signal

The thresholds above are illustrative. Set your own from the cost of a wrong action. A wrong FAQ answer is an annoyance. A wrong refund is a support escalation and a chargeback.

Track pass^k over time as a regression metric in its own right. A model upgrade that lifts pass@1 by 3 points and drops pass^4 by 10 points is a regression for most products, and a single-number eval will not show it.

Common pitfalls

• Too few trials. With n = 3, pass^3 is almost always a coin flip between 0 and 1 per task. Use n >= 8 for anything you gate on. For high-stakes flows, go higher.
• Averaging over tasks with very different difficulty. A few trivially easy tasks can hide a hard one that always fails. Report the per-task breakdown, and look at the worst tasks, not just the mean.
• Non-independent trials. Shared caches, a warm vector index that mutates between runs, or a rate-limited sandbox that returns different errors later in the batch all violate the assumption. Check that trial 8 is not systematically different from trial 1.
• Temperature 0 as a fix. Setting temperature to 0 reduces sampling variance but does not remove it: provider-side nondeterminism, tool latency, and batching all still move outputs. Measure, do not assume.
• Conflating simulator and agent variance. If the user simulator is stochastic, a low pass^k may be the simulator's fault. Pin the simulator, then re-measure.
• Reporting pass@k as reliability. Retry loops make pass@k look great. If the retry is invisible to the user, the user experiences the pass^k number on the first attempt.

Next steps

If you have an existing eval suite, do three things this week:

1. Pick the 10 tasks that correspond to real writes or money movement. Run each n = 8 times with fresh state.
2. Compute pass@1, pass@4 and pass^4 with the functions above. Write all three into the eval report, not just the first.
3. Find the single task with the lowest pass^4. Read five of its failing transcripts. You will usually find one tool with ambiguous arguments or one policy the agent interprets two ways. That is a tool-design fix, not a model fix.

The reliability work is mostly unglamorous: deterministic checks, fresh state, enough trials, and honest reporting. It is also the part that separates a demo from a product your users can trust twice in a row.

If you're building and shipping AI agents and want to go deeper on evals, tool design and production reliability with a cohort of engineers doing the same, take a look at the Joinloop engineering cohort.