Evaluating Multi-Turn Chatbots: Simulated Users, Scenario Tests, and Conversation-Level Metrics

Most LLM eval suites are a list of independent (input, expected output) pairs. That works for classification, extraction, and one-shot Q&A. It tells you almost nothing about a support bot, a coding assistant, or an onboarding copilot, because the failures that hurt users happen between turns:

The user says "I'm on the free plan" in turn 2, and in turn 7 the bot recommends a paid-only feature.
The user corrects the bot, and two turns later the bot repeats the original mistake.
The bot asks for the order number three times.
The bot never reaches a resolution and politely loops until the user gives up.

None of these show up if you grade each reply in isolation. This post covers how to build a multi-turn eval that does catch them, in three layers: scripted scenarios, simulated users, and conversation-level metrics.

Why single-turn evals miss conversation failures

A single-turn test fixes the history. In production, the history is whatever the model said before, including its own mistakes. Errors compound: a slightly wrong turn 3 becomes the context for turn 4. Three properties make multi-turn different:

State accumulates. Facts, constraints, and corrections from earlier turns must stay in force.
The user adapts. Real users rephrase, push back, and change goals mid-conversation. A static transcript cannot react to what your bot actually said.
Success is a trajectory property. "Was the issue resolved in a reasonable number of turns without violating policy?" is a question about the whole conversation.

Layer 1: Scripted scenarios with state assertions

Start with the cheapest layer. A scenario is a fixed sequence of user messages plus assertions you check at specific points. Because the user side is scripted, runs are cheap and deterministic enough to put in CI.

The key idea is to assert on behavior at turn N that depends on information from turn K, not on exact wording.

from dataclasses import dataclass, field
from typing import Callable

@dataclass
class Turn:
    user: str
    # Each check receives the bot reply and the full history so far.
    checks: list[Callable[[str, list[dict]], tuple[bool, str]]] = field(default_factory=list)

@dataclass
class Scenario:
    name: str
    turns: list[Turn]

def not_mentions(term: str):
    def check(reply: str, history: list[dict]):
        ok = term.lower() not in reply.lower()
        return ok, f"reply should not mention '{term}'"
    return check

def asks_at_most_once(slot_phrase: str):
    def check(reply: str, history: list[dict]):
        prior = sum(
            slot_phrase.lower() in m["content"].lower()
            for m in history if m["role"] == "assistant"
        )
        ok = not (prior >= 1 and slot_phrase.lower() in reply.lower())
        return ok, f"bot re-asked for '{slot_phrase}'"
    return check

free_plan_scenario = Scenario(
    name="respects_free_plan_constraint",
    turns=[
        Turn("Hi, I'm on the free plan and need to export my data."),
        Turn("What formats can I use?"),
        Turn("Can you turn on the scheduled export for me?",
             checks=[not_mentions("upgrade to pro to enable it automatically")]),
    ],
)

def run_scenario(scenario: Scenario, bot) -> list[str]:
    """bot(history) -> reply string. Returns a list of failure messages."""
    history, failures = [], []
    for i, turn in enumerate(scenario.turns, start=1):
        history.append({"role": "user", "content": turn.user})
        reply = bot(history)
        for check in turn.checks:
            ok, msg = check(reply, history)
            if not ok:
                failures.append(f"[{scenario.name}] turn {i}: {msg}")
        history.append({"role": "assistant", "content": reply})
    return failures

Keep the assertions as cheap as you can: string and regex checks first, then tool-call checks (did the bot call cancel_order without confirmation?), and only then an LLM judge for fuzzy requirements. Write scenarios from real incidents. Every escalated or thumbs-down conversation is a candidate; trim it to the minimum turns that reproduce the failure.

Layer 2: Simulated users

Scripted scenarios cannot react. If your bot asks a clarifying question you did not anticipate, the script's next line makes no sense. A simulated user is an LLM given a persona, a goal, and rules about when to stop, which then plays the user side against your bot.

import json
from anthropic import Anthropic  # any provider works; check current SDK docs

client = Anthropic()
SIM_MODEL = "claude-sonnet-5-5"  # use a model different from, or stronger than, your bot's

SIM_SYSTEM = """You are role-playing a customer talking to a support assistant.
Persona: {persona}
Hidden goal: {goal}
Hidden facts you know (reveal only if asked, one at a time): {facts}

Rules:
- Write one short message per turn, like a real person typing.
- Never reveal that you are simulated or that you have a goal list.
- If your goal is clearly achieved, reply with exactly: [DONE]
- If you are frustrated after 3 unhelpful replies, reply with exactly: [GIVE_UP]
"""

def simulate(bot, persona, goal, facts, max_turns=10):
    history = []  # from the bot's point of view: user/assistant
    sim_view = []  # from the simulator's point of view: roles flipped
    system = SIM_SYSTEM.format(persona=persona, goal=goal, facts=json.dumps(facts))

    user_msg = "Hi, I need some help."
    for turn in range(max_turns):
        history.append({"role": "user", "content": user_msg})
        reply = bot(history)
        history.append({"role": "assistant", "content": reply})

        sim_view.append({"role": "user", "content": reply})  # bot's reply is "input" to the sim
        out = client.messages.create(
            model=SIM_MODEL, max_tokens=200, system=system, messages=sim_view,
        ).content[0].text.strip()

        if out in ("[DONE]", "[GIVE_UP]"):
            return history, out
        sim_view.append({"role": "assistant", "content": out})
        user_msg = out
    return history, "[MAX_TURNS]"

Practical notes:

Vary personas deliberately: terse, rambling, impatient, non-native English, adversarial, someone who changes their mind. Ten well-chosen personas find more bugs than a hundred near-duplicates.
Hide the goal and the facts. If the simulator dumps everything in turn one, you are testing a form, not a conversation.
Use a different model family or a stronger model for the simulator where you can. A simulator that shares the bot's blind spots is too polite.
Run each (persona, goal) pair several times. Conversations are stochastic. Report rates, not single outcomes.
Treat the simulator as a test fixture that needs its own checks. Read a sample of transcripts to confirm it behaves like a plausible user. Simulated users are known to be more cooperative and more articulate than real ones, so never use them as the only signal.

Layer 3: Conversation-level metrics

Once you can generate conversations, score them as whole objects. A good starting set:

Metric	How to compute	What it catches
Task success rate	Final-state check (tool call made, ticket closed) or judge with a rubric	Did the user get what they came for?
Turns to resolution	Count of turns until `[DONE]`, median and p90	Verbosity, unnecessary clarification
Constraint retention	Scenario assertions at turn N for facts from turn K	Forgotten context, instruction drift
Repeat-question rate	Same slot requested more than once	Poor state tracking
Self-contradiction rate	Judge compares assistant claims across turns	Inconsistent facts
Loop/stall rate	Near-duplicate assistant replies (embedding similarity above a threshold)	Stuck conversations
Policy violation rate	Rule checks plus judge on every assistant turn	Unsafe or out-of-scope behavior
Recovery after correction	After a user correction, does the next reply comply and does it stay fixed for the next 3 turns?	Sycophantic apology followed by regression
Give-up rate	Share of runs ending in `[GIVE_UP]`	Overall frustration proxy

A simple loop detector that needs no extra model call:

from difflib import SequenceMatcher

def loop_rate(history: list[dict], threshold: float = 0.85) -> float:
    replies = [m["content"] for m in history if m["role"] == "assistant"]
    if len(replies) < 2:
        return 0.0
    near_dupes = sum(
        SequenceMatcher(None, a, b).ratio() >= threshold
        for a, b in zip(replies, replies[1:])
    )
    return near_dupes / (len(replies) - 1)

For LLM-judged metrics, give the judge the whole transcript and a narrow question ("List every place the assistant contradicts something it said earlier, quoting both turns"), and have it return structured output. Narrow questions with quoted evidence are far more reliable than "rate this conversation 1 to 10." Calibrate the judge against a few dozen human-labelled conversations before trusting it; see our guide on LLM-as-judge calibration.

Choosing the right layer

Approach	Cost	Determinism	Finds	Best used for
Scripted scenarios	Very low	High	Known regressions, constraint retention	CI on every PR
Simulated users	Medium (2 LLM calls per turn)	Low, needs repeats	Unknown unknowns, tone, dead ends	Nightly runs, pre-release
Replay of real conversations	Low	Medium	Distribution-realistic failures	Mining from production traces
Human review of samples	High	n/a	Everything the above miss	Weekly calibration

A reasonable setup: 30 to 100 scripted scenarios in CI, 20 to 50 simulated (persona, goal) pairs run several times nightly, and a weekly human read of 20 transcripts. Feed any new failure back into the scripted set so it never regresses.

Common pitfalls

Grading the final reply only. A conversation can end well after violating policy in turn 4. Score every assistant turn for hard rules.
A cooperative simulator. If success rate is 98% in simulation and 70% in production, the simulator is too nice. Add impatient and vague personas, and compare against real-conversation outcomes.
Comparing runs with different simulator randomness. When A/B testing two prompts, use the same seeds, personas, and run counts, and report confidence intervals. Differences of a few points on 20 conversations are noise.
Letting the simulator and judge share a model with the bot. Same-family models tend to favor each other's style. Mix providers where practical.
Ignoring tool side effects. Run bots against mocked tools that record calls so you can assert "no refund was issued" rather than trusting what the bot says it did.
No turn cap. Always bound conversation length, and count [MAX_TURNS] as a failure category of its own.
Optimizing one metric. Pushing turns-to-resolution down can raise policy violations. Track the set together.

A checklist for tomorrow

Pull the last 20 thumbs-down or escalated conversations and turn 5 of them into scripted scenarios with state assertions.
Add repeat-question and loop-rate metrics to your existing eval run; they need no new infrastructure.
Write 5 personas and 5 goals; run each pair 3 times with a simulated user and read every transcript once.
Decide your pass thresholds per metric and wire the scripted layer into CI.
Schedule a weekly 30-minute human review to recalibrate the simulator and judge.

Exact SDK calls and model names change, so check your provider's current documentation before copying the simulator code. The structure (persona, hidden goal, stop tokens, role flipping) carries over to any provider.

Next steps

If you are building conversational AI features, whether as an engineer wiring up evals or a PM defining what "resolved" should mean, this is the kind of work we practice hands-on in our cohorts. Explore the engineering cohort or the AI for product managers cohort to build and evaluate real AI products with a group of peers.