Trajectory Evals: Scoring How Your Agent Got There, Not Just the Answer

Your agent eval suite is green. Task completion is at 88%. Then a user reports that the agent confidently told them their refund was processed — and it wasn't. You dig in: the agent called search_orders, got a timeout, hallucinated an order ID, called issue_refund with it, got a 404, and then wrote "Your refund of $49 has been processed." Your eval graded the final message against a rubric that said "confirms refund politely." It passed.

This is the core failure of final-answer evaluation for agents. The answer is a single sample from a process with many steps, and the process is where agents break. If you only grade the last token, you are blind to the tool call that failed, the argument that was wrong, and the recovery that never happened.

Trajectory evaluation fixes this by making the trajectory — the ordered sequence of reasoning, tool calls, tool arguments, tool results, and retries — the unit you score.

Why per-step reliability is brutal

Agent steps depend on each other. A wrong argument in step 3 poisons steps 4 through 20. That means per-step accuracy multiplies across the chain:

Per-step reliability	5-step task	10-step task	20-step task
99%	95%	90%	82%
95%	77%	60%	36%
90%	59%	35%	12%
80%	33%	11%	1%

An agent that feels "pretty good" at 95% per step completes a 20-step workflow about a third of the time. You cannot find the leak by staring at end-to-end pass rates. You have to attribute failure to a step.

The six dimensions worth scoring

Don't collapse the trajectory into one number. Score these independently, each with its own rubric:

Dimension	Question it answers	Typical scorer
Tool selection	Did it call the right tool for this sub-goal?	Exact match / LLM judge
Argument correctness	Were the tool arguments well-formed and grounded in context?	Schema check + LLM judge
Result utilization	Did it actually use what the tool returned, or ignore/hallucinate?	LLM judge over step
Error recovery	After a tool error, did it retry sensibly or plow ahead?	Rule + LLM judge
Plan adherence	Did the executed steps follow a coherent plan?	LLM judge over full trace
Task completion	Was the goal genuinely achieved (verified, not claimed)?	Ground-truth assertion

The first four are step-level. The last two are trajectory-level. Keeping them separate tells you what to fix: low tool selection means your tool descriptions or system prompt are ambiguous; low result utilization means the model isn't attending to tool output; low error recovery means you need explicit retry scaffolding.

Capturing a trajectory

You can't evaluate what you don't record. Instrument the loop to emit a structured trace. Here's a minimal shape in Python:

from dataclasses import dataclass, field
from typing import Any, Literal

@dataclass
class Step:
    kind: Literal["reasoning", "tool_call", "tool_result", "final"]
    tool_name: str | None = None
    tool_args: dict[str, Any] | None = None
    tool_output: Any = None
    error: str | None = None
    text: str | None = None

@dataclass
class Trajectory:
    task_id: str
    goal: str
    steps: list[Step] = field(default_factory=list)
    final_answer: str | None = None
    # ground truth for assertions
    expected_tools: list[str] | None = None
    world_check: Any = None  # callable or expected end state

If you use an OpenTelemetry-based tracer (most agent observability tools now emit GenAI semantic-convention spans), you can reconstruct this from spans instead of hand-rolling it. The key is that each tool call, its arguments, its result, and any error land in the record.

Scoring the trajectory

Combine cheap deterministic checks with an LLM judge for the fuzzy dimensions. Deterministic first — it's free and catches the obvious stuff:

def score_deterministic(traj: Trajectory) -> dict:
    tool_calls = [s for s in traj.steps if s.kind == "tool_call"]
    results = [s for s in traj.steps if s.kind == "tool_result"]
    errors = [s for s in results if s.error]

    called_tools = [s.tool_name for s in tool_calls]
    tool_recall = None
    if traj.expected_tools:
        hit = len(set(traj.expected_tools) & set(called_tools))
        tool_recall = hit / len(set(traj.expected_tools))

    # did any tool call repeat identical args 3+ times? (stuck loop)
    seen, looped = {}, False
    for s in tool_calls:
        key = (s.tool_name, repr(s.tool_args))
        seen[key] = seen.get(key, 0) + 1
        if seen[key] >= 3:
            looped = True

    # error followed by a non-retry final answer = unrecovered failure
    unrecovered = False
    for i, s in enumerate(traj.steps):
        if s.kind == "tool_result" and s.error:
            rest = traj.steps[i + 1 :]
            if rest and all(r.kind in ("reasoning", "final") for r in rest):
                unrecovered = True

    return {
        "tool_recall": tool_recall,
        "step_count": len(tool_calls),
        "error_count": len(errors),
        "looped": looped,
        "unrecovered_error": unrecovered,
    }

Then the judge, scored per step so you get attribution:

import json
from anthropic import Anthropic

client = Anthropic()

STEP_RUBRIC = """You are grading one step of an AI agent's trajectory.

Goal: {goal}
Recent context (previous steps): {context}

This step:
{step}

Score each 1-5 (5 = ideal). Return JSON only:
{{"tool_selection": int, "argument_correctness": int,
  "result_utilization": int, "rationale": str}}

Guidance:
- tool_selection: was this the right tool for the immediate sub-goal?
- argument_correctness: are args well-formed and grounded in prior context
  (not invented)? If no tool call, score null.
- result_utilization: for a tool_result step, did later reasoning use it
  faithfully? Score based on whether the agent ignored or misread it.
"""

def judge_step(goal: str, context: list[Step], step: Step) -> dict:
    msg = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": STEP_RUBRIC.format(
                goal=goal,
                context=json.dumps([s.__dict__ for s in context[-4:]], default=str),
                step=json.dumps(step.__dict__, default=str),
            ),
        }],
    )
    return json.loads(msg.content[0].text)

Roll step scores up to a trajectory verdict, and — critically — verify task completion against the world, not the transcript:

def score_trajectory(traj: Trajectory) -> dict:
    det = score_deterministic(traj)

    step_scores = []
    for i, s in enumerate(traj.steps):
        if s.kind in ("tool_call", "tool_result"):
            step_scores.append(judge_step(traj.goal, traj.steps[:i], s))

    def avg(key):
        vals = [s[key] for s in step_scores if s.get(key) is not None]
        return sum(vals) / len(vals) if vals else None

    # ground-truth completion: run the check against real end state
    completed = bool(traj.world_check and traj.world_check())

    return {
        **det,
        "avg_tool_selection": avg("tool_selection"),
        "avg_argument_correctness": avg("argument_correctness"),
        "avg_result_utilization": avg("result_utilization"),
        "task_completed": completed,
        "false_success": (not completed)
            and traj.final_answer is not None
            and "done" in (traj.final_answer or "").lower(),
    }

That false_success flag — the agent claimed completion but the world check failed — is the single most valuable number in the whole suite. It's the refund bug, caught automatically.

Building the dataset

Trajectory evals need tasks with verifiable end states, not just Q&A pairs.

1. Mine production traces. Pull 50–100 real sessions. Label the ones that failed and why (wrong tool, bad arg, no recovery). These become regression cases.
2. Write world checks, not answer keys. For "cancel my subscription," the check is db.subscription(user).status == "cancelled", not a string match on the reply.
3. Seed failure conditions. Include tasks where a tool will error (invalid ID, empty result, rate limit) so you actually measure error recovery.
4. Keep a golden trajectory per task for the happy path, so expected_tools and step count have a reference.
5. Version it. Freeze the set; add cases as bugs appear. Track score-per-dimension over time, not a single trend line.

Common pitfalls

• One score to rule them all. A blended 0.82 hides that error recovery is at 0.3. Report the vector.
• Grading the transcript for completion. LLMs write convincing "task complete" messages regardless of reality. Always check end state.
• Over-strict tool-order matching. Many trajectories have valid alternate orderings. Score tool set recall and let the judge assess coherence; reserve exact-sequence checks for truly linear workflows.
• Judge sees only the final step. Give the judge the preceding steps as context or it can't assess whether an argument was grounded.
• No cost/step budget. An agent that solves the task in 40 steps when 6 suffice is failing. Track step count and token cost as first-class metrics.
• Ignoring non-determinism. Run each task 3–5 times. Report pass rate and variance; a task that passes 2/5 is not "passing."
• Static dataset forever. If the eval set never changes, the agent silently overfits to it. Rotate in fresh production cases monthly.

Next steps

1. Instrument your agent loop to emit structured traces this week — even a JSON log is enough to start.
2. Pull 30 production sessions, hand-label failures, and turn them into tasks with world checks.
3. Wire up the deterministic scorer first. It's free and will immediately flag loops and unrecovered errors.
4. Add the LLM step judge for the three fuzzy dimensions. Calibrate it against your hand labels before trusting it.
5. Put the suite in CI with per-dimension thresholds. Block merges that regress error recovery or raise false_success.

Trajectory evals are the difference between "our agent scores well" and "our agent works." The path is the product.

---

If you're building and shipping AI agents and want to go deep on evals, tool design, and production reliability with a cohort of engineers doing the same, take a look at the Joinloop engineering cohort.