Tool-Contract Testing: Catching Silent Agent Regressions Before Production

A team ships a one-line change: renaming a tool parameter from user_id to userId to match a new API convention. Evals pass — the eval suite tests end-to-end task completion, and the model is good enough at few-shot inference that it mostly still fills in the right argument name from context. In production, under real traffic with more varied prompts, the model gets it wrong often enough that a support queue fills up with silent failures no one notices for three days.

This is the failure mode evals don't catch. Evals answer "did the agent solve the task." They usually don't answer "did this specific tool call still match its contract." When a tool schema changes — a renamed field, a stricter enum, a parameter that became required — nothing in a typical eval suite pins down that the agent's calling convention is still correct, because evals sample outcomes, not interfaces.

Tool-contract testing closes that gap. It borrows a well-worn idea from API testing — verify the producer and consumer agree on a contract, independent of whether the end-to-end task happens to succeed — and applies it to the boundary between your LLM and your tools.

Why this is a different problem than evals

Evals and contract tests check different things, and you need both.

An eval runs the full agent loop and grades the final outcome: did the agent book the right flight, answer the support question correctly, generate a valid SQL query. It's necessarily slow, non-deterministic (or expensively pinned), and expensive to run on every commit.

A contract test isolates one link in the chain: given this tool schema and this scenario, does the model produce a call that matches the contract — right tool, right argument names, right types, right required fields. It doesn't care whether the overall task succeeded. It cares whether the interface was honored.

The reason this matters: tool schemas change far more often than task requirements do. Someone adds a field, tightens a type, splits one tool into two, deprecates a parameter. Each of those changes is invisible to your eval suite unless the specific eval scenario happens to exercise that exact tool in that exact way — which, with more than a handful of tools, it usually doesn't.

	Evals	Contract tests
What it checks	Task outcome (did the agent solve it)	Interface compliance (did the call match the schema)
Granularity	Full trajectory	Single tool call
Runs on	Nightly / pre-release	Every commit, every schema change
Catches	Reasoning failures, bad tool choice, wrong final answer	Silent drift between prompt, schema, and model behavior
Cost per run	High (multi-turn, often multi-model-call)	Low (often single-turn, cacheable)
Flakiness tolerance	Expected, managed statistically	Should be near-zero — it's a schema diff, not a judgment call

The three things a tool contract needs to guarantee

1. Schema validity — the arguments the model produces parse against the JSON Schema you declared, including required fields, types, and enums.
2. Semantic correctness — the values are right, not just the shape. Passing date: "tomorrow" when the tool expects an ISO date validates against a loose schema but is still wrong.
3. Selection correctness — given an ambiguous prompt, the model picks the tool you intended, not a plausible-looking neighbor.

Most teams test #1 informally (their code will throw if the JSON doesn't parse) and skip #2 and #3 until a user reports a bug. Contract tests make all three explicit and automated.

Building a contract test suite

Start with a fixture format that pairs a scenario with the expected call shape — this is your golden trajectory for a single tool decision, not a whole conversation.

# fixtures/tool_contracts.py
from dataclasses import dataclass, field

@dataclass
class ToolCallFixture:
    name: str
    prompt: str
    expected_tool: str
    expected_args: dict          # values checked with matchers, see below
    context: list[dict] = field(default_factory=list)  # prior turns, if any

FIXTURES = [
    ToolCallFixture(
        name="reschedule_with_relative_date",
        prompt="Move my 3pm meeting with Sam to next Tuesday",
        expected_tool="reschedule_event",
        expected_args={
            "event_query": "3pm meeting with Sam",
            "new_date": ISO_DATE,   # matcher, not a literal string
        },
    ),
    ToolCallFixture(
        name="ambiguous_cancel_vs_delete",
        prompt="Get rid of the standup on Friday",
        expected_tool="cancel_event",   # not delete_event — cancel notifies attendees
        expected_args={"event_query": "standup on Friday"},
    ),
]

Values rarely match exactly — "next Tuesday" resolves to a concrete date only relative to today. Use matchers instead of string equality:

# fixtures/matchers.py
import re
from datetime import date, timedelta

class ISODateMatcher:
    def __eq__(self, other):
        try:
            return bool(re.fullmatch(r"\d{4}-\d{2}-\d{2}", other)) and \
                   date.fromisoformat(other) >= date.today()
        except (TypeError, ValueError):
            return False

class FuzzyTextMatcher:
    """Loose containment check for free-text args the model paraphrases."""
    def __init__(self, must_contain: list[str]):
        self.must_contain = must_contain

    def __eq__(self, other):
        if not isinstance(other, str):
            return False
        lowered = other.lower()
        return all(term.lower() in lowered for term in self.must_contain)

ISO_DATE = ISODateMatcher()

The test runner calls the model once per fixture with the real tool schemas, then checks tool selection and argument shape against the fixture:

# test_tool_contracts.py
import json
import anthropic
from fixtures.tool_contracts import FIXTURES
from tools import TOOL_SCHEMAS  # the same schemas you ship to production

client = anthropic.Anthropic()

def run_fixture(fixture):
    messages = fixture.context + [{"role": "user", "content": fixture.prompt}]
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        tools=TOOL_SCHEMAS,
        messages=messages,
    )

    tool_use = next((b for b in response.content if b.type == "tool_use"), None)
    assert tool_use is not None, f"[{fixture.name}] model did not call a tool"
    assert tool_use.name == fixture.expected_tool, (
        f"[{fixture.name}] expected tool '{fixture.expected_tool}', "
        f"got '{tool_use.name}'"
    )

    for key, expected in fixture.expected_args.items():
        actual = tool_use.input.get(key)
        assert actual == expected, (
            f"[{fixture.name}] arg '{key}': expected {expected!r}, got {actual!r}"
        )

def test_all_tool_contracts():
    failures = []
    for fixture in FIXTURES:
        try:
            run_fixture(fixture)
        except AssertionError as e:
            failures.append(str(e))
    assert not failures, "\n".join(failures)

Two details matter for this to be trustworthy rather than noisy:

• Pin the model version explicitly in the test config (claude-sonnet-5-20260115, not a floating alias) so a provider-side model update doesn't get blamed on your schema change, or vice versa. Bump the pin deliberately, in its own commit, and re-baseline fixtures at that point.
• Set temperature to 0 for contract tests. You're testing interface compliance, not creativity — variance here is pure noise.

Wiring it into schema changes, not just CI cadence

The highest-leverage place to run this suite isn't nightly — it's as a pre-merge check on any diff that touches a tool schema file. That turns "the eval suite will probably catch it eventually" into "you cannot merge a breaking schema change without updating the fixtures that depend on it."

# .github/workflows/tool-contracts.yml
name: tool-contract-tests
on:
  pull_request:
    paths:
      - "tools/**"
      - "fixtures/tool_contracts.py"
jobs:
  contracts:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest test_tool_contracts.py -v
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Scoping the trigger to tools/** keeps this fast and cheap — it only fires when the actual contract surface changes, not on every unrelated PR. For a suite of 30-50 fixtures at single-turn, temperature-0 calls, this typically runs in under a minute and costs a few cents per run.

A pre-merge checklist for schema changes

Use this whenever a tool schema changes, whether it's a rename, a new required field, or a whole new tool:

• Does an existing fixture exercise this tool? If not, add one before merging the schema change.
• If a field was renamed, does the old name appear anywhere in cached prompts, few-shot examples, or system prompt text? Update those too — schema and prompt drifting apart is the most common silent break.
• If a field became required, does at least one fixture prompt not obviously supply that value, to confirm the model still infers it or asks for it rather than hallucinating a default?
• If a new tool overlaps semantically with an existing one (e.g. cancel_event vs delete_event), add a fixture that specifically discriminates between them.
• Re-run the full contract suite, not just fixtures you think are affected — schema changes have non-local effects on selection because the model sees every tool description together.
• If the model version is being bumped, re-baseline in a separate commit from any schema change, so a regression can be attributed to one cause.

Common pitfalls

Testing only the happy path. Contract tests are cheapest to write for clean, unambiguous prompts. The value is disproportionately in the ambiguous ones — ties between similar tools, missing information the model has to ask for, prompts that reference entities by nickname instead of ID. Bias your fixture set toward these.

String-equality on free text. Args like a search query or event description are paraphrased by the model every time, even at temperature 0 across different prompts. Exact-match assertions on these fields will flake constantly and teams end up disabling the whole suite. Use containment or semantic matchers instead, and reserve exact match for structured fields (IDs, enums, dates).

Letting fixtures rot. A contract suite that isn't updated when tools change becomes actively misleading — it asserts an old contract the schema no longer honors, and either fails spuriously or (worse) someone loosens the assertion to make it pass, defeating the point. Treat fixture files as owned by whoever owns the tool schema, updated in the same PR.

Conflating this with your eval suite. Don't fold contract tests into your nightly eval run and call it done. Evals sample a small number of end-to-end scenarios for cost reasons; contract tests need to cover every tool and every likely ambiguity, and need to run on every relevant diff, not on a schedule.

No selection fixtures, only argument fixtures. It's easy to test "does the model fill in the right date" and forget to test "does the model pick the right tool at all" when two tools are semantically close. Tool selection errors are usually more damaging than argument errors, because the wrong tool can have side effects the user never intended.

Next steps

Start small: pick your three most-used tools, write five fixtures each — three clean, two ambiguous — and wire the suite to run on any diff touching those schemas. Expand coverage as you find real production failures; every silent regression you catch after the fact is a fixture you were missing, not evidence the approach doesn't work.

If you're building out eval and testing infrastructure for production agents and want a structured way to think through this alongside evals, observability, and guardrails, Joinloop's AI Engineering cohort covers this as part of a broader production-readiness track for teams shipping agents.