Guardrails for AI Agents: Validating Inputs and Outputs in Production

Your agent works in the demo. Then it ships, and within a week it has: returned a JSON blob with a trailing comma that broke your parser, quoted a competitor's pricing as your own, leaked a customer's phone number into a support ticket summary, and called a refund_order tool with a negative amount because the user typed something ambiguous. None of these are prompt injection attacks. Nobody was trying to break the system. The model just did what LLMs do — produce plausible-looking output that isn't always correct, safe, or well-formed.

Guardrails are the layer that catches this. They sit between your application and the model, on both sides of the call, and they reject, repair, or re-route anything that doesn't meet a defined contract. If you've already read up on prompt injection defense, guardrails are the broader discipline it sits inside — injection is one threat model among several that guardrails address.

What a guardrail actually is

A guardrail is a check that runs synchronously in the request path, either before the model sees the input or after it produces output, and that can block, modify, or flag the result. It is not a system prompt instruction ("please don't leak PII") — instructions are suggestions the model can ignore. A guardrail is code that runs regardless of what the model decided to do.

Four categories cover most production needs:

1. Input validation — reject or sanitize what goes into the prompt (PII in user messages, oversized inputs, disallowed topics)
2. Output validation — check what comes out (schema conformance, factual claims, banned content, tone)
3. Tool-call validation — verify arguments before executing a function the model requested
4. Behavioral guardrails — rate limits, scope enforcement, escalation triggers based on conversation patterns

Why "just ask the model nicely" doesn't work

A common first attempt is stuffing rules into the system prompt: "Always respond in valid JSON. Never mention competitor names. Refuse requests for refunds over $500." This works most of the time, which is the problem — "most of the time" is not a production guarantee. Models don't have a mechanism for guaranteed compliance with prose instructions; they have a mechanism for producing statistically likely completions given those instructions. Under adversarial input, unusual phrasing, or just an off day in sampling, the instruction gets dropped.

Guardrails move the guarantee out of the model and into deterministic code. The model still does the reasoning; the guardrail does the enforcement.

Building an output schema guardrail

The simplest and highest-value guardrail is schema validation on structured output. Even with native structured-output modes (JSON mode, function calling with strict schemas), you should validate independently — provider-side schema enforcement has edge cases (nested unions, recursive schemas) where it silently falls back to best-effort.

from pydantic import BaseModel, ValidationError, field_validator
from anthropic import Anthropic

class RefundDecision(BaseModel):
    order_id: str
    amount_cents: int
    reason: str
    approved: bool

    @field_validator("amount_cents")
    @classmethod
    def amount_must_be_reasonable(cls, v):
        if v < 0:
            raise ValueError("amount_cents cannot be negative")
        if v > 100_000:  # $1000 cap — anything above needs human review
            raise ValueError("amount_cents exceeds auto-approval limit")
        return v

client = Anthropic()

def get_refund_decision(order_context: str, max_retries: int = 2) -> RefundDecision:
    last_error = None
    for attempt in range(max_retries + 1):
        prompt = order_context
        if last_error:
            prompt += f"\n\nYour previous response failed validation: {last_error}. Fix it."

        response = client.messages.create(
            model="claude-sonnet-5",
            max_tokens=500,
            tools=[{
                "name": "submit_refund_decision",
                "input_schema": RefundDecision.model_json_schema(),
            }],
            tool_choice={"type": "tool", "name": "submit_refund_decision"},
            messages=[{"role": "user", "content": prompt}],
        )
        tool_use = next(b for b in response.content if b.type == "tool_use")
        try:
            return RefundDecision.model_validate(tool_use.input)
        except ValidationError as e:
            last_error = str(e)

    raise RuntimeError(f"Model failed schema validation after {max_retries} retries")

The pattern: validate, and if validation fails, feed the specific error back to the model for one or two retries before giving up and escalating to a human or a fallback path. This "retry-with-error" loop recovers most transient failures without silently accepting bad output.

Tool-call argument guardrails

The highest-blast-radius failures come from tool calls, because tools have side effects — a bad refund_order call moves real money. Validate arguments before execution, independent of whatever schema you gave the model:

interface ToolGuardrail<T> {
  validate(args: T): { ok: true } | { ok: false; reason: string };
}

const refundGuardrail: ToolGuardrail<{ orderId: string; amountCents: number }> = {
  validate(args) {
    if (args.amountCents <= 0) {
      return { ok: false, reason: "amount must be positive" };
    }
    if (args.amountCents > 100_000) {
      return { ok: false, reason: "exceeds auto-approval limit, route to human" };
    }
    if (!/^ord_[a-zA-Z0-9]+$/.test(args.orderId)) {
      return { ok: false, reason: "malformed order id" };
    }
    return { ok: true };
  },
};

async function executeTool(name: string, args: unknown) {
  const guardrail = toolGuardrails[name];
  if (guardrail) {
    const result = guardrail.validate(args as any);
    if (!result.ok) {
      // Return the failure to the model as a tool result, don't execute
      return { error: result.reason };
    }
  }
  return toolImplementations[name](args);
}

Note that the guardrail rejection goes back to the model as a tool result — this lets the model retry with corrected arguments or ask the user for clarification, rather than the request just dying.

PII detection on input and output

For anything that touches customer data — support tickets, transcripts, generated summaries — run a PII scanner both before the prompt is built and after the response is generated. Use a purpose-built library rather than regex-only detection; names and addresses don't have consistent patterns.

from presidio_analyzer import AnalyzerEngine

analyzer = AnalyzerEngine()

def redact_pii(text: str, entities: list[str] | None = None) -> tuple[str, bool]:
    results = analyzer.analyze(
        text=text,
        entities=entities or ["PHONE_NUMBER", "EMAIL_ADDRESS", "CREDIT_CARD", "US_SSN"],
        language="en",
    )
    if not results:
        return text, False

    redacted = text
    for r in sorted(results, key=lambda x: x.start, reverse=True):
        redacted = redacted[: r.start] + f"[{r.entity_type}]" + redacted[r.end :]
    return redacted, True

Run this on the input before it reaches the prompt (so PII never enters the model's context if you don't need it there) and again on the output before it's logged, displayed, or sent onward. Two passes catch cases where the model echoes back PII from context it legitimately needed.

Comparison: where to enforce what

Guardrail type	Where it runs	Catches	Miss cost
Input PII redaction	Before prompt construction	Customer data entering logs/traces unnecessarily	Compliance violation, data leak
Schema validation	After generation, before parsing	Malformed JSON, missing fields, wrong types	Downstream crash
Tool-call argument checks	Before tool execution	Out-of-range values, malformed IDs, privilege escalation	Real-world side effect (money, data loss)
Output content filter	After generation, before display	Banned topics, competitor mentions, off-brand tone	Reputational, legal
Factuality / grounding check	After generation, before display	Claims not supported by retrieved context	User trust, misinformation
Rate / scope limiter	Across the conversation	Runaway loops, scope creep, cost blowout	Budget overrun

A rough rule: the more expensive the failure to reverse, the earlier and more deterministically you should block it. Tone problems can go through a lighter-weight LLM-as-judge check; financial transactions need hard-coded numeric bounds, not a model's opinion.

Using an LLM as the guardrail — carefully

For fuzzy checks (tone, brand safety, "is this response actually answering the question"), a second cheaper/faster model call as judge is common. Two rules keep this from becoming a second unreliable layer on top of the first:

• Use a different model or a much simpler prompt than the generator. If the same model that produced a bad output is asked to judge it, correlated failure modes mean it often approves its own mistake.
• Make the judge output structured and narrow — a boolean plus a reason, not free text. "Does this response mention a competitor by name? yes/no" is checkable; "is this response good?" is not.

def check_no_competitor_mention(response_text: str, competitors: list[str]) -> bool:
    check = client.messages.create(
        model="claude-haiku-4-5-20251001",  # cheap, fast — this is a narrow binary check
        max_tokens=10,
        messages=[{
            "role": "user",
            "content": f"Does this text mention any of these companies: {', '.join(competitors)}?\n\nText: {response_text}\n\nAnswer only 'yes' or 'no'.",
        }],
    )
    return "yes" not in check.content[0].text.lower()

Common pitfalls

• Guardrails that only log, never block. A guardrail that flags a violation but still returns the bad output to the user is a monitoring tool, not a guardrail. Decide up front which failures are hard-blocks versus soft-warnings, and make the hard-blocks actually block.
• No retry path, only rejection. If a schema validation failure just throws a 500, you've traded a bad output for a broken feature. Build the retry-with-error loop; it recovers the majority of failures.
• Validating only the final output, not intermediate tool calls in a multi-step agent. A guardrail on the final response doesn't catch a destructive tool call the agent made three steps earlier.
• Guardrail latency nobody budgeted for. A PII scan plus an LLM-as-judge call can add real latency. Measure it, and consider running independent checks in parallel rather than in series.
• Treating guardrails as a one-time build. New failure modes show up as usage grows. Guardrails need the same iteration loop as evals — log violations, review them weekly, add new checks.
• No visibility into how often guardrails fire. If you don't track block/retry rates per guardrail, you won't notice when one is silently rejecting 20% of legitimate traffic because a regex is too strict.

Next steps

Start with the guardrail that guards your most expensive failure mode — usually tool-call argument validation if your agent has side effects, or PII redaction if you're in a regulated domain. Add schema validation next since it's nearly free and catches a large share of production breakage. Treat everything else as an incremental addition driven by what actually goes wrong in your logs, not a checklist you build all at once.

If you're building agentic systems and want a structured way to work through evals, guardrails, and production failure modes with people doing this for real, take a look at Joinloop's AI engineering cohort.