AI Incident Response: A Runbook for When Your LLM Feature Goes Wrong in Production

Traditional services fail loudly: 500s, timeouts, a pager at 3 a.m. LLM features fail quietly. The endpoint returns 200, latency looks normal, and the answers are subtly wrong, unsafe, or off-brand. You find out from a customer tweet or a support ticket three days later.

Most teams have an incident process for outages and none for quality incidents. This post gives you a runbook you can adopt this week: how to classify AI incidents, what to build before they happen, how to triage in the first 30 minutes, and how to turn each incident into permanent protection.

Why AI incidents are different

Property	Classic service incident	LLM feature incident
Detection	Error rate, latency alarms	Often none; user reports, evals, spot checks
Cause	Deploy, infra, dependency	Prompt change, model version drift, retrieval data change, new input distribution, tool failure
Blast radius	Usually clear (all requests)	Partial: only some intents, languages, tenants
"Fixed" means	Error rate back to baseline	Quality metric back to baseline (needs an eval to prove it)
Rollback	Redeploy previous build	Revert prompt/model/index/config, and each is versioned separately

The practical consequence: you need separate versioned levers (prompt, model, retrieval index, tool set) and a way to measure quality continuously. Without both, you are debugging blind.

Step 1: Define AI-specific severity levels

Reuse your existing SEV scale, but define what counts for AI. Example:

Severity	Definition	Example	Response
SEV1	Harm, legal, or security exposure	Leaks another tenant's data; gives dangerous advice; prompt injection triggers a destructive tool call	Kill switch immediately, page on-call + PM + legal
SEV2	Widespread quality failure	Refusal rate jumps 5x after a model update; hallucinated citations in a core flow	Rollback or degrade within 1 hour
SEV3	Localised quality failure	One language or one intent is bad	Fix within days, add eval cases
SEV4	Cosmetic	Tone drift, verbose answers	Backlog

PMs: this table is your job to own. Engineers can't decide alone what "harm" means for your product.

Step 2: Build the levers before you need them

You cannot respond in 10 minutes if changing the prompt requires a deploy and a review cycle. Build these four things now.

2a. A runtime config with versioned components

Keep prompt, model, and retrieval index version in config, not code. Every response should log which versions produced it.

# llm_config.py
from dataclasses import dataclass
import json, os

@dataclass(frozen=True)
class LLMConfig:
    model: str
    prompt_version: str
    index_version: str
    tools_enabled: tuple[str, ...]
    feature_enabled: bool
    degraded_mode: bool  # e.g. no tools, retrieval-only answers

def load_config() -> LLMConfig:
    # Replace with your flag service (LaunchDarkly, Unleash, a DB row, etc.)
    raw = json.loads(os.environ.get("LLM_CONFIG_JSON", "{}"))
    return LLMConfig(
        model=raw.get("model", "your-default-model"),
        prompt_version=raw.get("prompt_version", "v12"),
        index_version=raw.get("index_version", "2026-10-01"),
        tools_enabled=tuple(raw.get("tools_enabled", [])),
        feature_enabled=raw.get("feature_enabled", True),
        degraded_mode=raw.get("degraded_mode", False),
    )

2b. A kill switch with graceful degradation

A kill switch that shows an error page is a blunt tool. Define three states:

Full: normal behaviour.
Degraded: disable write-capable tools, fall back to a smaller safer prompt, or return retrieval results without generation.
Off: a static fallback ("This assistant is temporarily unavailable. Here is how to reach support.").

async def handle_request(req, cfg: LLMConfig):
    if not cfg.feature_enabled:
        return fallback_response(req)

    tools = [] if cfg.degraded_mode else [t for t in ALL_TOOLS if t.name in cfg.tools_enabled]
    prompt = load_prompt(cfg.prompt_version)

    resp = await call_model(cfg.model, prompt, req, tools=tools)

    log_event({
        "request_id": req.id,
        "model": cfg.model,
        "prompt_version": cfg.prompt_version,
        "index_version": cfg.index_version,
        "degraded": cfg.degraded_mode,
    })
    return resp

Test the kill switch monthly. An untested kill switch is a hope, not a control.

2c. Trace logging you can actually query

For triage you need to filter traces by version, tenant, intent, and outcome. At minimum log: request ID, user/tenant ID (hashed if needed), config versions, input, retrieved document IDs, tool calls and results, output, latency, token counts, and any user feedback. Mind your PII policy; redact before storage where required.

2d. A small "canary" eval that runs continuously

Run 50 to 200 representative golden cases against production config on a schedule (hourly or daily) and alert when the score drops. This catches silent provider-side model drift, which no deploy log will show you.

def run_canary(cases, cfg):
    failures = []
    for c in cases:
        out = call_model_with_config(c.input, cfg)
        if not c.check(out):          # deterministic check or LLM judge
            failures.append(c.id)
    score = 1 - len(failures) / len(cases)
    if score < 0.92:                  # threshold from your baseline
        alert(f"Canary score {score:.2%}; failing: {failures[:10]}")
    return score

Step 3: The first 30 minutes

Print this and put it in your on-call doc.

Minutes 0 to 5: Contain.

Assign an incident lead and a scribe.
If SEV1: flip to degraded or off first, investigate second.
If unsure, degrade. A temporarily less capable feature is cheaper than a repeat offender.

Minutes 5 to 15: Scope. Answer these with trace queries, not guesses:

When did bad outputs start? (Plot the canary score and thumbs-down rate over time.)
Which segment? (Tenant, language, intent, model version, prompt version.)
What changed near that time? Check the change log for all four levers plus upstream data.

-- Example: find the version combo where the bad-rate spiked
SELECT prompt_version, model, index_version,
       COUNT(*) AS n,
       AVG(CASE WHEN user_feedback = 'down' THEN 1.0 ELSE 0 END) AS down_rate
FROM llm_traces
WHERE ts > NOW() - INTERVAL '48 hours'
GROUP BY 1,2,3
ORDER BY down_rate DESC;

Minutes 15 to 30: Mitigate. Revert the most likely lever, one at a time, and confirm with the canary:

Suspect	Check	Mitigation
Prompt change	Does the old prompt version pass the canary?	Revert `prompt_version`
Model update (yours or provider's)	Does pinning an earlier dated model snapshot fix it?	Pin the snapshot; open a ticket with the provider
Retrieval/index	Bad docs ingested? Stale index? Embedding model changed?	Roll back `index_version`; quarantine bad sources
Tool failure	Tool returning errors or malformed data the model "works around"	Disable the tool; fix the contract
Input shift	New traffic type, new language, adversarial users	Add input guard; route to a stronger model for that segment
Prompt injection	Untrusted content in context triggering actions	Degrade (no tools), sanitise sources, add confirmation gates

Change one lever at a time. If you change three, you will never know which one fixed it.

Step 4: Communication

Internal: one channel, one lead, timestamped updates every 30 minutes for SEV1/SEV2.
Customers: say what happened in plain language, what data (if any) was affected, and what you changed. Don't blame "the AI". You shipped it.
Regulators/legal: for SEV1 involving personal data or safety, involve legal early. Check your contractual and regulatory notification obligations; they vary by jurisdiction, so consult counsel rather than relying on a blog post.

Step 5: The postmortem is where you win

The incident is only fully closed when it becomes a test. Every AI postmortem should produce:

A timeline (detect, contain, mitigate, resolve) with real timestamps.
Root cause across the levers: which of prompt, model, retrieval, tools, or input caused it, and why wasn't it caught?
Eval cases: add every failing trace (sanitised) to your regression set. This is the highest-value output.
Detection gap: what alert would have fired earlier? Add it.
Process fix: e.g. "prompt changes now require canary pass before rollout."

def add_incident_to_evals(trace, expected_behaviour, incident_id):
    case = {
        "id": f"inc-{incident_id}-{trace['request_id'][:8]}",
        "input": redact(trace["input"]),
        "context": trace.get("retrieved_docs"),
        "must": expected_behaviour,   # e.g. "refuses", "cites source", "does not call delete_tool"
        "source": "incident",
    }
    append_jsonl("evals/regression.jsonl", case)

Keep it blameless. The goal is a system where the same failure can't recur silently.

Checklist: are you ready?

Prompt, model, index, and tool set are separately versioned and logged per request
Rollback of any lever takes minutes, without a full deploy
A kill switch with full / degraded / off states, tested in the last 30 days
Canary eval running on a schedule with an alert threshold
Traces queryable by version, segment, and outcome
Severity table agreed by engineering and product
Named on-call owner for AI quality, not just infra
Postmortem template that outputs eval cases

Common pitfalls

Treating "no errors" as "healthy." Quality needs its own metrics.
Pinning nothing. If you call a floating model alias, the provider can change behaviour under you. Pin dated snapshots where offered and check your provider's deprecation docs.
Changing multiple levers at once during mitigation.
Fixing the prompt with a patch for the one failing case. That overfits. Fix the class of failure and add several eval cases.
No owner for severity decisions. Engineers will under- or over-react without a product-defined harm threshold.
Skipping the postmortem because "it's fixed now." It will come back.

Next steps

Today: add version fields to your trace logs.
This week: implement the degraded-mode switch and test it.
This month: stand up a 100-case canary and wire an alert.
After your next incident: run the postmortem template above and measure how many new eval cases it produced.

If you want to build these practices hands-on with other engineers, our AI Engineering cohort covers evals, observability, and production hardening. Product managers owning AI features can look at the PM cohort.