Designing UX for AI Uncertainty: Confidence, Citations, and Undo Patterns for PMs

Every AI feature ships with an error rate. Your eval suite tells you what it is; your UX decides what it costs. Two teams can ship the same model at 90% accuracy. One gets angry support tickets and churn. The other gets users who say "it's like a fast junior colleague." The difference is almost never the model. It is how the product behaves in the 10% of cases where the model is wrong, and in the larger share where it is merely unsure.

This post is a practical toolkit for PMs and the engineers who work with them: how to decide how much uncertainty to show, which patterns fit which risk level, and a small implementation you can adapt.

The core principle: match friction to cost of error

Users do not need to see uncertainty everywhere. They need friction proportional to what a mistake costs. A wrong song recommendation costs nothing. A wrong refund approval costs money. A wrong medication note costs far more.

Start by classifying each AI action on two axes:

• Reversibility: can the user undo it in seconds, or is it permanent (an email sent, a payment made)?
• Blast radius: does it affect only the user, or other people, customers, or records?

Low on both axes: act automatically and make undo easy. High on either: ask first, show evidence, and require a deliberate confirm.

Risk tier	Example	Default pattern	Friction
Low	Autocomplete, tag suggestions	Act, show undo	Near zero
Medium	Drafted reply, summarised ticket	Suggest, user edits and accepts	One click
High	Sending email, changing records	Preview + explicit confirm	Deliberate
Critical	Payments, deletes, compliance outputs	Human approval with evidence	Gated

If you do one thing after reading this, tag every AI action in your product with a tier. Most teams discover they applied the same flow to everything.

Pattern 1: Calibrated confidence, not raw scores

The tempting move is to display "87% confident." Resist it unless that number is calibrated. Raw model probabilities and self-reported confidence ("I'm 90% sure") are often poorly calibrated: when the model says 90%, it may be right 70% of the time. Showing a precise-looking number that is wrong teaches users to distrust all your numbers.

Better options:

1. Bucket, don't quantify. Show two or three states ("Looks good", "Check this", "Needs review") derived from thresholds you validated against labelled data.
2. Calibrate against outcomes. Take a few hundred labelled examples, bin predictions by score, and check that each bin's actual accuracy matches its label. Re-run when you change the model or prompt.
3. Use signals beyond the model's opinion. Agreement across multiple samples, retrieval score, whether a citation verifies, or whether output passed schema validation are often better predictors than the model's own confidence.

A minimal TypeScript gate that maps validated signals to a UI state:

type Signals = {
  retrievalScore: number;      // 0..1 from your reranker
  citationsVerified: boolean;  // every claim traced to a source span
  sampleAgreement: number;     // 0..1 agreement across N samples
};

type UiState = "auto" | "review" | "escalate";

// Thresholds must come from your own labelled data, not from this example.
const THRESHOLDS = { auto: 0.85, review: 0.6 };

export function decideUiState(s: Signals): UiState {
  if (!s.citationsVerified) return "escalate";
  const score = 0.5 * s.sampleAgreement + 0.5 * s.retrievalScore;
  if (score >= THRESHOLDS.auto) return "auto";
  if (score >= THRESHOLDS.review) return "review";
  return "escalate";
}

The weights above are placeholders. The point is the structure: the UI state is a function of verifiable signals, and the thresholds are tuned against real outcomes.

Pattern 2: Citations that users can actually check

Citations are the highest-leverage trust pattern for anything grounded in documents. But a footnote that says "Source: Policy Handbook" is decoration. A useful citation:

• Links to the exact passage, highlighted, not just the document.
• Appears next to the claim it supports, not in a list at the bottom.
• Is verified on the backend: the cited span must actually support the claim, otherwise the claim is dropped or flagged.

That last point matters. Models can attach plausible-looking citations to unsupported statements. If you only render what the model returned, you have built a more convincing way to be wrong. Run a claim-level check (an entailment or judge pass against the cited span) before display. We cover that approach in the RAG citation verification post on this blog.

Design rule: when a claim has no verifiable source, say so visibly ("No source found") rather than hiding the claim or presenting it as equally solid.

Pattern 3: Undo beats confirmation dialogs

Confirmation dialogs are a tax on every correct action to catch a few wrong ones. Users click through them by the third time. For reversible actions, prefer act then offer undo: Gmail's "Undo send" is the canonical example.

Implementation notes for engineers:

• Make AI actions soft by default: stage a change, apply it after a short window (5 to 30 seconds) or on the next user action, and keep a revert path afterwards.
• Store the pre-change state so undo is exact, not best effort.
• Log undo events. An undo is the cheapest, most honest label you will ever get that the model was wrong. Feed those into your eval dataset.

Reserve hard confirmation dialogs for irreversible actions, and make the dialog specific: "Send this email to 214 customers?" rather than "Are you sure?"

Pattern 4: Show the work in proportion

For agents and multi-step features, users trust what they can inspect. But dumping a full reasoning trace overwhelms them. Use progressive disclosure:

• Default: the result plus a one-line summary of what was done ("Checked 3 sources, updated 2 fields").
• On click: the steps, sources, and tool calls.
• For admins: full traces and replay.

Keep the summary honest. If the agent skipped a step or hit an error and recovered, the one-liner should say so.

Pattern 5: Make correction cheap and visible

When users edit or reject AI output, that is signal. Design for it:

• Inline editing of the output, not "regenerate" as the only option.
• Quick reasons on rejection (wrong, outdated, tone, unsafe) as one-tap chips.
• Show the user that corrections matter, even if it is just "Thanks, we'll use this to improve." Only say that if it is true.

Be careful with implicit signals. A user ignoring a suggestion is not necessarily a thumbs-down; they may not have seen it. Treat accept, edit, and explicit reject as stronger signals than silence.

Pattern 6: Graceful abstention

Sometimes the right answer is "I don't know." A feature that abstains on 8% of queries but is right on the rest usually earns more trust than one that answers everything. Design the abstention state as a first-class screen:

• Say what is missing ("I couldn't find this in your uploaded documents").
• Offer a next step: search manually, ask a human, rephrase.
• Never dead-end the user.

Comparison: which pattern for which problem

Problem	Best pattern	Watch out for
Users over-trust output	Confidence buckets, visible sources	Fake precision
Users ignore output entirely	Show work, inline evidence	Too much detail
Costly irreversible actions	Preview and specific confirm	Dialog fatigue
Frequent small mistakes	Undo and soft actions	Missing pre-change state
Need better training data	Cheap correction, reason chips	Treating silence as a label
Out-of-scope queries	Abstention with next step	Over-abstaining

Common pitfalls

1. Uniform disclaimers. A footer reading "AI can make mistakes" on every screen becomes invisible. Put warnings where the risk is.
2. Uncalibrated percentages. Precise numbers imply precision you do not have.
3. Unverified citations. Rendering model-supplied sources without checking them.
4. Confirm-everything flows. Training users to click "Yes" blindly.
5. No instrumentation. Without tracking accept, edit, reject, and undo rates per tier, you cannot tell whether your UX is working.
6. Designing only the happy path. Test empty, low-confidence, and error states with the same care as the success state.

A checklist before you ship

• Every AI action has a risk tier (reversibility and blast radius).
• Confidence states are derived from signals validated on labelled data.
• Citations link to exact spans and are verified server-side.
• Reversible actions have undo; irreversible ones have specific previews.
• Abstention and error states are designed, not default.
• Accept, edit, reject, and undo events are logged and reviewed weekly.
• You have a target for each (for example, undo rate below X% per tier) and an owner.

Next steps

This week, take your most-used AI feature and run a 30-minute audit: list every action it performs, assign a risk tier, and compare the current UX friction to the tier. You will likely find one high-risk action with no friction and one low-risk action with too much. Fix those two first, add instrumentation for undo and edit rates, and review the numbers after a week of traffic. Check your own product analytics and user research before choosing thresholds; the values here are illustrative.

If you are a PM who wants to practise this on real AI product problems, from defining error budgets to designing the human-in-the-loop experience, take a look at Joinloop's PM cohort.