Calibrated Abstention: How to Make Your LLM Feature Say 'I Don't Know'

Most LLM features fail in one specific way: they answer everything. A support bot invents a refund policy. A contract extractor fills in a termination date that is not in the document. A SQL agent picks a plausible column and returns a number that looks right. None of these outputs throw an error. The system is working as designed, and the design is the problem.

Abstention is the feature's ability to decline, escalate, or ask for more information when it is not reliable enough. Calibration is whether the confidence you attach to an answer matches how often that answer is actually right. The two go together. If your confidence is calibrated, you can set a threshold below which the system abstains. If it is not, the threshold is a guess.

This post covers why the model's own confidence is a poor signal, how to build a composite score from signals you control, how to pick thresholds per slice of traffic, and how to measure whether the whole thing is working.

Why you cannot trust the model's self-reported confidence

The obvious move is to ask the model: "How confident are you, 0 to 100?" Teams do this, and the numbers come back clustered at 85 to 95 regardless of whether the answer is right. Three things drive this:

Preference training rewards confident-sounding answers. Reward models tend to score fluent, assertive responses higher, so the model learns that certainty reads well.
Verbalized confidence is not a measurement. The number is generated text. It is conditioned on the answer already written, so it mostly reflects the tone of that answer.
Closed APIs often hide token log-probabilities. Even when you have them, token probability is not the same as factual correctness, especially for long outputs.

You can still use the model as one input. You should not use it as the only input.

The fix: compose confidence from independent signals

A production confidence score should be built from signals that fail in different ways. If two signals fail for the same reason, averaging them gives you false comfort. Three signals cover most failure modes in retrieval-backed features:

Signal	What it measures	How to compute it	Failure it catches
Sample agreement	Does the model give the same answer when re-asked?	Sample N answers at temperature > 0, compare normalized answers	The model is guessing
Retrieval strength	Did we find evidence that is actually relevant?	Top reranker or similarity score, normalized to 0-1	The answer is outside your corpus
Grounding check	Does every claim trace to a retrieved span?	Verify each cited claim against its chunk (string match, NLI, or a judge)	The model wrote something plausible but unsupported

Each signal is cheap to compute on its own. Together, they give you a score you can inspect when something goes wrong, which a single number from the model never gives you.

Implementation: a gate you can measure

The code below is self-contained. It uses a stub call_llm for sampling so you can see the shape; swap in your real client. The composite score is a weighted sum, the gate decides whether to answer or abstain, and the ECE function tells you whether the scores mean what you think they mean.

# abstain_gate.py
from collections import Counter
from dataclasses import dataclass
from typing import Callable


@dataclass
class Evidence:
    answer: str
    retrieval_score: float  # normalized to 0..1 by your reranker
    claims_grounded: int    # claims that matched a retrieved span
    claims_total: int       # claims the answer made


def agreement_score(samples: list[str]) -> float:
    """Fraction of samples that match the most common normalized answer."""
    normalized = [s.strip().lower() for s in samples]
    top_count = Counter(normalized).most_common(1)[0][1]
    return top_count / len(normalized)


def grounding_score(ev: Evidence) -> float:
    if ev.claims_total == 0:
        return 0.0  # an answer with no checkable claims is not grounded
    return ev.claims_grounded / ev.claims_total


def composite_confidence(agreement: float, ev: Evidence) -> float:
    # Weights are a starting point. Tune them against labeled outcomes.
    # Grounding is weighted highest because an ungrounded answer is the costliest error.
    return (
        0.30 * agreement
        + 0.25 * min(ev.retrieval_score, 1.0)
        + 0.45 * grounding_score(ev)
    )


def answer_or_abstain(
    question: str,
    retrieve_and_answer: Callable[[str], Evidence],
    call_llm: Callable[[str], str],
    threshold: float,
    n_samples: int = 5,
) -> dict:
    ev = retrieve_and_answer(question)
    samples = [call_llm(question) for _ in range(n_samples)]
    agree = agreement_score(samples)
    conf = composite_confidence(agree, ev)

    if conf >= threshold:
        return {"action": "answer", "text": ev.answer, "confidence": conf}
    return {
        "action": "abstain",
        "route": "human_review",
        "confidence": conf,
        "reason": {
            "agreement": round(agree, 2),
            "retrieval": round(ev.retrieval_score, 2),
            "grounding": round(grounding_score(ev), 2),
        },
    }


def expected_calibration_error(
    confidences: list[float], correct: list[bool], bins: int = 10
) -> float:
    """ECE: weighted average gap between predicted confidence and observed accuracy."""
    n = len(confidences)
    ece = 0.0
    for b in range(bins):
        lo, hi = b / bins, (b + 1) / bins
        idx = [
            i
            for i, c in enumerate(confidences)
            if lo <= c < hi or (b == bins - 1 and c == 1.0)
        ]
        if not idx:
            continue
        acc = sum(correct[i] for i in idx) / len(idx)
        avg_conf = sum(confidences[i] for i in idx) / len(idx)
        ece += (len(idx) / n) * abs(acc - avg_conf)
    return ece


if __name__ == "__main__":
    # Synthetic labeled outcomes: (composite confidence, was the answer correct?)
    labeled = [(0.92, True), (0.88, True), (0.85, False), (0.74, True),
               (0.66, False), (0.61, True), (0.48, False), (0.35, False),
               (0.31, True), (0.22, False)]
    confs = [c for c, _ in labeled]
    correct = [ok for _, ok in labeled]

    print(f"ECE: {expected_calibration_error(confs, correct):.3f}")

    # Pick the lowest threshold where precision on answered items meets the target.
    target_precision = 0.90
    for t in [x / 100 for x in range(20, 100, 5)]:
        answered = [ok for c, ok in labeled if c >= t]
        if answered and sum(answered) / len(answered) >= target_precision:
            print(f"Lowest threshold meeting {target_precision:.0%} precision: {t:.2f} "
                  f"(answers {len(answered)}/{len(labeled)} items)")
            break

Run it and you will see the core tradeoff in miniature: a higher threshold improves precision on what you answer, but you answer fewer questions and send more to humans. The ECE number tells you whether the scores themselves are honest. A high ECE with a good threshold means you got lucky with the cutoff, and the next prompt change will break it.

Set thresholds per slice, not globally

A single global threshold hides real differences. A refund-policy question and a product-feature question have different risk and different corpus coverage. Slice your traffic by intent, document source, or user tier, and set a threshold for each slice based on the cost of a wrong answer in that slice.

A practical starting table:

Slice	Cost of a wrong answer	Threshold posture	Abstention route
Billing, legal, medical	High	Strict, precision-first	Human review, always
Product how-to	Medium	Balanced	Ask a clarifying question, then retry
Casual or general chat	Low	Permissive	Answer with a caveat

Revisit these thresholds whenever you change the prompt, the model version, or the retriever. A sudden drop in abstention rate after a release is a warning sign. It often means the restraint got trained out of the system by accident, not that quality improved.

Measure abstention as its own metric

A single accuracy number tells you nothing about the gate. Track these instead, per slice and per week:

Answer rate: the share of traffic the system answers. A sudden jump needs an explanation.
Precision on answered items: of the answers you gave, how many were correct. This is the one users feel.
Escalation quality: of the abstentions a human reviewed, how many would have been correct if answered. High numbers here mean your gate is too strict.
ECE over time: calibration drifts. Recompute it on a fresh labeled sample every few weeks.

To get labels, sample a slice of answered and abstained cases every week and have a reviewer mark them. The same approach works well with the production trace pipeline described in the eval dataset post.

Common pitfalls

Treating abstention as failure. An abstention that routes to a human with the evidence attached is a successful outcome. Measure it that way.
Using the model's own confidence as the only signal. As covered above, it is mostly a restatement of the answer's tone.
Calibrating on the same data you tuned on. Hold out a labeled set. A threshold chosen and evaluated on the same ten examples is a coin flip dressed up as a number.
Ignoring the cost of sampling. Five samples per request multiplies your inference cost. Sample only when the cheap signals are ambiguous, or sample a smaller model for agreement.
Abstaining with no user path. "I don't know" with nowhere to go is a dead end. Offer the next step: a human handoff, a link to the source, or a clarifying question.
Forgetting that grounding checks can be wrong too. A string-match grounding check will miss paraphrases and may pass a claim that matches the wrong chunk. Spot-check it.

Next steps

Pick one feature where a wrong answer is expensive. Log the three signals for every request for two weeks, without gating anything yet.
Label 200 to 300 of those requests. Compute ECE and the precision curve using the script above.
Choose a threshold for the highest-risk slice first, route abstentions to a human queue, and measure answer rate and precision weekly.
Only then tune the weights in the composite score. Change one thing at a time so you know what moved the numbers.

Check the current documentation for any model or retrieval API you adopt, since sampling parameters and log-probability access vary by provider and change between versions.

If you are building AI features and want to learn how to design evals, gates, and production monitoring like this with hands-on reviews, join the Joinloop engineering cohort.