โ† Back to Blog
AI EngineeringLLMsAI for PMs

Self-Consistency and Best-of-N: Buying LLM Reliability With Extra Samples

When one LLM call is right 80% of the time, sampling several and picking the winner can push that much higher. Learn majority voting, best-of-N with a verifier, and how to decide whether the extra cost is worth it.

By Joinloopยท16 November 2026ยท8 min read
ShareLinkedInXReddit
๐Ÿ—ณ๏ธ

You have a classification, extraction, or reasoning step that is right about 80% of the time. Prompt tweaks have stopped helping. Fine-tuning is weeks away. There is a cheaper lever you can pull today: call the model more than once and combine the answers.

This is the idea behind self-consistency (Wang et al., 2022) and best-of-N sampling. Instead of trusting one sample, you draw several at a non-zero temperature, then either vote or let a verifier choose. It trades tokens and latency for accuracy, and it is one of the few techniques where you can estimate the payoff on paper before writing any code.

Why it works (and when it does not)

A single sample from an LLM is one draw from a distribution over reasoning paths. For problems with a single checkable answer (math, SQL results, a category label, an extracted field), wrong paths tend to scatter across many different wrong answers, while right paths converge on the same one. Majority voting exploits that asymmetry.

If each independent sample is correct with probability p and wrong answers are spread out, voting over N samples pushes accuracy above p. The catch is the word independent:

  • Samples from the same model share the same blind spots. If the model confidently believes a wrong fact, ten samples will agree on it.
  • At temperature 0 you get the same answer every time, so there is nothing to vote on. You need diversity, typically temperature 0.5 to 1.0.
  • Free-form text has no natural "same answer." You must canonicalise the output first (normalise a number, parse JSON, map to a label).

Rule of thumb: self-consistency helps when errors are random, and does little when errors are systematic. Check which one you have by running the same input 10 times and looking at the disagreement.

Technique 1: Majority vote (self-consistency)

Sample N answers, extract a canonical final answer from each, return the most common. Here is a runnable Python version using the Anthropic SDK with parallel calls. Swap in any provider; check the official docs for current model IDs.

import asyncio
import re
from collections import Counter

from anthropic import AsyncAnthropic

client = AsyncAnthropic()
MODEL = "claude-sonnet-5-5"  # verify the current model ID in the docs

PROMPT = """Solve the problem. Think step by step, then finish with a line
of the form `ANSWER: <number>` and nothing after it.

Problem: {problem}"""


def extract(text: str) -> str | None:
    m = re.search(r"ANSWER:\s*(-?[\d,]*\.?\d+)", text)
    return m.group(1).replace(",", "") if m else None


async def one_sample(problem: str, temperature: float) -> str | None:
    resp = await client.messages.create(
        model=MODEL,
        max_tokens=1024,
        temperature=temperature,
        messages=[{"role": "user", "content": PROMPT.format(problem=problem)}],
    )
    return extract(resp.content[0].text)


async def self_consistent(problem: str, n: int = 5, temperature: float = 0.8):
    answers = await asyncio.gather(
        *[one_sample(problem, temperature) for _ in range(n)]
    )
    valid = [a for a in answers if a is not None]
    if not valid:
        return None, 0.0
    winner, votes = Counter(valid).most_common(1)[0]
    return winner, votes / len(answers)  # agreement ratio


if __name__ == "__main__":
    ans, agreement = asyncio.run(
        self_consistent("A train travels 120 km in 1.5 hours. Speed in km/h?")
    )
    print(ans, agreement)

Two details matter. First, extract discards unparseable samples, so formatting failures do not poison the vote. Second, the agreement ratio is a free confidence signal: 5/5 votes is very different from 2/5. That leads to the most useful production pattern below.

Technique 2: Best-of-N with a verifier

Voting needs a canonical answer. For open-ended outputs (a SQL query, a code patch, a summary), generate N candidates and score each with something that can tell good from bad:

Verifier type Example Strength Weakness
Programmatic check Run SQL, execute unit tests, validate JSON schema Objective, cheap, deterministic Only exists for some tasks
Reward or ranker model A trained scorer Fast at scale Needs training data
LLM judge A second prompt rating each candidate Works on anything Biased, needs calibration

Programmatic verifiers are the gold standard. Code generation with tests is the classic case: sample 5 patches, run the test suite, return the first that passes.

import asyncio
import sqlite3


async def best_of_n_sql(question: str, schema: str, n: int = 4):
    candidates = await asyncio.gather(
        *[generate_sql(question, schema, temperature=0.9) for _ in range(n)]
    )
    scored = []
    for sql in candidates:
        ok, rows = try_execute(sql)
        scored.append((ok, sql, rows))

    # Prefer candidates that execute; among those, vote on the result set.
    runnable = [(sql, tuple(map(tuple, rows))) for ok, sql, rows in scored if ok]
    if not runnable:
        return None
    from collections import Counter
    top_result, _ = Counter(r for _, r in runnable).most_common(1)[0]
    return next(sql for sql, r in runnable if r == top_result)


def try_execute(sql: str, db_path: str = "app.db"):
    try:
        con = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
        rows = con.execute(sql).fetchall()
        return True, rows
    except sqlite3.Error:
        return False, []

Notice the hybrid: filter by executes, then vote on the result set rather than the SQL text. Two differently written queries that return the same rows count as agreeing. (generate_sql is your own single-call generator; run read-only connections like above when executing model output.)

If you must use an LLM judge, calibrate it first; our guide to LLM-as-judge calibration covers how.

Technique 3: Escalate only when uncertain

Running N=5 on every request multiplies cost by 5. A better design uses agreement as a gate:

  1. Draw 3 samples.
  2. If all agree, return immediately (cost: 3x).
  3. If they disagree, draw more samples, or escalate to a stronger model, or route to a human.

Most inputs are easy, so the average multiplier stays close to 3x, and you spend extra only where it pays. The disagreement rate is also a monitoring metric: if it creeps up after a prompt or model change, something regressed.

Cost and latency: do the arithmetic first

Parallel sampling keeps latency near one call (you wait for the slowest sample), but cost scales linearly with N. Before committing, estimate:

expected_cost = N * (input_tokens * in_price + output_tokens * out_price)
net_benefit   = (acc_with_voting - acc_single) * value_per_correct_answer
                - (expected_cost - single_cost)

Two levers reduce the bill. Prompt caching makes the repeated input cheap, since all N calls share the same prefix (see prompt caching). Batch APIs halve cost for offline jobs (see batch API cost optimization). Also compare against the alternative: a stronger model once may beat a weaker model five times, so test both.

Decision table

Situation Use it? Why
Math, label, or field extraction with a single right answer Yes Easy to canonicalise and vote
Code or SQL with tests or execution Yes Strong programmatic verifier
Long free-form writing Rarely No canonical answer; judge noise dominates
Model errors are systematic (missing knowledge) No Samples agree on the same wrong answer
Latency budget is tight and calls cannot be parallelised No Serial sampling multiplies latency
Reasoning models with built-in thinking Test first They may already internalise this; compare on your eval set

Common pitfalls

  • Temperature 0 with N samples. You pay N times for the same answer. Raise temperature.
  • Voting on raw text. "42", "42.0" and "The answer is 42" are different strings. Canonicalise before counting.
  • Ignoring ties. Decide the tie-break rule in advance (higher-confidence sample, stronger model, or abstain).
  • Correlated errors mistaken for noise. Measure on a labelled set whether voting actually helps. If accuracy at N=5 is within noise of N=1, stop.
  • Treating agreement as truth. 5/5 agreement means the model is consistent, not correct. Pair it with an eval set.
  • Silent cost creep. Log N, agreement, and tokens per request so finance does not discover it for you.

A minimal rollout plan

  1. Build a labelled eval set of 100 to 300 real inputs for the step.
  2. Measure single-sample accuracy, then accuracy at N = 3, 5, 9 with temperature 0.7.
  3. Plot accuracy against cost. Pick the knee of the curve, usually N = 3 to 5.
  4. Add the escalate-on-disagreement gate and log agreement per request.
  5. Re-run the eval whenever the model or prompt changes. The benefit shrinks as base models improve.

For the eval-set mechanics, see evaluating LLM agents in production. For the statistics of "did it really improve", see pass@k vs pass^k.

Extra samples are not magic, but they are measurable, reversible, and shippable this week. That makes them a good first move before heavier investments.

If you want to practise techniques like this on real codebases, the Joinloop Engineering Cohort is built for that. Product managers deciding where extra inference spend is justified can start with the Joinloop PM Cohort.

ShareLinkedInXReddit

Want to build this yourself?

Join a live cohort and go from reading about AI to shipping it, with practitioners who do it daily.