Self-Consistency and Best-of-N: Buying LLM Reliability With Extra Samples
When one LLM call is right 80% of the time, sampling several and picking the winner can push that much higher. Learn majority voting, best-of-N with a verifier, and how to decide whether the extra cost is worth it.
You have a classification, extraction, or reasoning step that is right about 80% of the time. Prompt tweaks have stopped helping. Fine-tuning is weeks away. There is a cheaper lever you can pull today: call the model more than once and combine the answers.
This is the idea behind self-consistency (Wang et al., 2022) and best-of-N sampling. Instead of trusting one sample, you draw several at a non-zero temperature, then either vote or let a verifier choose. It trades tokens and latency for accuracy, and it is one of the few techniques where you can estimate the payoff on paper before writing any code.
Why it works (and when it does not)
A single sample from an LLM is one draw from a distribution over reasoning paths. For problems with a single checkable answer (math, SQL results, a category label, an extracted field), wrong paths tend to scatter across many different wrong answers, while right paths converge on the same one. Majority voting exploits that asymmetry.
If each independent sample is correct with probability p and wrong answers are spread out, voting over N samples pushes accuracy above p. The catch is the word independent:
- Samples from the same model share the same blind spots. If the model confidently believes a wrong fact, ten samples will agree on it.
- At temperature 0 you get the same answer every time, so there is nothing to vote on. You need diversity, typically temperature 0.5 to 1.0.
- Free-form text has no natural "same answer." You must canonicalise the output first (normalise a number, parse JSON, map to a label).
Rule of thumb: self-consistency helps when errors are random, and does little when errors are systematic. Check which one you have by running the same input 10 times and looking at the disagreement.
Technique 1: Majority vote (self-consistency)
Sample N answers, extract a canonical final answer from each, return the most common. Here is a runnable Python version using the Anthropic SDK with parallel calls. Swap in any provider; check the official docs for current model IDs.
import asyncio
import re
from collections import Counter
from anthropic import AsyncAnthropic
client = AsyncAnthropic()
MODEL = "claude-sonnet-5-5" # verify the current model ID in the docs
PROMPT = """Solve the problem. Think step by step, then finish with a line
of the form `ANSWER: <number>` and nothing after it.
Problem: {problem}"""
def extract(text: str) -> str | None:
m = re.search(r"ANSWER:\s*(-?[\d,]*\.?\d+)", text)
return m.group(1).replace(",", "") if m else None
async def one_sample(problem: str, temperature: float) -> str | None:
resp = await client.messages.create(
model=MODEL,
max_tokens=1024,
temperature=temperature,
messages=[{"role": "user", "content": PROMPT.format(problem=problem)}],
)
return extract(resp.content[0].text)
async def self_consistent(problem: str, n: int = 5, temperature: float = 0.8):
answers = await asyncio.gather(
*[one_sample(problem, temperature) for _ in range(n)]
)
valid = [a for a in answers if a is not None]
if not valid:
return None, 0.0
winner, votes = Counter(valid).most_common(1)[0]
return winner, votes / len(answers) # agreement ratio
if __name__ == "__main__":
ans, agreement = asyncio.run(
self_consistent("A train travels 120 km in 1.5 hours. Speed in km/h?")
)
print(ans, agreement)
Two details matter. First, extract discards unparseable samples, so formatting failures do not poison the vote. Second, the agreement ratio is a free confidence signal: 5/5 votes is very different from 2/5. That leads to the most useful production pattern below.
Technique 2: Best-of-N with a verifier
Voting needs a canonical answer. For open-ended outputs (a SQL query, a code patch, a summary), generate N candidates and score each with something that can tell good from bad:
| Verifier type | Example | Strength | Weakness |
|---|---|---|---|
| Programmatic check | Run SQL, execute unit tests, validate JSON schema | Objective, cheap, deterministic | Only exists for some tasks |
| Reward or ranker model | A trained scorer | Fast at scale | Needs training data |
| LLM judge | A second prompt rating each candidate | Works on anything | Biased, needs calibration |
Programmatic verifiers are the gold standard. Code generation with tests is the classic case: sample 5 patches, run the test suite, return the first that passes.
import asyncio
import sqlite3
async def best_of_n_sql(question: str, schema: str, n: int = 4):
candidates = await asyncio.gather(
*[generate_sql(question, schema, temperature=0.9) for _ in range(n)]
)
scored = []
for sql in candidates:
ok, rows = try_execute(sql)
scored.append((ok, sql, rows))
# Prefer candidates that execute; among those, vote on the result set.
runnable = [(sql, tuple(map(tuple, rows))) for ok, sql, rows in scored if ok]
if not runnable:
return None
from collections import Counter
top_result, _ = Counter(r for _, r in runnable).most_common(1)[0]
return next(sql for sql, r in runnable if r == top_result)
def try_execute(sql: str, db_path: str = "app.db"):
try:
con = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
rows = con.execute(sql).fetchall()
return True, rows
except sqlite3.Error:
return False, []
Notice the hybrid: filter by executes, then vote on the result set rather than the SQL text. Two differently written queries that return the same rows count as agreeing. (generate_sql is your own single-call generator; run read-only connections like above when executing model output.)
If you must use an LLM judge, calibrate it first; our guide to LLM-as-judge calibration covers how.
Technique 3: Escalate only when uncertain
Running N=5 on every request multiplies cost by 5. A better design uses agreement as a gate:
- Draw 3 samples.
- If all agree, return immediately (cost: 3x).
- If they disagree, draw more samples, or escalate to a stronger model, or route to a human.
Most inputs are easy, so the average multiplier stays close to 3x, and you spend extra only where it pays. The disagreement rate is also a monitoring metric: if it creeps up after a prompt or model change, something regressed.
Cost and latency: do the arithmetic first
Parallel sampling keeps latency near one call (you wait for the slowest sample), but cost scales linearly with N. Before committing, estimate:
expected_cost = N * (input_tokens * in_price + output_tokens * out_price)
net_benefit = (acc_with_voting - acc_single) * value_per_correct_answer
- (expected_cost - single_cost)
Two levers reduce the bill. Prompt caching makes the repeated input cheap, since all N calls share the same prefix (see prompt caching). Batch APIs halve cost for offline jobs (see batch API cost optimization). Also compare against the alternative: a stronger model once may beat a weaker model five times, so test both.
Decision table
| Situation | Use it? | Why |
|---|---|---|
| Math, label, or field extraction with a single right answer | Yes | Easy to canonicalise and vote |
| Code or SQL with tests or execution | Yes | Strong programmatic verifier |
| Long free-form writing | Rarely | No canonical answer; judge noise dominates |
| Model errors are systematic (missing knowledge) | No | Samples agree on the same wrong answer |
| Latency budget is tight and calls cannot be parallelised | No | Serial sampling multiplies latency |
| Reasoning models with built-in thinking | Test first | They may already internalise this; compare on your eval set |
Common pitfalls
- Temperature 0 with N samples. You pay N times for the same answer. Raise temperature.
- Voting on raw text. "42", "42.0" and "The answer is 42" are different strings. Canonicalise before counting.
- Ignoring ties. Decide the tie-break rule in advance (higher-confidence sample, stronger model, or abstain).
- Correlated errors mistaken for noise. Measure on a labelled set whether voting actually helps. If accuracy at N=5 is within noise of N=1, stop.
- Treating agreement as truth. 5/5 agreement means the model is consistent, not correct. Pair it with an eval set.
- Silent cost creep. Log
N, agreement, and tokens per request so finance does not discover it for you.
A minimal rollout plan
- Build a labelled eval set of 100 to 300 real inputs for the step.
- Measure single-sample accuracy, then accuracy at N = 3, 5, 9 with temperature 0.7.
- Plot accuracy against cost. Pick the knee of the curve, usually N = 3 to 5.
- Add the escalate-on-disagreement gate and log agreement per request.
- Re-run the eval whenever the model or prompt changes. The benefit shrinks as base models improve.
For the eval-set mechanics, see evaluating LLM agents in production. For the statistics of "did it really improve", see pass@k vs pass^k.
Extra samples are not magic, but they are measurable, reversible, and shippable this week. That makes them a good first move before heavier investments.
If you want to practise techniques like this on real codebases, the Joinloop Engineering Cohort is built for that. Product managers deciding where extra inference spend is justified can start with the Joinloop PM Cohort.
Want to build this yourself?
Join a live cohort and go from reading about AI to shipping it, with practitioners who do it daily.