Dynamic Few-Shot Prompting: Pick Examples Per Query Instead of Hard-Coding Them
Static few-shot examples work until your inputs get diverse. Learn how to retrieve the most relevant examples per request, keep them diverse, and measure whether it actually beats a fixed prompt.
You wrote a prompt with five good examples. It scored 91% on your test set. Two weeks into production, accuracy on real traffic is 78%, and the failures cluster around inputs that look nothing like your five examples.
This is the ceiling of static few-shot prompting: the same examples are shown for every input, so they can only help inputs that resemble them. The fix is not more examples in the prompt (that costs tokens and can hurt). It is choosing which examples to show for each query.
This post covers dynamic few-shot selection: how it works, a runnable implementation, how to keep the selected examples diverse, and how to prove it beats your static baseline before you ship it.
Why static examples plateau
Few-shot examples do three jobs: they define the output format, show the decision boundary, and demonstrate tricky cases. A fixed set can do the first well, but the second and third depend on the input.
Consider a support-ticket classifier with 12 categories. Five fixed examples can cover at most five categories well. A ticket about a refund dispute gets examples about password resets. The model falls back to its prior, and your prompt is doing nothing for that query.
Dynamic selection flips this: keep a bank of labelled examples (hundreds is fine), and for each incoming query retrieve the k most useful ones.
The architecture
- Example bank: input/output pairs, ideally from real traffic that a human verified.
- Embedding index: embed each example's input once, offline.
- Selector: at request time, embed the query, retrieve candidates, optionally re-rank for diversity.
- Prompt builder: format the chosen examples consistently, then append the real query.
It is the same retrieval machinery as RAG, but you are retrieving demonstrations, not knowledge. That distinction matters for how you evaluate it.
A minimal implementation
This uses sentence-transformers for embeddings and plain NumPy for search. For a bank under ~50k examples you do not need a vector database. Check the library docs for current model names and install steps.
import numpy as np
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
BANK = [
{"input": "I was charged twice for my order", "output": "billing_dispute"},
{"input": "How do I reset my password?", "output": "account_access"},
{"input": "App crashes when I open settings", "output": "bug_report"},
# ... hundreds more, human-verified
]
bank_vecs = model.encode(
[ex["input"] for ex in BANK], normalize_embeddings=True
)
def select_examples(query: str, k: int = 4, pool: int = 20, lam: float = 0.7):
"""Pick k examples: relevant to the query, but not redundant with each other (MMR)."""
q = model.encode([query], normalize_embeddings=True)[0]
sims = bank_vecs @ q
candidates = list(np.argsort(-sims)[:pool])
chosen: list[int] = []
while candidates and len(chosen) < k:
def score(i):
redundancy = max((bank_vecs[i] @ bank_vecs[j] for j in chosen), default=0.0)
return lam * sims[i] - (1 - lam) * redundancy
best = max(candidates, key=score)
chosen.append(best)
candidates.remove(best)
return [BANK[i] for i in chosen]
def build_prompt(query: str) -> str:
shots = select_examples(query)
# Most similar example last: recency bias helps it influence the answer.
shots = list(reversed(shots))
blocks = [f"Ticket: {ex['input']}\nCategory: {ex['output']}" for ex in shots]
return (
"Classify the support ticket into one category.\n\n"
+ "\n\n".join(blocks)
+ f"\n\nTicket: {query}\nCategory:"
)
Three details worth noticing:
- MMR (maximal marginal relevance) balances relevance against redundancy. Without it, you often get four near-duplicates of the same example, which wastes your token budget.
- Embed the input only, not the input plus label. You want "which examples have inputs like this query", not "which examples mention this label".
- Ordering is a hyperparameter. Putting the most similar example last is a common heuristic, but test it on your own data rather than assuming.
Selection strategies compared
| Strategy | How it picks | Strength | Weakness |
|---|---|---|---|
| Static | Fixed hand-picked set | Simple, cacheable, predictable | Only helps inputs near the examples |
| Random from bank | Uniform sample | Cheap diversity | No relevance signal |
| Top-k similarity | Nearest neighbours | High relevance | Redundant, label-skewed picks |
| Similarity + MMR | Relevance with diversity penalty | Best default | Needs tuning of lam and pool |
| Stratified + similarity | Force one example per candidate label | Guarantees label coverage | Needs a label set; can show irrelevant labels |
| Learned selector | Train a model to rank demonstrations | Highest ceiling | Training cost, more moving parts |
Start with similarity + MMR. Move to stratified only if your eval shows rare labels being missed.
The cost you are trading: prompt caching
Static prompts have one big advantage: the prefix is identical across requests, so provider-side prompt caching can discount it. Dynamic examples change the prefix on every call, which usually breaks that benefit for the example section.
A practical layout that keeps most of the savings:
- System instructions and output schema (static, cacheable)
- Dynamic examples
- The user query
Keep everything stable before the examples. Check your provider's caching docs for how prefix matching works, since the rules differ across vendors. Then compare total cost per request, not just accuracy.
Prove it beats the static baseline
Dynamic selection feels smarter, but it adds latency (an embedding call, a search) and complexity. Earn it with an experiment.
def evaluate(predict, dataset):
correct = sum(predict(x["input"]) == x["output"] for x in dataset)
return correct / len(dataset)
# held_out must NOT overlap with BANK
static_acc = evaluate(predict_static, held_out)
dynamic_acc = evaluate(predict_dynamic, held_out)
print(f"static={static_acc:.3f} dynamic={dynamic_acc:.3f}")
Rules that keep the comparison honest:
- Split before you build the bank. If a held-out item (or a near-duplicate) is in the bank, the selector retrieves its own answer and your score is meaningless. Deduplicate by embedding similarity across the split, not only exact match.
- Report per-slice results. Dynamic selection often helps rare categories most and barely moves common ones. An overall average hides that.
- Hold k constant between arms, or you are measuring "more tokens" rather than "better examples".
- Run enough samples. With a few dozen test items, a 3-point difference is noise.
Checklist before shipping
- Bank examples are human-verified, not raw model outputs
- Held-out set is deduplicated against the bank
- Output format is identical across all examples (inconsistency leaks into answers)
- Selector latency measured at p95, not just average
- Fallback to a static set if the embedding service fails
- Cost per request compared against the cached static prompt
- Bank has an owner and a process for adding new verified examples
Common pitfalls
Label leakage through near-duplicates. If your bank contains the same ticket phrased three ways and your test set has a fourth, you are testing memorisation. Cluster and dedupe.
Skewed label distribution. If 70% of the bank is bug_report, top-k will mostly return bug_report examples and bias the model toward it. Rebalance the bank or use stratified selection.
Bad examples compound. A mislabelled example in a static prompt hurts every request equally and is easy to spot. In a bank of 500, a wrong label hurts only some queries and is much harder to find. Log which example IDs were used per request so you can trace a bad answer back to its demonstrations.
Ignoring the instruction. Examples do not replace a clear instruction and schema. If accuracy is low, fix the instruction first; examples are the second lever, not the first.
Over-stuffing k. More examples is not monotonically better. Past a handful, returns flatten while cost and latency keep climbing. Sweep k from 2 to 10 on your eval and pick the knee.
When not to do this
If your input distribution is narrow (one document type, one task shape), a well-chosen static set is cheaper, cacheable, and easier to reason about. Dynamic selection pays off when inputs are diverse, labels are many, or you have a growing bank of verified production data you are otherwise not using.
Next steps
- Export 200 to 500 verified input/output pairs from your logs into a bank.
- Build the selector above and run the static-vs-dynamic comparison on a deduplicated held-out set.
- Log selected example IDs on every request so failures are traceable.
- Revisit monthly: add newly verified failures to the bank. That loop is where the compounding gains come from.
If you want to practise building and evaluating prompt, retrieval, and eval pipelines like this with feedback from working engineers, take a look at the Joinloop AI Engineering cohort.
Want to build this yourself?
Join a live cohort and go from reading about AI to shipping it, with practitioners who do it daily.