From Production Traces to Eval Datasets: A Weekly Failure-Mining Loop for LLM Apps
Hand-written eval sets go stale the week after you ship. Here's a practical loop for mining real production traces, clustering failures, and promoting them into a golden set that grows every week.
You wrote 50 eval cases before launch. They passed. Two weeks into production, users are hitting failures none of those 50 cases anticipated: odd phrasings, pasted spreadsheets, questions in Hinglish, a tool that times out on Tuesdays. Your eval suite is green and your product is not.
The fix is not "write more cases by imagination." It is to treat production traffic as the source of your eval data, and to run a small, boring, weekly loop that converts failures into permanent regression tests.
Why synthetic and hand-written sets drift
Pre-launch eval sets encode what you think users will do. Production encodes what users actually do. The gap shows up in three ways:
- Distribution shift. Real inputs are longer, messier, and more ambiguous than anything a developer types.
- Long-tail failures. The failures that hurt are rare per-category but numerous in aggregate. You will never enumerate them by brainstorming.
- Silent regressions. A prompt tweak fixes failure A and quietly reintroduces failure B. Without B in your suite, you won't know.
Production-trace mining closes this gap. Every failure you see once becomes a test you never see fail again.
The loop at a glance
- Capture full traces (input, retrieved context, tool calls, output, metadata).
- Score them cheaply (rules, user signals, an LLM judge).
- Sample the worst-scoring plus a random slice.
- Cluster failures so you triage themes, not individual traces.
- Label a small batch by hand.
- Promote labelled cases into the golden set with an expected behaviour.
- Gate releases on the golden set in CI.
A realistic budget is one hour a week reviewing 20โ40 traces. At that pace the golden set grows by 10โ20 real cases a week, which puts you in the hundreds of cases within a quarter. Treat those numbers as a rule of thumb, not a guarantee.
Step 1: Capture traces you can actually reuse
A trace is only useful as an eval case if you can replay it. That means logging more than the final answer. Using OpenTelemetry-style spans (or Langfuse, Braintrust, LangSmith, etc.), store:
- the raw user input and conversation history
- retrieved chunks and their IDs (for RAG)
- tool calls with arguments and results
- the model, prompt version, and parameters
- the final output and latency/cost
Redact PII at capture time, not later. If you cannot legally keep a trace, you cannot turn it into a test.
Step 2: Score cheaply, in layers
You cannot read everything. Build a cheap triage score from signals you already have:
from dataclasses import dataclass
@dataclass
class Trace:
id: str
user_input: str
output: str
thumbs_down: bool = False
retried_by_user: bool = False
tool_errors: int = 0
judge_score: float | None = None # 0..1 from an LLM judge, optional
def triage_score(t: Trace) -> float:
"""Higher = more likely a failure worth looking at."""
score = 0.0
score += 3.0 if t.thumbs_down else 0.0
score += 2.0 if t.retried_by_user else 0.0 # user rephrased: we missed
score += 1.5 * min(t.tool_errors, 3)
if t.judge_score is not None:
score += 2.0 * (1.0 - t.judge_score)
if len(t.output.strip()) < 20: # suspiciously empty answer
score += 1.0
return score
Implicit signals (rephrasing, abandonment, copy-paste of the answer, escalation to a human) are often more honest than thumbs-down buttons, which few users click. If you use an LLM judge, calibrate it against human labels first; an uncalibrated judge just launders noise into your dataset.
Step 3: Sample the worst and a random slice
Only reviewing the worst-scoring traces biases your set toward failures your scorer already understands. Always add a random sample (say 20% of the batch) to catch failures the scorer is blind to.
import random
def weekly_sample(traces: list[Trace], k: int = 40, random_frac: float = 0.2):
ranked = sorted(traces, key=triage_score, reverse=True)
n_random = int(k * random_frac)
worst = ranked[: k - n_random]
rest = ranked[k - n_random :]
return worst + random.sample(rest, min(n_random, len(rest)))
Step 4: Cluster so you triage themes
Forty individual traces is a chore; five named clusters is an insight. Embed the user inputs (or a short failure summary) and cluster them. HDBSCAN is a good fit because you don't know the number of clusters in advance and it labels outliers as noise instead of forcing them into a group.
import numpy as np
from sklearn.cluster import HDBSCAN # scikit-learn >= 1.3
def cluster_failures(embeddings: np.ndarray, min_cluster_size: int = 3):
labels = HDBSCAN(min_cluster_size=min_cluster_size).fit_predict(embeddings)
clusters: dict[int, list[int]] = {}
for idx, lab in enumerate(labels):
clusters.setdefault(int(lab), []).append(idx)
return clusters # label -1 = unclustered noise; review it separately
Then name each cluster by reading three examples ("refund questions that hit the wrong policy doc", "date math in the calendar tool"). Naming is the valuable human step; it turns a pile of traces into a roadmap. Don't skip the -1 noise bucket: one-off weirdness there is often your next cluster.
Step 5: Label with an expected behaviour, not just "bad"
A failed trace is not yet a test. Each promoted case needs a definition of correct. Pick the cheapest check that works:
| Case type | Expected-behaviour field | Checker |
|---|---|---|
| Factual / RAG | Must-cite chunk IDs, key facts | Retrieval recall + substring/judge |
| Tool use | Expected tool and argument shape | Deterministic assertion |
| Format | JSON schema or regex | Schema validation |
| Refusal / safety | Must refuse or must not leak | Rule + judge |
| Open-ended quality | Rubric (2โ4 criteria) | Calibrated LLM judge |
Prefer deterministic checks wherever they fit. Keep the judge for what rules cannot express.
A golden-set record might look like this:
{
"id": "prod-2026-10-w1-014",
"source_trace": "tr_8f21...",
"input": "can i get my money back if i pause my cohort",
"tags": ["refund-policy", "paraphrase"],
"expect": {
"must_cite": ["policy/refunds#pause"],
"must_not_contain": ["30 days no questions asked"],
"judge_rubric": ["States the pause rule", "Does not invent a deadline"]
},
"added": "2026-10-05"
}
Step 6: Gate releases in CI
The point of all this is that nothing ships if it re-breaks a known failure. A minimal runner:
import json, sys
def run_suite(path: str, answer_fn, check_fn, threshold: float = 0.9) -> bool:
cases = [json.loads(l) for l in open(path)]
passed = sum(check_fn(c, answer_fn(c["input"])) for c in cases)
rate = passed / len(cases)
print(f"{passed}/{len(cases)} passed ({rate:.1%})")
return rate >= threshold
if __name__ == "__main__":
ok = run_suite("evals/golden.jsonl", my_app, my_checker)
sys.exit(0 if ok else 1)
Track pass rate per tag, not just overall. A flat 92% can hide a collapse in one cluster that you just fixed last week.
Tooling options
| Approach | Good for | Trade-off |
|---|---|---|
| Langfuse (open source, self-hostable) | Tracing, datasets, scores in one place | You operate it if self-hosted |
| Braintrust / LangSmith / Confident AI | One-click trace-to-test promotion, hosted UI | Vendor lock-in, per-seat cost |
| Homegrown (OTel + Postgres + JSONL) | Full control, data stays in-house | You build the review UI |
Check each vendor's current docs for feature details; they change quickly. The loop above works with any of them, or with none.
Common pitfalls
- Training on your test set. If you also use promoted cases as few-shot examples or fine-tuning data, hold out a slice that is never used for tuning.
- Dataset rot. Cases tied to an old policy or product version become wrong. Date every case and review for staleness quarterly.
- Judge drift. Re-check judge agreement with human labels whenever you change the judge model or rubric.
- Privacy leaks. Raw traces contain user data. Redact before storage and restrict who can browse the dataset.
- Over-weighting loud failures. Weight clusters by frequency and business impact, not by how annoying a single example was.
- Never retiring cases. A suite that only grows gets slow and expensive. Keep a fast "smoke" subset for every PR and run the full set nightly.
A checklist for this week
- Traces include retrieved context, tool calls, and prompt version
- PII redaction happens at capture
- A triage score exists (even three signals is enough)
- A 45-minute weekly review is on the calendar, with a named owner
- Promoted cases carry an expected-behaviour field
- CI fails the build when the golden set pass rate drops
- A held-out slice is never used for tuning
Next steps
Start small: pull last week's traces, apply the triage score, and promote ten cases. Wire them into CI before you do anything fancier. After a month you will have an eval set that reflects your real users, and a regression you can no longer ship by accident.
If you want to build this loop hands-on with feedback from practitioners, our AI engineering cohort covers evals, observability, and production agents end to end. Product managers owning AI quality can look at the PM cohort.
Want to build this yourself?
Join a live cohort and go from reading about AI to shipping it, with practitioners who do it daily.