โ† Back to Blog
AI EngineeringLLMsTools

From Production Traces to Eval Datasets: A Weekly Failure-Mining Loop for LLM Apps

Hand-written eval sets go stale the week after you ship. Here's a practical loop for mining real production traces, clustering failures, and promoting them into a golden set that grows every week.

By Joinloopยท5 October 2026ยท8 min read
ShareLinkedInXReddit
โ›๏ธ

You wrote 50 eval cases before launch. They passed. Two weeks into production, users are hitting failures none of those 50 cases anticipated: odd phrasings, pasted spreadsheets, questions in Hinglish, a tool that times out on Tuesdays. Your eval suite is green and your product is not.

The fix is not "write more cases by imagination." It is to treat production traffic as the source of your eval data, and to run a small, boring, weekly loop that converts failures into permanent regression tests.

Why synthetic and hand-written sets drift

Pre-launch eval sets encode what you think users will do. Production encodes what users actually do. The gap shows up in three ways:

  • Distribution shift. Real inputs are longer, messier, and more ambiguous than anything a developer types.
  • Long-tail failures. The failures that hurt are rare per-category but numerous in aggregate. You will never enumerate them by brainstorming.
  • Silent regressions. A prompt tweak fixes failure A and quietly reintroduces failure B. Without B in your suite, you won't know.

Production-trace mining closes this gap. Every failure you see once becomes a test you never see fail again.

The loop at a glance

  1. Capture full traces (input, retrieved context, tool calls, output, metadata).
  2. Score them cheaply (rules, user signals, an LLM judge).
  3. Sample the worst-scoring plus a random slice.
  4. Cluster failures so you triage themes, not individual traces.
  5. Label a small batch by hand.
  6. Promote labelled cases into the golden set with an expected behaviour.
  7. Gate releases on the golden set in CI.

A realistic budget is one hour a week reviewing 20โ€“40 traces. At that pace the golden set grows by 10โ€“20 real cases a week, which puts you in the hundreds of cases within a quarter. Treat those numbers as a rule of thumb, not a guarantee.

Step 1: Capture traces you can actually reuse

A trace is only useful as an eval case if you can replay it. That means logging more than the final answer. Using OpenTelemetry-style spans (or Langfuse, Braintrust, LangSmith, etc.), store:

  • the raw user input and conversation history
  • retrieved chunks and their IDs (for RAG)
  • tool calls with arguments and results
  • the model, prompt version, and parameters
  • the final output and latency/cost

Redact PII at capture time, not later. If you cannot legally keep a trace, you cannot turn it into a test.

Step 2: Score cheaply, in layers

You cannot read everything. Build a cheap triage score from signals you already have:

from dataclasses import dataclass

@dataclass
class Trace:
    id: str
    user_input: str
    output: str
    thumbs_down: bool = False
    retried_by_user: bool = False
    tool_errors: int = 0
    judge_score: float | None = None  # 0..1 from an LLM judge, optional

def triage_score(t: Trace) -> float:
    """Higher = more likely a failure worth looking at."""
    score = 0.0
    score += 3.0 if t.thumbs_down else 0.0
    score += 2.0 if t.retried_by_user else 0.0   # user rephrased: we missed
    score += 1.5 * min(t.tool_errors, 3)
    if t.judge_score is not None:
        score += 2.0 * (1.0 - t.judge_score)
    if len(t.output.strip()) < 20:                # suspiciously empty answer
        score += 1.0
    return score

Implicit signals (rephrasing, abandonment, copy-paste of the answer, escalation to a human) are often more honest than thumbs-down buttons, which few users click. If you use an LLM judge, calibrate it against human labels first; an uncalibrated judge just launders noise into your dataset.

Step 3: Sample the worst and a random slice

Only reviewing the worst-scoring traces biases your set toward failures your scorer already understands. Always add a random sample (say 20% of the batch) to catch failures the scorer is blind to.

import random

def weekly_sample(traces: list[Trace], k: int = 40, random_frac: float = 0.2):
    ranked = sorted(traces, key=triage_score, reverse=True)
    n_random = int(k * random_frac)
    worst = ranked[: k - n_random]
    rest = ranked[k - n_random :]
    return worst + random.sample(rest, min(n_random, len(rest)))

Step 4: Cluster so you triage themes

Forty individual traces is a chore; five named clusters is an insight. Embed the user inputs (or a short failure summary) and cluster them. HDBSCAN is a good fit because you don't know the number of clusters in advance and it labels outliers as noise instead of forcing them into a group.

import numpy as np
from sklearn.cluster import HDBSCAN  # scikit-learn >= 1.3

def cluster_failures(embeddings: np.ndarray, min_cluster_size: int = 3):
    labels = HDBSCAN(min_cluster_size=min_cluster_size).fit_predict(embeddings)
    clusters: dict[int, list[int]] = {}
    for idx, lab in enumerate(labels):
        clusters.setdefault(int(lab), []).append(idx)
    return clusters  # label -1 = unclustered noise; review it separately

Then name each cluster by reading three examples ("refund questions that hit the wrong policy doc", "date math in the calendar tool"). Naming is the valuable human step; it turns a pile of traces into a roadmap. Don't skip the -1 noise bucket: one-off weirdness there is often your next cluster.

Step 5: Label with an expected behaviour, not just "bad"

A failed trace is not yet a test. Each promoted case needs a definition of correct. Pick the cheapest check that works:

Case type Expected-behaviour field Checker
Factual / RAG Must-cite chunk IDs, key facts Retrieval recall + substring/judge
Tool use Expected tool and argument shape Deterministic assertion
Format JSON schema or regex Schema validation
Refusal / safety Must refuse or must not leak Rule + judge
Open-ended quality Rubric (2โ€“4 criteria) Calibrated LLM judge

Prefer deterministic checks wherever they fit. Keep the judge for what rules cannot express.

A golden-set record might look like this:

{
  "id": "prod-2026-10-w1-014",
  "source_trace": "tr_8f21...",
  "input": "can i get my money back if i pause my cohort",
  "tags": ["refund-policy", "paraphrase"],
  "expect": {
    "must_cite": ["policy/refunds#pause"],
    "must_not_contain": ["30 days no questions asked"],
    "judge_rubric": ["States the pause rule", "Does not invent a deadline"]
  },
  "added": "2026-10-05"
}

Step 6: Gate releases in CI

The point of all this is that nothing ships if it re-breaks a known failure. A minimal runner:

import json, sys

def run_suite(path: str, answer_fn, check_fn, threshold: float = 0.9) -> bool:
    cases = [json.loads(l) for l in open(path)]
    passed = sum(check_fn(c, answer_fn(c["input"])) for c in cases)
    rate = passed / len(cases)
    print(f"{passed}/{len(cases)} passed ({rate:.1%})")
    return rate >= threshold

if __name__ == "__main__":
    ok = run_suite("evals/golden.jsonl", my_app, my_checker)
    sys.exit(0 if ok else 1)

Track pass rate per tag, not just overall. A flat 92% can hide a collapse in one cluster that you just fixed last week.

Tooling options

Approach Good for Trade-off
Langfuse (open source, self-hostable) Tracing, datasets, scores in one place You operate it if self-hosted
Braintrust / LangSmith / Confident AI One-click trace-to-test promotion, hosted UI Vendor lock-in, per-seat cost
Homegrown (OTel + Postgres + JSONL) Full control, data stays in-house You build the review UI

Check each vendor's current docs for feature details; they change quickly. The loop above works with any of them, or with none.

Common pitfalls

  • Training on your test set. If you also use promoted cases as few-shot examples or fine-tuning data, hold out a slice that is never used for tuning.
  • Dataset rot. Cases tied to an old policy or product version become wrong. Date every case and review for staleness quarterly.
  • Judge drift. Re-check judge agreement with human labels whenever you change the judge model or rubric.
  • Privacy leaks. Raw traces contain user data. Redact before storage and restrict who can browse the dataset.
  • Over-weighting loud failures. Weight clusters by frequency and business impact, not by how annoying a single example was.
  • Never retiring cases. A suite that only grows gets slow and expensive. Keep a fast "smoke" subset for every PR and run the full set nightly.

A checklist for this week

  • Traces include retrieved context, tool calls, and prompt version
  • PII redaction happens at capture
  • A triage score exists (even three signals is enough)
  • A 45-minute weekly review is on the calendar, with a named owner
  • Promoted cases carry an expected-behaviour field
  • CI fails the build when the golden set pass rate drops
  • A held-out slice is never used for tuning

Next steps

Start small: pull last week's traces, apply the triage score, and promote ten cases. Wire them into CI before you do anything fancier. After a month you will have an eval set that reflects your real users, and a regression you can no longer ship by accident.

If you want to build this loop hands-on with feedback from practitioners, our AI engineering cohort covers evals, observability, and production agents end to end. Product managers owning AI quality can look at the PM cohort.

ShareLinkedInXReddit

Want to build this yourself?

Join a live cohort and go from reading about AI to shipping it, with practitioners who do it daily.