User Feedback Signals for AI Features: Thumbs, Edits, and Implicit Behaviour as Eval Data

You shipped an AI feature. You added a thumbs-up and thumbs-down button. Two weeks later, 0.8% of responses have been rated, 70% of those ratings are thumbs-down from people who were already annoyed, and nobody can say whether the feature is getting better or worse.

This is the default outcome. Explicit ratings are sparse, biased toward extremes, and carry no information about why. The richer signal is in what users do after the model responds: whether they edit the output, regenerate it, copy it, or leave. This post shows how to capture those signals, store them in a way you can query, and convert them into evaluation data.

Why explicit ratings alone fail

Three structural problems:

1. Sparsity. Most users never click. Rates of a percent or two are common for inline rating widgets; check your own numbers rather than trusting any benchmark.
2. Selection bias. The people who click are either delighted or furious. The silent middle, which is most of your traffic, is invisible.
3. No label for the failure. A thumbs-down does not tell you whether the answer was wrong, too long, off-tone, or simply slow.

You still want explicit feedback, but treat it as one signal among several, not the source of truth.

The signal hierarchy

Think of feedback as a ladder, from cheap-and-noisy to expensive-and-precise.

Signal	Type	Volume	Precision	Example
Thumbs up/down	Explicit	Very low	Low	Rating widget
Reason tag after thumbs-down	Explicit	Very low	High	"Incorrect", "Too long", "Wrong tone"
Edit distance between draft and final	Implicit	High	High	User rewrites 60% of an email draft
Accept / reject of a suggestion	Implicit	High	High	Code completion tab-accepted or dismissed
Regenerate / retry	Implicit	Medium	Medium	User asks again within 30 seconds
Copy / export / send	Implicit	Medium	Medium	Output copied or used downstream
Abandonment	Implicit	High	Low	Session ends right after the response
Escalation to human	Implicit	Low	High	"Talk to an agent" clicked

The highest-value signals for most products are edits and accept/reject, because they are the user doing the grading as part of their normal workflow, with no extra effort.

Design principle: log the event, derive the label later

Do not decide at capture time what counts as "good". Log raw events with enough context to re-derive labels as your understanding improves. Every event needs:

• a stable response_id linking to the exact prompt, model, prompt version, and retrieved context that produced it
• the user action and a timestamp
• any payload (the edited text, the reason tag)

If you cannot join a feedback event back to the full generation trace, the event is nearly useless for evals. If you already trace your LLM calls, reuse that trace ID as response_id.

Schema

A single narrow events table is enough to start. Postgres example:

CREATE TABLE ai_feedback_events (
  id            BIGSERIAL PRIMARY KEY,
  response_id   UUID        NOT NULL,   -- joins to your generation trace
  user_id       TEXT        NOT NULL,
  event_type    TEXT        NOT NULL,   -- thumb, edit, accept, reject, regenerate, copy, escalate
  payload       JSONB       NOT NULL DEFAULT '{}',
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX ON ai_feedback_events (response_id);
CREATE INDEX ON ai_feedback_events (event_type, created_at);

Capturing edits as feedback

Edits are the richest implicit signal. When a user modifies a generated draft before using it, the diff tells you what the model got wrong. Here is a small Python helper that logs a normalised edit ratio:

import difflib
import json
import uuid
from datetime import datetime, timezone


def edit_ratio(draft: str, final: str) -> float:
    """0.0 = identical, 1.0 = completely rewritten."""
    return 1.0 - difflib.SequenceMatcher(None, draft, final).ratio()


def log_edit(db, response_id: uuid.UUID, user_id: str, draft: str, final: str):
    ratio = edit_ratio(draft, final)
    payload = {
        "edit_ratio": round(ratio, 3),
        "draft_len": len(draft),
        "final_len": len(final),
        # Store the texts only if your privacy policy allows it.
        "final_text": final,
    }
    db.execute(
        """
        INSERT INTO ai_feedback_events (response_id, user_id, event_type, payload, created_at)
        VALUES (%s, %s, 'edit', %s, %s)
        """,
        (str(response_id), user_id, json.dumps(payload), datetime.now(timezone.utc)),
    )

Interpretation guide: an edit ratio under roughly 0.1 usually means "accepted with minor polish", while above 0.6 usually means "the draft was a bad starting point". Calibrate these thresholds by reading a sample of real edits, since they depend heavily on content type.

Capturing retries

A regenerate within a short window of the previous response is a strong negative signal. In TypeScript, on the server:

type FeedbackEvent = {
  responseId: string;
  userId: string;
  eventType: "regenerate" | "accept" | "reject" | "copy" | "thumb";
  payload?: Record<string, unknown>;
};

const lastResponseAt = new Map<string, number>(); // userId -> epoch ms (use Redis in production)

export function classifyRequest(userId: string, now = Date.now()): "fresh" | "retry" {
  const prev = lastResponseAt.get(userId);
  lastResponseAt.set(userId, now);
  return prev !== undefined && now - prev < 30_000 ? "retry" : "fresh";
}

When classifyRequest returns "retry", log a regenerate event against the previous response_id. Be careful: a retry can also mean the user is exploring variations, so combine it with other signals before treating it as a failure.

Turning signals into a composite label

Raw events are not labels. Build a simple, transparent rule first, and only move to a learned model once you have enough volume to validate it.

def derive_label(events: list[dict]) -> str:
    types = {e["event_type"] for e in events}
    edits = [e for e in events if e["event_type"] == "edit"]
    max_edit = max((e["payload"]["edit_ratio"] for e in edits), default=0.0)

    if "escalate" in types or "reject" in types:
        return "bad"
    if any(e["event_type"] == "thumb" and e["payload"].get("value") == "down" for e in events):
        return "bad"
    if "regenerate" in types and "accept" not in types:
        return "bad"
    if max_edit > 0.6:
        return "bad"
    if "accept" in types or "copy" in types or (edits and max_edit < 0.15):
        return "good"
    return "unknown"  # do not guess; most traffic lands here

Note the explicit "unknown" bucket. Forcing every response into good or bad adds noise. Keep unknowns out of your eval set.

From labels to an eval set

The payoff is a pipeline like this:

1. Weekly, query responses labelled bad with high confidence (multiple agreeing signals).
2. Review a sample by hand. Tag the failure category (hallucination, tone, missing context, format).
3. Promote reviewed cases into a regression eval set, with the user's final edited text as a candidate reference answer.
4. Run that set against every prompt or model change.

Using the user's edited version as a reference is powerful but not free of bias: users edit for their own style. Treat it as a candidate reference and have a reviewer confirm it before it becomes ground truth. This pairs naturally with failure mining from production traces and with calibrating an LLM judge against human labels.

Segment before you aggregate

A blended "satisfaction" number hides the story. Always slice by:

• prompt or feature version (did the last release help?)
• user cohort (new vs power users behave very differently)
• input type or intent
• model and retrieval configuration

A feature can look flat overall while improving for one cohort and regressing for another.

Common pitfalls

Pitfall	Why it hurts	Fix
Logging events without a response_id	You cannot reconstruct what the model saw	Generate the ID server-side at generation time
Treating "no feedback" as "good"	Silence is mostly unknown, not approval	Keep an unknown label and exclude it from metrics
Optimising directly for thumbs-up rate	Models learn to please, not to be correct	Pair with task-success signals like edits and escalations
Storing user text carelessly	Privacy and compliance risk	Redact PII, get consent, set retention limits
Feedback loops reinforcing bias	Training on accepted outputs amplifies what already exists	Keep a held-out, hand-labelled set that never feeds back
Ignoring latency	Slow responses cause abandonment unrelated to quality	Log latency in the same trace and control for it

A checklist you can use this week

• Every generation has a server-side response_id tied to prompt version and model
• At least two implicit signals are logged (edit and accept/reject are the best start)
• Thumbs-down opens a one-click reason picker
• A derived good / bad / unknown label exists with documented rules
• A weekly review pulls high-confidence bad cases into a regression set
• PII handling and retention are agreed with whoever owns privacy
• A hand-labelled holdout set is kept separate from anything the feedback loop touches

Next steps

Start small: add the events table, log edits and accept/reject for one feature, and read fifty real examples by hand before building any dashboard. The reading will teach you which signals your users actually produce, which no generic guide can. Once you trust the labels, wire them into your CI evals so every prompt change is tested against real failures instead of imagined ones.

If you want to build this kind of feedback-to-eval loop with guidance and peer review, take a look at the Joinloop engineering cohort, or the PM cohort if you own the product side of AI features.