The AI Feature Fitment Scorecard: Decide Before You Build
Most AI features fail because the problem was a bad fit for a probabilistic system, not because the model was weak. Here's a scorecard PMs can run in an afternoon to catch that early.
Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027. An MIT review of enterprise pilots put the failure rate of generative AI proofs-of-concept at 95%. When teams do the post-mortems, the cause is rarely "the model wasn't smart enough." It's that the feature was scoped against a problem where a probabilistic, hard-to-evaluate system was the wrong tool — and nobody checked that before the roadmap locked.
This post gives you a concrete fitment scorecard: seven dimensions, a 0–2 score on each, and a threshold for go / rework / kill. You can run it on a feature idea in an afternoon, before writing a PRD, before pulling in eng.
Why "can an LLM do this?" is the wrong question
Given a good demo, an LLM can appear to do almost anything. The demo is not the product. The product is the same task run 50,000 times a day against inputs you didn't anticipate, evaluated by users who will notice the 3% of outputs that are wrong, wired into workflows that assume determinism.
The right question is: when this feature is wrong — and it will be wrong sometimes — what happens, how will we know, and can we make it better over time? Fitment is about the shape of the problem, not the capability of the model.
Three properties make a problem a good fit for an LLM feature:
- The cost of a wrong answer is bounded and recoverable. A user can notice, correct, or ignore it without lasting damage.
- "Good" is checkable. Either a human can glance at the output and know if it's right, or you can write a programmatic check, or you have ground-truth data.
- Volume or variability defeats rules. If a team of analysts with a spreadsheet could handle it, or a regex could, you don't need an LLM — and you'll regret the latency, cost, and eval burden.
The scorecard operationalises these.
The scorecard
Score each dimension 0, 1, or 2. Definitions below.
| # | Dimension | 0 (bad fit) | 1 (marginal) | 2 (good fit) |
|---|---|---|---|---|
| 1 | Error tolerance | Wrong output causes financial loss, safety risk, or legal exposure | Wrong output causes rework or mild user frustration | Wrong output is trivially ignored or corrected |
| 2 | Verifiability | No way to tell if an output is correct without deep investigation | A domain expert can judge correctness in minutes | Correctness is obvious at a glance, or programmatically checkable |
| 3 | Ground truth availability | No labelled examples, no way to generate them | Could label 50–200 examples with effort | Hundreds of real input/output pairs already exist in logs or docs |
| 4 | Human in the loop | Output is auto-executed with no review point | Human reviews high-risk cases only | Every output is reviewed or edited before it matters |
| 5 | Problem variability | A rules engine or lookup table handles 95%+ of cases | Rules handle the common path; long tail is messy | Inputs are open-ended natural language / documents; rules break down |
| 6 | Latency budget | Needs sub-100ms, synchronous, in a tight loop | 1–5s acceptable in an interactive flow | Seconds-to-minutes fine; async or background work |
| 7 | Improvement loop | No feedback signal; can't tell good from bad in production | Implicit signals (edits, thumbs, re-runs) available | Explicit outcome labels flow back automatically |
Total out of 14:
- 11–14 — Strong fit. Proceed to PRD. Your risk is execution, not premise.
- 7–10 — Conditional. There's a real fit but one or two dimensions are load-bearing. Fix them in scope (add a review step, build a golden set, narrow the input space) or descope until they're 2s.
- 0–6 — Kill or radically reframe. You are likely building a demo that won't survive contact with production traffic. Look for a narrower slice of the problem that scores higher.
Any single 0 on dimensions 1, 2, or 3 is a veto regardless of total. You cannot ship what you cannot evaluate, and you cannot afford to be wrong where being wrong is catastrophic.
Worked examples
Example A: "Summarise this support ticket thread for the agent picking it up"
| Dimension | Score | Reasoning |
|---|---|---|
| Error tolerance | 2 | Agent still reads the thread; summary is a head start |
| Verifiability | 2 | Agent immediately sees if the summary misrepresents the thread |
| Ground truth | 2 | Thousands of resolved threads with resolution notes |
| Human in loop | 2 | Agent is the human, always present |
| Variability | 2 | Free-text threads, every one different |
| Latency | 2 | 2–3s while the ticket opens is fine |
| Improvement loop | 1 | Can track whether agents expand the thread anyway; no explicit rating yet |
| Total | 13 | Strong fit — build it |
Example B: "Auto-approve expense reports under $500 using an LLM policy check"
| Dimension | Score | Reasoning |
|---|---|---|
| Error tolerance | 0 | A wrong approval is money out the door and an audit finding |
| Verifiability | 1 | Finance can audit, but not in real time |
| Ground truth | 2 | Years of approved/rejected reports |
| Human in loop | 0 | The whole point is to remove the human |
| Variability | 1 | Most reports are routine; policy edge cases are the hard part |
| Latency | 2 | Async, no problem |
| Improvement loop | 2 | Reversed approvals are a clean label |
| Total | 8, but two vetoes (dims 1 and 4) | Kill as scoped. Reframe: LLM flags likely-non-compliant reports for a reviewer, humans still approve. That version scores 12. |
The reframe is the actual output of the exercise. The scorecard didn't say "no AI" — it said "not this shape of AI."
Turning a marginal score into a strong one
Each dimension has a standard move:
| Weak dimension | The move |
|---|---|
| Error tolerance | Add a confirmation step, a preview, an undo, or a confidence threshold that routes low-confidence cases to humans |
| Verifiability | Constrain the output format (structured JSON, citations, extractive-only) so checking is mechanical |
| Ground truth | Spend a week building a 100–200 example golden set before committing the roadmap slot |
| Human in loop | Ship as "draft" or "suggestion" mode first; earn autonomy with measured accuracy |
| Variability | Narrow the input space — one document type, one language, one customer segment |
| Latency | Move the work off the critical path; precompute; stream partial results |
| Improvement loop | Instrument edits, rejections, and downstream outcomes from day one, even before you use them |
If you can't afford the move, that's your answer: the feature isn't ready for this quarter.
Common pitfalls
- Scoring the demo, not the deployment. Run the scorecard against peak production volume and the messiest 10% of inputs, not the three examples in the pitch deck.
- Treating "we'll add evals later" as a plan. Dimension 3 is a veto for a reason. No golden set means no way to know if a prompt change made things worse. Build it first.
- Confusing "a human could review" with "a human will review." If the UX makes review optional and slow, users will skip it. Score dimension 4 based on the workflow you'll actually ship.
- Ignoring the improvement loop until launch. If you don't instrument feedback signals on day one, you'll spend month three trying to reconstruct them from incomplete logs.
- Letting a high total mask a veto. A 12/14 with a 0 on error tolerance is not a 12. It's a no.
- Running it once. Re-score after the first eval results come in. Real accuracy numbers change error-tolerance and human-in-loop calculus.
Next steps
- Take your top three roadmap AI features and score each one this week. Timebox to 30 minutes per feature.
- For anything scoring 7–10, write the one-paragraph "reframe or descope" plan before it gets a PRD.
- For anything with a veto, either kill it publicly (so the team stops assuming it's coming) or find the narrower slice.
- Bake dimension 3 into your definition of ready: no AI feature enters a sprint without a golden set of at least 50 examples.
- Add the improvement-loop instrumentation to the first ticket, not the last.
The scorecard won't tell you how to build the feature. It will stop you from spending a quarter building the wrong one — which, given the base rates, is the more valuable outcome.
Joinloop runs a hands-on cohort for product managers building AI features — evals, fitment, PRDs for probabilistic systems, and shipping past the demo. See the PM cohort →
Sources: LogRocket — the LLM context problem in 2026, ideaplan — AI product-market fit framework, ideaplan — product requirements for AI features, orq.ai — LLM product development in 2026
Want to build this yourself?
Join a live cohort and go from reading about AI to shipping it, with practitioners who do it daily.