Reasoning Models in Production: When Thinking Tokens Are Worth Paying For

Most teams adopt a reasoning model the same way: swap the model name, watch the eval score jump a few points, ship it, and discover a month later that the bill tripled and p95 latency went from 2 seconds to 14. The model is not the problem. Using extended thinking on every request is.

Reasoning (or "thinking") models spend extra output tokens working through a problem before producing the final answer. Those tokens are billed as output tokens, they add wall-clock time, and on many tasks they add nothing. This post is about deciding, per request, whether thinking is worth it, and building the plumbing to enforce that decision.

What you are actually paying for

With a standard model, cost is roughly input_tokens × input_price + answer_tokens × output_price. With a reasoning model, add a third term: the hidden reasoning tokens, which are billed at the output rate and often dwarf the visible answer.

A support-ticket classifier might return 20 visible tokens. With thinking enabled it may burn 1,500 reasoning tokens first. Same answer, ~75x the output tokens.

Three costs follow:

1. Money: reasoning tokens are billed as output tokens.
2. Latency: tokens are generated sequentially, so 1,500 extra tokens is seconds, not milliseconds. Time-to-first-visible-token suffers most.
3. Variance: thinking length is not fixed. Your p99 gets much worse than your median, which breaks timeouts and retry budgets you tuned for a non-reasoning model.

Where thinking helps, and where it does not

Reasoning helps when the task needs multi-step inference the model cannot do "in one forward pass per token":

• Multi-constraint planning (scheduling, routing, itinerary building)
• Math, logic, and code with non-obvious bugs
• Agent steps that choose among many tools with interacting preconditions
• Ambiguous documents where the answer requires cross-referencing several passages

It helps little or not at all for:

• Classification, extraction, and formatting
• Simple retrieval-grounded Q&A where the answer sits in one passage
• Rewriting, summarising, tone changes
• Anything where the failure mode is missing knowledge rather than missing reasoning

A useful rule: if a competent human could answer in a few seconds without scratch paper, thinking is probably wasted.

Step 1: Measure the lift before you commit

Do not decide from vibes. Run your eval set twice, once with thinking off (or minimal) and once with a generous budget, and compare accuracy, cost, and latency per task category.

import time
from dataclasses import dataclass

@dataclass
class RunResult:
    correct: bool
    latency_s: float
    input_tokens: int
    output_tokens: int  # includes reasoning tokens on most providers

def run_eval(cases, call_model, grade, thinking_budget: int | None):
    results = []
    for case in cases:
        t0 = time.perf_counter()
        resp = call_model(case["prompt"], thinking_budget=thinking_budget)
        latency = time.perf_counter() - t0
        results.append(RunResult(
            correct=grade(case, resp["text"]),
            latency_s=latency,
            input_tokens=resp["usage"]["input_tokens"],
            output_tokens=resp["usage"]["output_tokens"],
        ))
    return results

def summarise(results, in_price, out_price):
    n = len(results)
    acc = sum(r.correct for r in results) / n
    cost = sum(r.input_tokens * in_price + r.output_tokens * out_price
               for r in results) / 1_000_000
    lat = sorted(r.latency_s for r in results)
    return {
        "accuracy": round(acc, 3),
        "cost_per_100": round(cost / n * 100, 4),
        "p50_s": round(lat[n // 2], 2),
        "p95_s": round(lat[int(n * 0.95)], 2),
    }

# Compare: no thinking vs 2k vs 8k budget, per task category
for budget in (None, 2_000, 8_000):
    r = run_eval(eval_cases, call_model, grade, budget)
    print(budget, summarise(r, in_price=3.0, out_price=15.0))

The prices above are placeholders; plug in your provider's current numbers. What you want out of this is a table like the one below, per task type, so the routing decision becomes arithmetic.

Task category	No thinking	2k budget	8k budget	Verdict
Ticket classification	94%	94%	94%	Off
Contract clause extraction	88%	91%	91%	Low budget
Multi-step refund policy decisions	61%	78%	84%	Medium-high
Agent tool-selection with 30+ tools	70%	82%	83%	Low-medium

(Illustrative numbers. Yours will differ, which is exactly why you measure.)

The pattern you will usually see is steeply diminishing returns: most of the gain arrives in the first couple of thousand thinking tokens.

Step 2: Route by task, not by model

Once you have per-category data, route requests. The cheapest reliable router is a static map keyed on the task type your application already knows. You do not need an LLM classifier to decide whether to think.

from enum import Enum

class Effort(Enum):
    OFF = 0
    LOW = 1_500
    MEDIUM = 4_000
    HIGH = 12_000

TASK_EFFORT = {
    "classify_ticket": Effort.OFF,
    "extract_clauses": Effort.LOW,
    "refund_decision": Effort.HIGH,
    "pick_tool": Effort.LOW,
}

def effort_for(task: str, retry_count: int = 0) -> Effort:
    base = TASK_EFFORT.get(task, Effort.LOW)
    # Escalate on retry: a failed validation is a signal that
    # the cheap path was not enough.
    if retry_count >= 1 and base is not Effort.HIGH:
        return Effort(min(base.value * 3, Effort.HIGH.value)) \
            if base is not Effort.OFF else Effort.LOW
    return base

The escalation-on-retry pattern is underused. Run cheap first, validate the output (schema check, test run, citation check), and only pay for thinking when validation fails. For tasks where 80% succeed cheaply, this cuts average cost sharply while keeping the hard-case accuracy.

Step 3: Set budgets and handle the provider differences

Providers expose thinking control differently: some take an explicit token budget, some take a coarse effort level (low/medium/high), and some let the model decide adaptively. Parameter names, minimums, and which models support them change between releases, so check your provider's current docs rather than copying a snippet from a blog post (including this one).

Things that are consistent across providers:

• Budget is a ceiling, not a target. The model may use far less on easy inputs.
• Max output tokens must exceed the thinking budget, or the visible answer gets truncated. Set max_tokens = thinking_budget + answer_headroom.
• Reasoning content may be summarised or hidden. Do not build features that depend on parsing it, and do not show it to end users as if it were a faithful explanation.
• Multi-turn and tool-use flows may require you to pass reasoning blocks back unchanged. Dropping them can degrade quality or raise errors. Read the tool-use section of your provider's docs.

Step 4: Fix the latency story

Thinking breaks UX assumptions. Options, in order of effort:

1. Show progress. Stream status ("Analysing policy...") while thinking runs. Users tolerate 10 seconds with feedback far better than 5 seconds of a frozen spinner.
2. Split fast and slow paths. Return a fast-model draft immediately and upgrade it when the reasoning model finishes. Works for assistive UIs, not for decisions that must be right the first time.
3. Go async. For back-office tasks (claims, reconciliation, report generation), queue the work. Latency stops mattering and you can use batch pricing where available.
4. Raise timeouts deliberately. A 30-second client timeout tuned for a non-reasoning model will fail a meaningful fraction of high-effort calls and trigger retries that double your spend.

Common pitfalls

Pitfall	Why it hurts	Fix
Thinking on for every call	Triples cost with zero gain on simple tasks	Per-task effort map
Judging only on accuracy	Hides 5-10x cost and latency regressions	Report accuracy, cost, p95 together
Budget higher than max_tokens	Answer truncated or empty	max_tokens = budget + headroom
Over-prompting reasoning models	"Think step by step" is redundant, and long scaffolding can hurt	State the goal and constraints, drop the CoT instructions
Treating reasoning text as an audit log	It may be a summary, not a faithful trace	Log inputs, tool calls, and outputs for audit
Same timeout and retry budget	Long-tail calls time out and retry	Separate timeout class for high-effort calls
Never re-testing	Model updates shift where thinking pays off	Re-run the budget sweep on each model change

Prompting differences worth knowing

Reasoning models usually do better with less hand-holding. Give the objective, the constraints, the output format, and the relevant context. Skip the worked chain-of-thought exemplars you wrote for older models; they can constrain the model to your reasoning style, which is often worse than its own. When in doubt, A/B the old and new prompts on your eval set. That is a 20-minute experiment and it frequently removes half the prompt.

A checklist you can apply this week

• Tag every LLM call in your app with a task type
• Run a no-think vs. low vs. high sweep per task type on your eval set
• Record accuracy, cost per 100 calls, p50 and p95 latency
• Set thinking to off for any task where the lift is under your noise floor
• Add validate-then-escalate for tasks with cheap automatic checks
• Ensure max_tokens covers budget plus answer
• Give high-effort calls their own timeout and retry policy
• Add a dashboard panel: reasoning tokens as a share of output tokens, by task

Next steps

Start with the dashboard panel. Once you can see which task types burn the most reasoning tokens, the sweep tells you which ones deserve it. Most teams find that 10-20% of their traffic justifies thinking and the rest was a tax.

If you want to build these routing, eval, and cost-control patterns hands-on with feedback from practitioners, take a look at the engineering cohort. If you are a PM who needs to make these tradeoffs call for AI features, the PM cohort covers cost, latency, and quality budgeting end to end.