Token Budgeting for LLM Apps: Count, Cap, and Truncate Without Breaking Answers

Most LLM bugs that look like "the model got dumber" are really "the prompt got bigger." A retrieval step returns twelve chunks instead of four. A user pastes a 40-page PDF. A tool result dumps a full JSON payload. The request either fails with a context-length error, or it succeeds and the model quietly ignores the system prompt because it was pushed into the middle of a long window.

Token budgeting is the discipline of deciding, before every call, how many tokens each part of the prompt is allowed to use. It is not a fancy algorithm. It is three steps: count what you have, cap each section against a budget, and truncate the sections that can afford it, in a deliberate order.

Why naive truncation breaks things

The default in many codebases is prompt[:N] or "drop the oldest messages." Both are wrong in predictable ways:

Character limits are not token limits. A 4,000-character string can be 900 tokens of English or 2,500 tokens of minified JSON or code.
Truncating from the end removes the question. If your user message sits after a long document, cutting the tail deletes the instruction the model needs.
Dropping old messages drops constraints. The system prompt and the user's first stated requirement are often the most important tokens in the window.
Silent truncation hides failures. Nothing errors. The answer just gets worse, and nobody logs why.

The fix is to treat the prompt as a set of named sections with explicit priorities and budgets.

Step 1: Count tokens accurately

You need a counter that matches the model you are calling. Three options, in order of accuracy:

Method	Accuracy	Cost	Use when
Provider count endpoint (e.g., Anthropic's `messages.count_tokens`)	Exact for that model	One extra network call	Final gate before expensive or long requests
Provider tokenizer library (e.g., `tiktoken` for OpenAI encodings)	Exact for that encoding	Local, free	Hot path, OpenAI-family models
Heuristic (`len(text) / 4`)	Rough, can be off by 30%+ on code and JSON	Free	Quick estimates, logging, UI hints only

A useful rule: use the heuristic for UI hints, a local tokenizer for the hot path, and the provider count endpoint in tests or as a guardrail when a request crosses a threshold. Check the official docs for the current tokenizer and endpoint names, because they change between model generations.

# pip install tiktoken
import tiktoken

_enc = tiktoken.get_encoding("o200k_base")  # check which encoding your model uses

def count_tokens(text: str) -> int:
    return len(_enc.encode(text, disallowed_special=()))

Count every section separately, not the whole prompt at once. You need per-section numbers to make budget decisions.

Step 2: Give each section a budget

Split the prompt into sections, each with a priority and a maximum:

system: must always fit. Priority 0. Never truncated; if it does not fit, that is a code bug.
user_query: must always fit. Priority 0. The question the model is answering.
history: recent turns matter most. Priority 1. Truncate oldest first.
retrieved_context: useful but lossy. Priority 2. Truncate lowest-ranked chunks first.
tool_outputs: often the biggest offender. Priority 2. Truncate with a summary or a pointer.

Reserve room for the output too. If the model's window is W, your input budget is W - max_output_tokens - safety_margin. A 5% margin covers tokenizer drift and per-message overhead.

Step 3: A runnable budgeter

This budgeter takes ordered sections, enforces per-section caps, and then trims lower-priority sections until the total fits. It never touches priority-0 sections, and it reports what it dropped so you can log it.

from dataclasses import dataclass, field

@dataclass
class Section:
    name: str
    text: str
    priority: int          # 0 = never truncate, higher = trim first
    cap: int | None = None # max tokens for this section
    items: list[str] = field(default_factory=list)  # optional: ordered chunks (best first)

def fit_text(text: str, max_tokens: int) -> str:
    """Hard truncate by token count, keeping the head."""
    ids = _enc.encode(text, disallowed_special=())
    if len(ids) <= max_tokens:
        return text
    return _enc.decode(ids[:max_tokens])

def build_prompt(sections: list[Section], window: int, max_output: int,
                 margin: float = 0.05) -> tuple[str, dict]:
    budget = int((window - max_output) * (1 - margin))
    report = {"dropped": [], "truncated": []}

    # 1. Apply per-section caps first.
    for s in sections:
        if s.cap is not None and count_tokens(s.text) > s.cap:
            s.text = fit_text(s.text, s.cap)
            report["truncated"].append(s.name)

    # 2. Trim until the total fits, lowest priority first.
    def total() -> int:
        return sum(count_tokens(s.text) for s in sections)

    while total() > budget:
        candidates = [s for s in sections if s.priority > 0 and s.text]
        if not candidates:
            raise ValueError(
                f"Non-negotiable sections use {total()} tokens; budget is {budget}."
            )
        victim = max(candidates, key=lambda s: s.priority)
        over = total() - budget
        if victim.items:
            victim.items.pop()                       # drop the least relevant chunk
            victim.text = "\n\n".join(victim.items)
            report["dropped"].append(f"{victim.name}:chunk")
        else:
            keep = max(0, count_tokens(victim.text) - over)
            victim.text = fit_text(victim.text, keep)
            report["truncated"].append(victim.name)

    prompt = "\n\n".join(f"## {s.name}\n{s.text}" for s in sections if s.text)
    return prompt, report

A few details matter here:

Chunks are dropped whole, not mid-sentence. Retrieved context should be a list of items ranked best-first, so trimming means removing the weakest item, not cutting a chunk in half.
The report is the point. Log report with the request ID. When answer quality drops, you can correlate it with truncation instead of guessing.
Order of sections in the output is independent of priority. Put the instructions and the user query where the model attends best for your model family, and test that choice rather than assuming.

Usage:

sections = [
    Section("system", SYSTEM_PROMPT, priority=0),
    Section("history", history_text, priority=1, cap=2_000),
    Section("retrieved_context", "", priority=2, cap=6_000,
            items=ranked_chunks),
    Section("user_query", user_message, priority=0),
]
prompt, report = build_prompt(sections, window=128_000, max_output=4_000)
if report["dropped"] or report["truncated"]:
    log.info("prompt_trimmed", extra=report)

Note that window=128_000 is a placeholder. Use the real context window for the model you call, which the provider documents.

Step 4: Put the budget where the cost is

Budgets are not only about avoiding errors. Input tokens are billed on every call, and tool outputs and retrieved context usually dominate the bill. A few places to look:

Cap tool outputs at the source. If a tool returns 200 rows, return the top 20 plus a count and a way to page. The model rarely needs the full table.
Summarize history instead of replaying it. Once a conversation passes a threshold, replace old turns with a short summary. (See the earlier post on context compaction for the trade-offs.)
Measure before you cap. Log per-section token counts for a week. The largest section is almost never the one you expected.

Comparison: what to do when the budget is exceeded

Strategy	What you lose	Good for	Risk
Drop lowest-ranked chunks	Weak retrieval hits	RAG context	Missing the one chunk that mattered
Truncate history oldest-first	Early turns	Chat	Losing user constraints stated early
Summarize old content	Detail, not meaning	Long sessions	Summary drops specifics like IDs and numbers
Hard cut a section by tokens	Whatever is at the end	Logs, dumps	Cutting mid-structure (JSON, code)
Reject the request	Nothing silently	Hard limits, paid tiers	Bad UX if it happens often

For structured content, prefer dropping whole elements over cutting bytes. A truncated JSON array that ends mid-object is worse than a shorter valid array.

Common pitfalls

Counting once, sending later. If you append tool output after counting, the budget is wrong. Count at the moment of assembly.
Forgetting message overhead. Chat formats add tokens per message and per role. Your section counts will undershoot the real total. The margin in Step 2 exists for this, and the provider count endpoint is the final check.
Budgeting only input. Set max_output_tokens deliberately. A large output cap on a small window causes the request to fail after you have already paid for the input.
Truncating the question. Keep the user query at priority 0. A common regression is a refactor that moves it into the same section as retrieved context.
Ignoring the non-English case. Token density varies by language and script. Measure with real traffic, not English samples.

Next steps

Instrument your prompt builder with per-section token counts and log them for one week.
Set priority-0 sections explicitly (system prompt, user query) and make the build fail loudly if they overflow.
Add a cap to your largest section first, usually retrieval or tool output.
Log every truncation with a request ID, and review the logs when answer quality dips.
Add a test that builds a prompt from a fixture that is deliberately too large, and asserts the instructions and user query survive.

Budgeting is one of those practices that looks like plumbing until it fails. Once you have it, prompt size becomes a number you control, not a surprise on the invoice.

If you are building LLM products and want hands-on practice with these patterns, the Joinloop Engineering Cohort works through them on real codebases. Product managers who need to set token and cost guardrails for AI features can start with the Joinloop PM Cohort.