The Tokenizer Tax: Why Hindi and Tamil Prompts Cost More, and How to Measure It Before You Ship
The same sentence can cost several times more tokens in Hindi, Tamil or Bengali than in English. Here is how to measure that gap in your own stack, and the design moves that keep multilingual AI features affordable.
You build a customer-support assistant in English. Unit economics look fine: a typical ticket is about 400 tokens in, 200 out. Then sales asks for Hindi, Tamil and Bengali support. You assume the cost per ticket stays roughly the same, because the meaning is the same.
It isn't. LLMs bill by tokens, not meaning, and tokenizers are trained mostly on English-heavy text. Scripts like Devanagari (Hindi), Tamil and Bengali get split into many more pieces per word. Researchers call this the tokenizer tax (or "token premium"), and for Indian-language products it affects three things at once: cost, latency and how much fits in the context window.
This post shows how to measure the tax for your own content, what drives it, and what you can do about it.
Why the tax exists
Most modern tokenizers use byte-pair encoding (BPE) or a variant. BPE learns frequent byte sequences from its training corpus and merges them into single tokens. If a script is rare in the corpus, its common words never earn their own tokens. The tokenizer falls back to smaller pieces: syllables, individual characters, vowel signs, or in the worst case raw UTF-8 bytes. Devanagari and Tamil characters take 3 bytes each in UTF-8, so byte-level fallback is expensive.
Two terms worth knowing:
- Fertility: average tokens produced per word. English is typically a bit above 1; heavily fragmented scripts can be several times higher.
- Token premium: fertility (or tokens for equivalent content) relative to English for the same meaning.
Published preprints on Indic tokenization consistently report that newer tokenizers (for example OpenAI's o200k_base) are much better on Indic text than older ones such as the GPT-4-era cl100k_base, but still fragment these languages more than English. Treat any single number you read as specific to that tokenizer, that language and that corpus. Most of these are unreviewed arXiv papers. Measure on your own text.
What the tax actually costs you
| Impact | Mechanism | Who feels it |
|---|---|---|
| Cost | More tokens in and out per request | Finance, PM owning margins |
| Latency | Output is generated token by token; more tokens means longer time-to-last-token | End users |
| Context budget | Fewer documents or turns fit in the window | RAG and agent quality |
| Rate limits | Tokens-per-minute limits are hit sooner | Platform and on-call |
| Quality | Heavily fragmented text can be harder for the model to represent well | Everyone (verify with evals) |
Note the output side: output tokens are usually priced higher than input tokens, and a Tamil answer that needs 3x the tokens costs 3x the (more expensive) output rate.
Step 1: Build a parallel test set
You need the same content in each language. Don't use random text; use your real traffic categories.
- Take 50 to 100 representative messages or documents in English: support tickets, product descriptions, FAQ answers, system-prompt snippets.
- Get them professionally translated (or translated by an LLM and reviewed by a native speaker). Keep them as parallel pairs.
- Include Hinglish (Hindi in Latin script) if your users type that way. Romanised text often tokenizes very differently from native script.
Step 2: Measure token counts
For OpenAI-family tokenizers, tiktoken runs locally and is free:
import tiktoken
from statistics import mean
PAIRS = [
{"en": "Where is my order? It was supposed to arrive yesterday.",
"hi": "मेरा ऑर्डर कहाँ है? यह कल तक आ जाना चाहिए था।",
"ta": "என் ஆர்டர் எங்கே? அது நேற்று வந்திருக்க வேண்டும்."},
# ... add 50-100 real pairs
]
def premium(encoding_name: str, lang: str) -> float:
enc = tiktoken.get_encoding(encoding_name)
ratios = []
for p in PAIRS:
en_tokens = len(enc.encode(p["en"]))
xx_tokens = len(enc.encode(p[lang]))
ratios.append(xx_tokens / en_tokens)
return mean(ratios)
for enc_name in ["cl100k_base", "o200k_base"]:
for lang in ["hi", "ta"]:
print(f"{enc_name:12} {lang}: {premium(enc_name, lang):.2f}x English")
For Claude, tokenization is not available as a local library, but the API exposes a token-counting endpoint, so you can measure without generating anything:
import anthropic
client = anthropic.Anthropic()
def claude_tokens(text: str, model: str = "claude-sonnet-5-5") -> int:
resp = client.messages.count_tokens(
model=model,
messages=[{"role": "user", "content": text}],
)
return resp.input_tokens
Token counts include a small fixed overhead for message framing, so compare ratios over realistic-length texts rather than single short sentences. Check the current docs for the token-counting endpoint's limits and model names.
For open-weight models, load the tokenizer with Hugging Face transformers (AutoTokenizer.from_pretrained(...)) and call len(tok.encode(text)). Run the same script across every model you are considering; the spread between models is often larger than the spread between languages.
Step 3: Turn the ratio into a unit-economics number
A ratio by itself isn't actionable. Convert it:
def cost_per_ticket(in_tokens, out_tokens, in_price_per_m, out_price_per_m, premium=1.0):
return (in_tokens * premium * in_price_per_m
+ out_tokens * premium * out_price_per_m) / 1_000_000
Plug in your current provider prices (check the pricing page; they change) and the measured premium for each language. Then multiply by expected ticket volume per language. This is the number a PM needs before committing to a multilingual launch with a flat per-seat price.
One subtlety: the premium applies to everything that is in that language. If your system prompt and few-shot examples stay in English while only the user message and answer are in Hindi, the effective premium on the total request is lower. Measure the full request, not just the user text.
Step 4: Reduce the tax
| Lever | How it helps | Tradeoff |
|---|---|---|
| Pick the tokenizer deliberately | Newer tokenizers with larger vocabularies and better multilingual coverage can cut counts substantially on Indic text | Model choice also affects quality and price; compare on your evals, not on tokens alone |
| Keep instructions in English | System prompts, schemas and tool descriptions are static and often large; English is the cheapest place for them | Test that the model still answers in the user's language reliably |
| Prompt caching | Static prefix is billed at a discount on repeat calls, which blunts a large fixed premium | Only helps stable prefixes; see our prompt caching guide |
| Response length caps and concise style | Output is the expensive side; instruct brevity per language | Over-tight caps truncate useful answers |
| Route by language | Send high-premium languages to the model with the best cost per meaningful unit | More routing logic; see model routing |
| Translate-then-process | Translate to English, run RAG or reasoning, translate back | Two extra calls, translation errors, loss of nuance and named entities |
| Smaller or distilled model for the language | Cheaper per token | Quality drop on low-resource languages; evaluate carefully |
Translate-then-process looks attractive on paper because English is cheap, but it adds latency and an extra failure mode. Evaluate it against direct handling rather than assuming it wins.
Step 5: Don't stop at cost; evaluate quality per language
A cheaper tokenizer does not guarantee a better answer. Build a per-language slice in your eval set:
- Same task, same rubric, scored separately for English, Hindi, Tamil, Bengali and Hinglish.
- Track token counts alongside the quality score, so you can plot quality per rupee.
- Include script-specific checks: correct numerals, no mixed-script garbage, names transliterated consistently.
If you already run an LLM-as-judge, make sure the judge itself is competent in the language; otherwise you are measuring the judge's weakness. Our calibration guide covers how to check that.
Common pitfalls
- Using character count or word count as a proxy. Characters and words say nothing about tokens across scripts. Always count tokens with the actual model's tokenizer.
- Budgeting from English traffic. Per-language cost and context limits differ; a
max_tokensthat is plenty in English can truncate a Tamil answer mid-sentence. - Comparing tokenizers on one sentence. Short strings are dominated by overhead. Use 50+ realistic samples.
- Ignoring chunk sizes in RAG. If you chunk by token count, a 512-token chunk of Hindi holds far less text than one of English. Retrieval quality and context packing both change. Revisit chunk size per language, as discussed in our chunking guide.
- Assuming the premium is stable. Providers ship new tokenizers with new model generations. Re-run your measurement script whenever you change models; put it in CI.
- Forgetting Hinglish. Romanised Hindi may tokenize more efficiently than Devanagari on some tokenizers but can hurt answer quality. Measure both.
A checklist before launching in a new language
- Parallel test set of 50+ real samples exists
- Token premium measured on every candidate model
- Full-request cost per interaction computed (system prompt, context, output)
-
max_tokens, chunk sizes and context budgets set per language - Prompt caching enabled for static prefixes
- Per-language quality eval with a competent judge or native-speaker review
- Alerting on tokens per request by language, so drift is visible
Next steps
Run the script above on 50 of your own messages today. It takes under an hour and the output is often a surprise. Then put the premium into your pricing model before sales commits to a multilingual plan.
If you want to build this kind of measurement-first habit into how your team ships AI features, our AI Engineering cohort works through evals, cost modelling and production tradeoffs hands-on, and the AI for PMs cohort covers turning numbers like these into pricing and launch decisions.
Want to build this yourself?
Join a live cohort and go from reading about AI to shipping it, with practitioners who do it daily.