Tokenization Efficiency

Choose a model or tokenizer that represents the same workload in fewer billable tokens, provided it delivers comparable task quality. Fewer input tokens lower token-priced API cost and use less of a fixed context window; fewer output tokens also lower cost and usually reduce generation work.

This is a property of the tokenizer plus corpus, not a universal model ranking. An agent implements tokenization efficiency when it measures representative prompts with each candidate model's own token counter and includes the resulting token count in routing or procurement decisions.

How It Works

For each candidate, tokenize an identical, representative corpus using the exact model, message shape, tools, and system prompt intended for production. Compare:

Token count — the directly billable quantity for the text under that model.
Characters per token (CPT) — characters / tokens; higher is more compact when character content is identical.
Fertility — tokens / word (or another declared linguistic unit); lower is more compact.
Effective cost — (input tokens × input price) + (output tokens × output price), with cache prices and expected output length included where relevant.

Route to the lowest-cost candidate that clears the same quality threshold. Do not infer cost from a generic “four characters per token” rule: the tokenizer, language, code/data format, and API framing all change the result. Providers expose model-specific counting endpoints for this purpose; Anthropic explicitly advises recounting after tokenizer changes, and Google exposes count_tokens before a request.

Leaderboard: Ukrainian Brown Corpus

The table below is a reproducible published snapshot rather than a universal leaderboard. It ranks the Ukrainian portion of the Brown corpus by fertility (tokens per word; lower is better) from Table 4 of Maksymenko and Turuta (2025). It compares tokenizers, not models at comparable quality or API prices; therefore it must not be read as a recommendation to select the first row.

Rank	Tokenizer / model family in study	Fertility ↓	Tokens vs. GPT-2 ↓
1	Ukrainian GPT-2	1.30	79% fewer
2	Llama 3.1	1.88	70% fewer
3	Gemma / Gemma 2 / Gemini	1.92	70% fewer
4	GPT-4o / 4o-mini	1.98	69% fewer
5	Mistral Nemo	2.04	68% fewer
6	Llama 2	2.29	64% fewer
7	Phi 3.5	2.29	64% fewer
8	Mistral 7B / Large / Mixtral	2.48	61% fewer
9	Qwen 1.5 / 2 VL	2.83	55% fewer
10	GPT-3.5 / GPT-4	3.32	47% fewer
11	Mamba	3.35	47% fewer
12	Claude 3 (approximated tokenizer)	3.45	45% fewer
13	GPT-2 / Phi 2	6.31	baseline

The source labels several rows as shared tokenizer families, and Claude 3 is explicitly an approximation, so the figures should not be used for current closed-model billing. The study reports different results for English and specialized corpora. A separate 2026 multilingual evaluation likewise finds that efficiency differs greatly by language and that intrinsic tokenizer metrics alone do not establish English downstream quality.

Decision Procedure

Sample real production traffic by language, modality, and format (prose, source code, JSON, tool schemas, and retrieved documents).
Count the exact request with every candidate's official counter, including wrappers that are billable.
Run the same quality evaluation and discard candidates below the minimum quality, safety, latency, or capability requirement.
Compute expected cost per successful task, then route workload segments separately when a tokenizer advantage is segment-specific.
Re-run the measurement after a provider changes models, tokenizers, pricing, or prompt structure.

Trade-offs

Lower token count is only cheaper when per-token prices and cache treatment do not erase the advantage.
A tokenizer that is compact for one language or source-code corpus can be inefficient for another.
Token compactness is not a quality metric. It can correlate with multilingual performance, but it does not prove comparable quality.
Provider counters can include API-added tokens or return estimates; compare billed usage after deployment as well.

Sources

Anthropic token counting documentation — documents model-specific counting and says its newer tokenizer produces roughly 30% more tokens for the same text than earlier models.
Google Gemini token counting documentation — documents the count_tokens API for measuring a request before it is sent.
Maksymenko and Turuta (2025), Tokenization efficiency of current foundational large language models for the Ukrainian language — source for the leaderboard and its limitations.
Chiu (2026), TokLens: A Multilingual Lens on Tokenizer Quality for LLMs — evaluates 24 tokenizers across 15 languages and cautions against treating intrinsic efficiency as a standalone quality measure.
The Register: Anthropic's extravagant tokenizer complicates AI pricing — motivating reporting on how a tokenizer change affects apparent token-priced cost.

Tools

OpenAI tiktoken Encoding.encode — count and compare text under OpenAI encodings.
Hugging Face Tokenizers BaseTokenizer.encode — run equivalent measurements for open tokenizer files.