AI models don't charge by the word. They charge by the token — and the same sentence breaks into far more tokens in a Ghanaian language than in English. This isn't an estimate. Every number below was measured with a real tokenizer, and you can see exactly where the cuts fall.
Lesson progress0%
0
First: a token is not a word
Before a model reads anything, it chops the text into pieces called tokens. A token is a sub-word unit — sometimes a whole word, sometimes a fragment, sometimes a single letter. The pieces aren't chosen by a rule you can guess. They're learned from data: the sequences that showed up most often in the training text became single tokens, and everything else gets assembled from smaller scraps.
That last sentence is the whole story of this page. Whichever language dominated the training text gets the cheap, whole-word tokens. Everyone else pays in fragments.
English, run through the real tokenizer — 88% of these tokens are whole words:
a whole worda fragmenta single character◆ half a character — a raw byte, not valid text on its own
Every number on this page is real. The splits shown here come from tiktoken, the actual tokenizer OpenAI ships, using o200k_base — the encoding behind GPT-4o and later. Nothing is approximated or illustrative. That matters, because a lot of "live tokenizer" demos on the web just chop text every four characters, which produces exactly the wrong intuition about where the cuts land.
1
The same sentence, eight languages
To compare fairly you need the same meaning in every language — so this uses Article 1 of the Universal Declaration of Human Rights, which has been officially translated into all of these. Same sentence. Same idea. Wildly different bills.
o200k_base — the current one.
Bars show the token count for the identical sentence. The number on the right is how many times more expensive it is than English.
Read that Ga bar again. The Ga translation is shorter than the English one in actual characters — 159 against 170 — and still costs two and a half times as many tokens. You are not paying for length. You are paying for how well the tokenizer knows your language.
Swahili is the interesting counter-example: it comes out nearly level with English. It has a large web presence, so the tokenizer learned it well. This isn't about African languages being inherently expensive — it's about how much of your language was on the internet when the tokenizer was built.
2
Look at where the cuts actually fall
The bar chart tells you the price. This tells you why. Here is the same sentence, tokenized for real — pick a language and watch what happens to the size of the pieces.
a whole worda fragmenta single character◆ half a character — a raw byte, not valid text on its own
English is read in words. Ga is read in letters. In the English sentence, 88% of the tokens are complete words and only 6% are single characters. In Ga it inverts: 12% whole words, 57% single characters. The model isn't reading Ga so much as spelling it out.
Notice which characters keep appearing alone: ɔ, ɛ, ŋ. These letters are ordinary in Ghanaian orthography and nearly absent from the English-dominated text the tokenizer learned from, so no merge was ever learned for them. Each one costs a token by itself.
And it gets worse than one-token-per-letter. Those ◆ marks in the Ga and Ewe text are tokens that aren't even a whole character — they're raw bytes. In the Ga sentence the letter ŋ is cut clean in half across two tokens, and neither half is valid text on its own. Written by itself, ŋ costs one token, the same as n. Sitting in an ordinary Ga sentence, it costs two.
3
Your name costs more than theirs
This isn't only about long documents. It shows up in the smallest, most personal unit of text there is. Tap a name to see how the tokenizer breaks it apart.
Ghanaian names — including the Akan day-names almost every Ghanaian carries:
Common English and European names, for comparison:
"Kwame Nkrumah" costs 6 tokens. "John Smith" costs 2. Both are two words; one is a founding president. The tokenizer knows John and Smith as single units and has to spell Nkrumah out as N · kr · um · ah.
This is not a rounding error in a billing statement. If you build search, matching, or a chatbot over Ghanaian customer records, every name is two to three times more expensive to process than the equivalent Western name — and fragmented names are also harder for the model to match, misspell-correct, and keep straight.
4
What that actually costs you
Tokens are the unit of the bill and the unit of the limit, so the penalty lands in three places at once. Put in what you'd spend serving English and see the same workload in another language.
—
Currency is whatever you typed — the multiplier is what matters, and it's the measured token ratio. Real bills also differ between input and output tokens; this is the underlying penalty, not a quote.
The third cost is the one people miss. A context window is measured in tokens too. So the same 128,000-token window that holds a long English report holds less than half as much Ga — your documents get truncated sooner, your chatbot forgets earlier, and your retrieval has less room to work with. Same model, less usable memory, purely because of the language.
5
Why it happens — nobody decided this
There's no line of code that charges more for Twi. The penalty is a side-effect of how the vocabulary was built, and it's worth understanding because it tells you what would actually fix it.
Tokenizers are usually built with byte-pair encoding. Start with individual characters. Look across an enormous pile of training text, find the two adjacent pieces that occur together most often, and merge them into one new token. Repeat tens of thousands of times. What you end up with is a vocabulary shaped entirely by what was frequent in that pile.
The pile was mostly English, mostly scraped from the web. So computer, government and Smith earned their own tokens. Sequences that are completely ordinary in Ga — but rare on the English-language internet — never came up often enough to earn a merge. When the model meets them, it falls back to the smallest pieces it has: single characters, sometimes single bytes.
Two compounding effects: first, no merges were learned for common Ghanaian letter sequences. Second, letters like ɔ ɛ ŋ live outside the plain ASCII range, so they take two bytes each in the underlying encoding — and without a learned merge to hold them together, that cost is paid one token at a time.
Which points at the fix: this is a data representation problem, not a hard limit. Swahili proves it — enough text on the web, and the tokenizer learns your language properly.
6
Is it getting better? Yes — and not fast enough
Worth being fair here, because the picture genuinely improved. GPT-4's tokenizer and GPT-4o's tokenizer split these same sentences very differently.
Each row shows the same sentence under the older cl100k_base (GPT-4) and the newer o200k_base (GPT-4o and later).
Real progress, still a real gap. Ga dropped from 3.7× English down to 2.55× — about a third cheaper than it was. Twi improved by a similar margin. That's what a bigger, better-balanced vocabulary buys you.
But English didn't move at all, because it had nothing to gain — and Ga at 2.55× is still Ga at more than double. The gap narrowed; it did not close. And every improvement so far has come from the vocabulary getting larger, which has limits. Closing it properly needs more Ghanaian-language text in the world for a tokenizer to learn from, which is a thing people have to go and build.
7
What to do about it
Knowing the tax exists changes some decisions. Practical, in rough order of how often it matters:
1Measure before you budget
If you're costing out a product that serves Ghanaian users in their own language, do not estimate from word counts. Run your real text through a real tokenizer. A 2.5× surprise is the difference between a viable product and a dead one.
2Budget context windows in tokens, not pages
"It's a 40-page document, it'll fit" is an English intuition. In Ga that document is more than twice the tokens. Check before you build a pipeline that assumes it fits.
3Test your product on real Ghanaian names
Fragmented names behave differently: they're more expensive, and matching, deduplication and spell-correction all get harder. If your test data is Smith and Johnson, you haven't tested your product.
4Compare tokenizers — they are not interchangeable
Different model families use different tokenizers, and the penalty differs between them. For a heavily Twi or Ga workload this is worth measuring across vendors, not assuming.
5Put text into the world
The long fix is corpora. Ghanaian-language text that is public, well-encoded and openly licensed is what future tokenizers and models learn from. Every properly-encoded Twi or Ga document is a small deposit against this tax.
The honest scope of this page: the numbers here are OpenAI's tokenizers on one carefully-matched sentence per language, plus a set of names. Other model families tokenize differently, and other texts will shift the exact ratios. The pattern — that under-represented languages pay a multiple — is well documented across tokenizers and languages in the research linked below. Treat the specific figures as a measured example, not a universal constant.
✓
Check your understanding
Five quick ones, or the full set. Every answer explains itself, and your first answer is the one that counts.
Five questions across the key ideas.
The lesson at a glance
✓ Measured, not estimated
Where these numbers come from
Token counts and splits were computed with tiktoken (OpenAI's tokenizer), encodings o200k_base and cl100k_base. The comparison text is Article 1 of the Universal Declaration of Human Rights in each language's official translation, so the meaning is held constant. Sources:
tiktoken— OpenAI · the tokenizer used for every count on this page
UDHR in Unicode— the Universal Declaration in hundreds of languages, from the OHCHR translations; the source of the Twi, Ga, Ewe, Yoruba, Swahili and French text
Tokens are where the bill starts. Embeddings are what happens next.
Where this sits: tokenizing turns your text into ID numbers. The very next step turns those IDs into meaning — that's the embeddings explainer, and it opens exactly where this one ends.
This is one of the interactive explainers we make at AgenticGHX — talks, learning tracks, and research out of Ghana.