What it is
A token is the unit of text an AI language model actually works with. You type words, but the model never sees words. Before your message reaches it, a small program called a tokenizer chops the text into pieces and swaps each piece for a number from a fixed list. The model reads those numbers, and when it answers, it produces numbers too, one at a time, which the tokenizer turns back into text you can read.
The pieces are usually not whole words, and not single letters either. They sit somewhere in between. Common short words like "the", "and" or "cat" are usually one token each. Longer or rarer words get split into chunks: a word like "unbelievable" might become "un", "believ" and "able". Spaces and punctuation get folded in too, so " cat" with a space in front can be a different token from "cat" at the start of a line. Numbers, emoji, names and words in other languages often break into many small pieces.
OpenAI's own rule of thumb, printed on its free tokenizer page, is that for ordinary English text one token is about four characters, or roughly three quarters of a word. So 100 tokens is about 75 words, and a 1,000-word blog post comes to something like 1,300 tokens. It is only a rule of thumb. Code, lists of numbers and non-English text usually need more tokens per word.
Why chop text this way at all? It's a compromise. If a model used whole words, it would need an impossibly long list to cover every name, typo and new slang word, and it would be stuck the first time it met a word it had never seen. If it used single letters, every sentence would become a very long sequence and the model would be much slower and more expensive to run. Chunks in between give the best of both. The tokenizer learns which chunks to use by looking at a huge amount of text before the model is trained and keeping the pieces that come up most often. The most common method is called byte pair encoding, which starts from single characters and repeatedly glues together the pairs that appear together most often, until it has a vocabulary of the size its designers want. GPT-2 and GPT-3, for example, shared a vocabulary of 50,257 tokens. Newer models tend to use larger ones.
Each company builds its own tokenizer, so the same sentence can be a different number of tokens in ChatGPT, Claude and Gemini, and can change when a company releases a new model. That's worth remembering whenever you compare prices or limits between tools.
Why it matters
You can use AI for years without ever thinking about tokens. But almost every practical limit you bump into is measured in them, so a little understanding explains a lot of behaviour that otherwise looks random.
Cost is counted in tokens. When a business connects to a model through an API (a way for software to talk to the model directly, billed by usage), the bill is based on tokens. At the time of writing, the major AI companies price their models per million tokens, and they charge separately for the tokens you send in (your instructions, documents and chat history) and the tokens the model writes back, with the output usually costing several times more per token than the input. Prices change often and vary widely between models, so check the provider's current pricing page before you plan anything around a number. Even on a flat monthly chat subscription, the usage limits behind the scenes are largely driven by how many tokens your conversations use.
Memory is counted in tokens. Every model has a context window, the maximum amount of text it can consider at once, and it's measured in tokens. Everything counts toward it: your instructions, any files you paste, the whole conversation so far, and the model's own replies. When a long chat goes past the limit, the oldest material has to be dropped or squeezed into a summary, which is why a chatbot can "forget" something you told it an hour ago. When Google announced Gemini 1.5 Pro on 15 February 2024, it described a one-million-token window as enough for about an hour of video, 11 hours of audio, or over 700,000 words. Pictures and audio are turned into tokens too.
Speed is counted in tokens. A model writes its answer one token after another. Each new token means another pass through the model, a step called inference. That's why answers appear on screen in bursts of word fragments, and why a long answer takes longer than a short one. Speed figures for AI services are usually quoted in tokens per second.
Thinking costs tokens. Models that reason before answering, using chain of thought or built-in extended thinking, do that reasoning by writing extra tokens. At the time of writing, those reasoning tokens generally count toward your usage and bill as output even when you don't see all of them, so a harder question can cost many times more than an easy one. Check how your provider counts them.
Tokens explain some famous blind spots. Because the model sees chunks rather than letters, tasks that depend on individual letters, such as counting them, spelling words backwards or writing a poem where every line starts with a given letter, are surprisingly hard for it. The same goes for some arithmetic, because long numbers get split into chunks in ways that don't line up with place values. Once you know this, these failures stop seeming mysterious, and you know when to hand the job to a calculator or a spreadsheet instead.
Tokens aren't fair across languages. The tokenizers were built mostly from English-heavy text, so English gets the most efficient chunks. The same message in many other languages needs more tokens, which means it costs more, runs slower and fits less into the context window. If you work in more than one language, this matters directly to your budget.
Real-world examples
SolidGoldMagikarp: the words that broke a chatbot
On 5 February 2023, Jessica Rumbelow and Matthew Watkins, working in the SERI-MATS research programme, published a post about a strange discovery. While studying the 50,257 tokens that GPT-2 and GPT-3 models share, they found a cluster of odd entries, including " SolidGoldMagikarp", " TheNitromeFan" and "embedreportprint", that made the models behave bizarrely. When they asked a GPT-3 model to repeat back the string " SolidGoldMagikarp", it replied "distribute". Other tokens in the cluster got evasive answers, insults, words spelled out letter by letter, or complete nonsense. The press picked the story up quickly, and the strings became known as glitch tokens.
A follow-up post on 14 February 2023 traced where some of them came from. SolidGoldMagikarp was the username of one of a handful of Reddit users who took part in r/counting, a community that has spent years counting upwards one comment at a time. Their names appeared so often in the text used to build the tokenizer that each earned a token of its own. The researchers' leading explanation is that the model itself then saw those tokens rarely or never in its training, so it never learned what they meant, and asking it to handle one was like asking it to read a word written in invisible ink.
The lesson for everyday users is not that you'll hit a glitch token (these ones have long been fixed in newer tokenizers). It's that the tokenizer is a separate piece of machinery, built before the model and with its own quirks, and it quietly shapes everything the model can and can't do.
How many r's in "strawberry"?
Through 2024, a simple question became a running joke online: ask a chatbot how many times the letter r appears in "strawberry". Many of the best-known models confidently answered two. There are three. On 27 August 2024, TechCrunch published an explainer titled "Why AI can't spell 'strawberry'", reporting that GPT-4o and Claude both gave the wrong count, and pinning the cause on tokenization. The model doesn't see ten letters. It sees a small number of chunks, perhaps something like "straw" and "berry", and it has to answer a question about letters inside chunks it has never looked inside.
It's a good illustration because the task is trivial for a child and hard for a system that can write a legal summary. That isn't a contradiction once you know about tokens: the model is excellent at patterns between chunks and poor at looking inside one.
The story had a punchline. On 12 September 2024, OpenAI announced o1, a new series of models trained to reason step by step before answering. One of the showcase examples on its announcement page was a scrambled code the model worked out letter by letter, and the decoded message read "THERE ARE THREE R'S IN STRAWBERRY". Reasoning models get the count right more often because they can spell the word out letter by letter and count what they wrote. The underlying token blind spot is still there. They just work around it.
The same message, fifteen times the price
In May 2023, Aleksandar Petrov, Emanuele La Malfa, Philip Torr and Adel Bibi at the University of Oxford released a study called "Language Model Tokenizers Introduce Unfairness Between Languages", later presented at the NeurIPS 2023 conference. They took the same passages translated into many languages and ran them through 17 tokenizers. The number of tokens needed for the same meaning varied enormously. In their words, the same text translated into different languages can have tokenization lengths that differ "up to 15 times in some cases". This held even for tokenizers designed to handle many languages.
The researchers spelled out what that means in practice. Because commercial services charge per token, speakers of some languages pay more for the same service. Their requests take longer to process. And because the context window is measured in tokens, they can fit much less text into it, which matters for tasks like summarising a long document. The authors argued that future models should be built with fairer tokenizers. Newer tokenizers have narrowed the gap for some languages, but it hasn't disappeared, so it's worth testing your own languages rather than assuming.
An everyday case: the translator's quote
Imagine Lucia, a freelance translator who uses an AI model through an API to produce first drafts in English, Spanish and Greek for a client's product catalogue. She sets her price per word, because that's how translation is sold, and estimates her AI costs from a test run in English.
The first invoice from the AI provider is higher than she expected. When she pastes a sample of each language into a free tokenizer, the reason is obvious: the Greek version of the same catalogue page uses noticeably more tokens than the English one, so every Greek draft costs her more to generate and takes longer to come back. Nothing is wrong with the AI. Her estimate simply assumed a token was a word. She reprices the Greek work to cover the difference, and for long catalogues she splits the text into smaller sections so each request stays comfortably inside the context window.
Try it yourself
You need a web browser and ten minutes. No account or payment is required for the first steps.
- Open a free tokenizer. Go to OpenAI's Tokenizer page at platform.openai.com/tokenizer. At the time of writing it lets you paste text without signing in, shows each token in a different colour, and counts both tokens and characters. A similar free tool, Tiktokenizer (tiktokenizer.vercel.app), does the same and lets you pick different models.
- Paste an ordinary paragraph. Take a paragraph from an email or a blog post you wrote. Note the token count and the word count. Divide words by tokens. For plain English you'll usually land near three quarters of a word per token.
- Look at the colours. Notice that common words are single tokens, the space usually travels with the word after it, and long or unusual words break into pieces. Try your own name, your business name, and a product name. Rare names often split into several tokens.
- Try the strawberry test. Type "strawberry" and see how it's split. Then type "Strawberry" with a capital and " strawberry" with a leading space. The splits may differ, which shows how differently the model can see what looks to you like the same word.
- Try numbers and emoji. Paste a long number such as a phone number or a price with decimals, then a row of emoji. Watch how many tokens they use compared with plain words.
- Compare languages. If you know another language, or use a translation tool, paste the same sentence in English and in that language. Compare the token counts. This is the Oxford study's finding in miniature.
- Estimate a real job. Paste a document you regularly give to AI, such as a report or a transcript. Note the tokens. Now you can estimate: will it fit in the model's context window, and roughly what would it cost through an API at the provider's current price per million tokens?
- Ask a chatbot about letters, then help it. Ask any chatbot the question below. Then ask again using the second prompt, which makes it write one letter per line so each letter becomes its own token, and compare the answers.
A first prompt to try:
How many times does the letter r appear in the word "strawberry"? Answer with a number only.
A second prompt that works around the blind spot:
Spell the word "strawberry" one letter per line, numbering each line.
Then count how many of those lines contain the letter r, and give the total.
Common mistakes and misunderstandings
Assuming a token is a word. It's roughly three quarters of a word in plain English, and it can be much less in code, numbers or other languages. Planning a budget or a document size on "one token per word" will underestimate every time.
Forgetting that the whole conversation counts. In a chat, the model re-reads the entire conversation on every turn, and through an API you usually pay for all those input tokens each time. A long thread gets more expensive and slower with every message. When you change topic, start a new chat. When a thread gets long, ask for a summary and continue from that in a fresh one.
Forgetting that output costs more. At the time of writing, output tokens are usually priced higher than input tokens. Asking for "a detailed report" when you need three bullet points costs more and takes longer. Say how long you want the answer to be.
Comparing token counts across companies. A thousand tokens on one model isn't the same amount of text as a thousand tokens on another, because each uses its own tokenizer. Compare real costs for your own sample text rather than headline prices.
Treating the context window as a promise of attention. A model can technically accept a very long input and still pay less attention to material buried in the middle. Fitting your document in the window isn't the same as the model using all of it well. Put the most important instructions at the start or end, and ask for specific sections to be checked.
Blaming the model's intelligence for letter and number tasks. Miscounting letters, garbling a reversed word, or slipping on long multiplication are often tokenization effects, not proof that the model can't reason. Ask it to spell things out one character at a time, or hand exact arithmetic to a calculator, a spreadsheet, or a tool the model can call.
Thinking tokens are only a technical detail. They're where the practical limits live: price, speed, and how much the AI can hold in mind. Five minutes with a tokenizer gives you a better feel for all three than any amount of reading.
Pasting in more than the task needs. Because everything you include uses tokens and competes for the model's attention, a lean prompt with only the relevant material often works better and costs less. Trimming the input is one of the easiest wins in context engineering.
Related elements
- Language Model: the system that reads and writes tokens, one at a time.
- Context Window: the model's working memory, measured in tokens.
- Inference: the step that produces each new token, and where speed and running costs come from.
- Embeddings: the lists of numbers each token is turned into inside the model.
- Transformer: the design that lets a model weigh every token against every other token.
- Pretraining: the stage where a model learns patterns across billions of tokens of text.
- API: the pay-per-token way businesses connect to models.
- Chain of Thought: reasoning written out step by step, which improves answers and uses more tokens.
- Extended Thinking: built-in reasoning that also consumes tokens, often billed as output.
- Context Engineering: choosing what to spend a limited token budget on.
- Translation: a task where token costs differ sharply by language.