Tokens in AI are the pieces of text into which an AI language model splits your input and its own answer. A token is sometimes a whole word, sometimes part of a word or a punctuation mark. A large language model (LLM) processes text as tokens, not as words.
The splitting is called tokenisation and follows a fixed vocabulary for each model. For English text, a rule of thumb is roughly four characters or three quarters of a word per token; for most other languages, no such fixed conversion exists.
You count tokens with a tokenizer or a counting feature from the provider, and they come in two kinds: input and output tokens. A model’s context window is expressed in tokens, and AI providers charge for their API per token. Images and PDFs are also converted into tokens, and once you know that, you use them more sparingly.
What is a token in AI?
A token in AI is the unit of text a language model calculates with. Each token gets a fixed number from the model’s vocabulary, and the model processes that sequence of numbers instead of the letters themselves.
Short, common words such as “the” or “and” are usually one token. A long or rare word breaks down into several tokens. Spaces and punctuation also count, sometimes as part of a word token and sometimes as a separate token.
Outside AI, the word “token” has other meanings, such as a crypto token or a login token. In the context of ChatGPT, Claude and Gemini, it refers to these pieces of text.
How does a language model split text into tokens?
A language model splits text into tokens with a tokenizer: a program that compares the text with a fixed vocabulary and divides it into the longest pieces that appear in that list. Most modern tokenizers are based on byte pair encoding.
Byte pair encoding starts with individual characters and, step by step, merges the character pairs that most often occur together into new tokens. That way, common words end up in the vocabulary as a whole, and rare words are built from familiar pieces. A long compound word such as the Dutch “arbeidsongeschiktheidsverzekering” (disability insurance) therefore breaks down into several tokens.
Tokenizers differ by provider and sometimes by model generation. Anthropic, for example, reports that its newer tokenizer produces roughly 30 percent more tokens for the same text than the previous one. A language model then predicts token by token what comes next, which explains why an answer appears on your screen piece by piece.
How many words is a token?
In English, a token is on average about three quarters of a word, but that ratio differs by language and by model. So there is no conversion that holds for every text.
How many tokens is English text?
According to OpenAI, English text comes to roughly one token per four characters, which works out at 100 tokens per 75 words. An English text of 1,500 words therefore comes to around 2,000 tokens.
OpenAI explicitly calls these figures an estimate. Code, tables and text with many numbers or capital letters deviate from them.
How many tokens is text in other languages?
For text in other languages, such as Dutch, OpenAI and Anthropic publish no rule of thumb. OpenAI only gives its ratio for English and writes that other languages have a different ratio between characters, words and tokens.
The cause lies in the tokenizer’s vocabulary. Compound words and inflected forms in other languages appear less often in that list as a whole, so they are more often split into several tokens. If you need an exact count for a text in another language, count it with the model’s own tokenizer.
How do you count tokens before you send a prompt?
You count tokens in advance with the tokenizer or counting feature of the provider whose model you use. A count made with another model’s tokenizer gives a different number.
OpenAI offers an online Tokenizer and the programming library tiktoken. Anthropic offers a free counting feature in its API that counts the input tokens of a complete message, including the system prompt, images and PDFs. According to Anthropic, that count is an estimate that in some cases differs slightly from actual usage.
In a chat interface such as ChatGPT or claude.ai, you do not see a token count per message. There, you only notice token usage through your plan’s usage limit.
What is the difference between input and output tokens?
Input tokens are the tokens you send to the model; output tokens are the tokens the model generates. Your question, your attachments and the earlier messages in the conversation count as input, and the answer counts as output.
At the major providers, output tokens cost more than input tokens, because the model generates output tokens one at a time, whereas it processes the input in one go. Reasoning models add a third stream: tokens for the intermediate steps that you do not see as answer text. According to OpenAI, these reasoning tokens count as output and are billed as such.
What does the context window have to do with tokens?
The context window is the maximum number of tokens a model processes at once, and that number covers both input and output. Anthropic describes a context window of 200,000 tokens as roughly 500 pages of text.
- Earlier messages
- Your question
- Attachments
- The answer
- Reasoning tokens
In a long conversation, the input grows with every message, because the earlier messages are sent to the model again. When the window is full, the tool summarises older messages, drops them or ends the conversation. That is why it pays to give context to an LLM concisely and to the point, instead of piling everything into one conversation.
Why do AI providers charge per token?
AI providers charge per token because the computing power a model uses grows with the number of tokens it processes and generates. A price per million input tokens and per million output tokens is therefore the most direct reflection of what a request costs.
That billing applies to the API. With a subscription, you pay a fixed amount per month and get a usage limit, with no token counter. If you compare Claude pricing for a subscription and for the API side by side, you see that the difference lies mainly in how predictable your usage is.
While developing a new workflow on the Claude API, I noticed that costs mainly rise during the testing phase. In that phase, you iterate more often than in production, and every test round counts in tokens.
Do images and files count as tokens too?
Yes, images and files count as tokens too, because a model converts them into a sequence of tokens before processing them. Claude divides an image into blocks of 28 by 28 pixels; an image of 1,000 by 1,000 pixels therefore costs 1,296 tokens.
With Claude, a PDF counts twice. Anthropic converts each page into an image and supplies the text of that page alongside it. According to Anthropic, the text of a page usually costs 1,500 to 3,000 tokens, plus the tokens for the image of that page.
How do you use tokens sparingly in your work?
Using tokens sparingly starts with short, focused input: share only the pages or passages that are needed, and start a new conversation as soon as the topic changes. That keeps room in the context window for what matters.
Also ask for a short answer when a long one adds no value, because output tokens are the most expensive. For the question of how many tokens you get with Claude Pro, a different logic applies: there, a usage limit per session determines how much room you have.
If you understand how tokens, context windows and usage limits fit together, you can make better choices about which tool and which plan suit a task. That knowledge forms the basis of the AI Literacy Course. In it, you learn how language models work, where they make mistakes and which agreements a team needs for responsible use.
Frequently asked questions about tokens in AI
If you work through an API, you deal with tokens in a few other ways besides the prompt and the answer.
What are cached tokens?
Cached tokens are input tokens that a provider reuses from an earlier request, so that a fixed system prompt or a long document is not fully processed again every time. With the Claude API, reading from the cache costs a tenth of the normal input price by default, and even less with some models. Writing to the cache, on the other hand, costs more than normal input. By default, a cache lasts five minutes, and each reuse extends that period at no extra cost.
What happens when an answer reaches the maximum number of tokens?
If an answer reaches the maximum number of tokens you set in your request, the model stops mid-text. The Claude API reports this with the stop reason max_tokens. Anthropic recommends marking the output as incomplete and raising the limit, or having the model continue from the point where it stopped.
Do tools count towards the number of tokens?
Yes, tools count towards the number of tokens, because the name, description and schema of each tool go to the model as input. According to Anthropic, the API also adds its own system prompt for tool use, the size of which varies by model. The model’s requests to use a tool and the results that come back are counted as tokens as well.
