What is Token?
A chunk of text (roughly a word-piece) that language models read and generate.
A token is a small chunk of text — often a word or part of a word — that a language model processes one at a time.
Usage and pricing for most AI APIs are measured in tokens, and a model's context window is also counted in tokens.
A token is the unit of text a language model actually reads and generates. Models don't process whole words or characters; they break text into tokens, which are roughly word-fragments — common words are often a single token, while longer or rarer words split into several. As a rough rule, a token is about four characters of English, or three-quarters of a word, so 1,000 tokens is roughly 750 words.
Tokens matter for two practical reasons. First, cost and limits are measured in them: API pricing is per token, and a model's context window — how much it can consider at once — is a token count, not a word count. Second, the model generates one token at a time, so longer outputs take proportionally longer and cost more. Understanding tokens explains why pasting a huge document can exceed a model's limits, and why usage-based AI bills scale with how much text flows in and out.
Why it matters
Tokens are the currency of language models — they determine cost, speed and how much a model can handle at once. Anyone using AI APIs, hitting context-window limits, or watching usage-based bills benefits from understanding them, because "too long" errors, pricing and performance all come back to token counts rather than word counts.
A concrete example
The word "cat" is one token, but "untokenizable" might split into several ("un," "token," "izable"). When a tool says it has a "128k context window," it means 128,000 tokens — roughly 96,000 words, or a few hundred pages — that the model can consider in a single conversation before older content drops out.
Where you’ll meet it in AI tools
Tokens surface whenever a tool exposes usage-based pricing (charged per token), a context-window limit (a token count), or rate limits. Most consumer tools hide tokens behind simpler plans, but developers building on AI APIs work in tokens directly, and anyone pasting long documents into an assistant is implicitly bumping against token limits.
Token: FAQ
How many tokens are in a word?
Roughly speaking, one token is about three-quarters of a word, or four characters of English — so 1,000 tokens is about 750 words. But it varies: common short words are usually a single token, while long, rare or non-English words split into several, and spaces and punctuation count too. The ratio is a useful estimate for gauging cost and context limits, but the exact token count depends on the specific text and the model's tokenizer.
Why does AI pricing use tokens instead of words?
Because tokens are what the model actually processes — they're the true unit of computation, while "words" are a human concept the model doesn't work in. Pricing and context limits are measured in tokens because that's what determines the cost and the model's capacity. Both your input and the model's output count, which is why a long prompt with a long answer costs more than a short exchange, and why usage-based AI bills track total tokens in and out.
Related terms
Find language models tools
Browse hand-reviewed AI tools — compare on pricing, features and real ratings.