Tokens
How AI reads and counts your words
An AI doesn't read whole words like you do. It reads small pieces called tokens. Tokens explain why AI has limits and why using it through an API costs money.
What is a token?
Think of LEGO bricks. A big LEGO castle is built from many small bricks. In the same way, a sentence is built from many small tokens.
A token is a small piece of text. It can be:
- A whole short word, like "cat" or "the"
- Part of a longer word, like "un" + "believ" + "able"
- A space, a number, or punctuation like "!" or ","
The AI turns your text into tokens before it reads it. It also writes its answer one token at a time.
| Feature | Tokens (example) | Count |
|---|---|---|
| hello | hello | 1 |
| I love pizza! | I · love · pizza · ! | 4 |
| unbelievable | un · believ · able | 3 |
| homemade | home · made | 2 |
A quick rule of thumb
You don't need to count tokens exactly. For English text, this rough rule is good enough:
- 1 token ≈ 4 characters
- 1 token ≈ ¾ of a word
- So 100 tokens ≈ 75 words
A one-page essay of about 750 words is roughly 1,000 tokens. Other languages and code often use more tokens for the same amount of text.
Want to see it? Try this prompt:
Roughly how many tokens is this paragraph?
Explain how you estimated it in one sentence.
"Our handmade candles are poured in small batches using
natural soy wax and essential oils."Why tokens matter
1. Limits
Every model can only handle a certain number of tokens at once. That includes your prompt, any files you paste, the chat so far, and its answer. Very long chats or huge documents can hit this limit. You'll learn more on the next page.
2. Cost
When your app talks to a model through an API, you pay per token. Both the tokens you send and the tokens you get back count. Short, focused prompts cost less.
This matters when you add AI features to your own apps. For example, every time a shopper asks an online shop's AI assistant a question, tokens are used. Check the provider's pricing page to see current costs.
3. Answer length
Models also have a limit on how long one answer can be. If a reply stops in the middle, just type "continue".
Common mistakes
- Thinking 1 token = 1 word. It is closer to ¾ of a word in English.
- Pasting huge documents when you only need one part. Paste the relevant section. It's cheaper and the AI stays focused.
- Forgetting the answer counts too. Asking for a 5,000-word answer uses many tokens. Ask for "a short summary" when that's enough.
Practice
Try these exercises to lock in what you just learned.
What's next?
Tokens fill up a limited space in the AI's memory. Next, let's look at that space: the context window.